TIBCO ModelOps Overview
This page introduces TIBCO ModelOps and explains different concepts within.
Contents
High-Level Concepts
TIBCO ModelOps is best defined as the intersection of DevOps and Model Management. It encompasses the systems and processes that automate the deployment, monitoring, governance, and continuous improvement of different models. These models include analytic, data science, artificial intelligence, and rules-based models. These features help you maintain and improve mission-critical processes. TIBCO ModelOps key aspects include:
- Operationalizing (many) predictive models as managed and reusable software artifacts using a repeatable deployment process.
- Providing an environment for collaborative, ongoing processes between data science and IT
- Encompassing unique management aspects that span model inference, scalability, maintenance, auditing, and governance.
- Incorporating ongoing model monitoring requirements in production environments, unlike software engineering that has a (generally) fixed output. For example, model predictions change/degrade over time. Monitoring means that feedback from the existing model performance can be incorporated into future iterations of models to improve results.
TIBCO ModelOps is a revision control system that maintains current and historical versions of artifacts in its repository. Through a web client, you can create, manipulate, and score versioned models. Models can be tested and deployed to your TIBCO Streaming applications over the cloud.
With TIBCO ModelOps, you can manage, store, and deploy many artifact types, including model files and decision tables. From the TIBCO ModelOps perspective, an artifact is considered a data file.
Users can be assigned one or more roles within TIBCO ModelOps, with different privileges assigned to each role. A TIBCO ModelOps administrator is typically responsible for creating new users and managing their roles and privileges. Roles and privileges set the guidelines for artifact access and management.
Terminology
The following terminology applies to models and model-related tasks.
What is a Model?
A model is an artifact that describes a computational process to convert one or more input data sources to one or more output data or other artifacts. An example of a model is a machine learning model to predict a continuous numeric outcome, based on several continuous or categorical numeric inputs, encoded in PMML4.1 format. An example of a complex model is a Python-based model that generates one or more nonnumeric outcomes, such as computer-generated or augmented pictures, based on one or more numeric or nonnumeric inputs. Models can also be used to monitor prediction models (Model Monitors), for example, to compute certain statistics based on a set of predictions from a prediction model and measure model accuracy or quality.
Model Scoring Mechanism
Models can be deployed in various ways. For example, to support a batch scoring process, real-time scoring process, services that are used by external business applications, or directly to an external application (such as to embed models into other processes). In some instances, the specific deployment mechanism might necessitate additional activities during the model-building lifecycle. This is done to map the model and model monitoring conceived during their creation to the actual data and data structure encountered during deployment. For example, models built against 1-minute-interval sensor data can be deployed against actual streaming data that is collected as raw data (events) that is not aggregated; in that case, additional data preparation and mapping is required before models can finally be committed to the production environment.
Model Management System
TIBCO ModelOps consists of services that organize models and model metadata (properties) into projects. These projects are arranged into manifests that are hierarchically or otherwise structured to align with organizational requirements and project hierarchies. TIBCO ModelOps also maintains versions, approvals, and other metadata about models. TIBCO ModelOps exposes the appropriate APIs to enable you to interact with and annotate models and model metadata consistent with the respective project’s requirements, as configured (for the project). Also, TIBCO ModelOps provides project and artifact-level permissions for specific users and groups of users (roles).
For example, you can commit or update models to TIBCO ModelOps and specify model metadata for projects where you are so authorized, but where you cannot create new projects or change the specific metadata that is required for a model to be successfully moved to production deployment. TIBCO ModelOps also enables capabilities for searching models and model metadata (for example, find all Models created by you within a certain date interval).
Model Metadata
As described under the definition of the model, the computations encoded into a model, when executed, generally convert one or more input data sources into one or more output data or other artifacts. In addition to the specific details that define the conversion of inputs to outputs, additional metadata is stored to describe the model, either as properties or as artifacts of various types. This metadata is generally dictated by the specific applications to which the model is to be applied.
Model Schemas
In addition, a model must have input and output schemas. Input schema describes the structure of the data that enters the model. On the other hand, Output schema describes the structure of the data that leaves the model. It is necessary to bind schemas to the model for the model to be used within a scoring flow.
Structure and Relationships between Models
Models rarely exist without a context of other models and the projects that they support. For example, a team of model builders, business stakeholders, and DevOps Engineers might create multiple models for credit-default risk, as part of a project to manage credit-default risk. Inside of that project you can have prediction models for auto loans, mortgages, consumer loans, and so on. Further, for each credit risk model, there can be additional “models” — that is, analytic templates that generate the required reports for monitoring, compliance reporting, and so on. So a credit risk model might be associated with numerous reports that validate compliance for the respective prediction models, and also define the model monitoring applied to the prediction model.
Artifact Workflow
As described earlier, a user with the appropriate role and privileges can sign in to TIBCO ModelOps and create a project and an artifact, or check out an existing artifact to work on. A business user can save the artifact locally and commit the changes for approval. A reviewer with approval privileges can review the changes the business user made and decide whether to approve or reject the changes.
In addition to this, a user with the appropriate role and privileges can add or modify artifacts at any point in time. The modified or added artifacts can be clearly distinguished in the Project Explorer pane with the letter M or A, respectively.
- Sandbox Space: Changes made locally to the artifacts by you can be seen here.
- Published Space: Once the changes are committed, they can be seen here. Unless the edits are not committed and approved, they are not visible to other users in Published Space.
Model artifacts can be continually evaluated through a scoring mechanism. Artifacts can be deployed to TIBCO Streaming applications across the cloud.
If the reviewer approves the committed changes:
- The artifact version becomes the published version in the repository.
- The changes become visible to any user who has privileges to view the artifact.
- The artifact state remains working, allowing you to continue making changes to it (thus requiring another save and publish cycle if necessary).
- The artifact history is updated and assigned an incremental version number.
- Users with deploy privileges can deploy model artifacts to TIBCO Streaming applications.
If the reviewer does not approve the committed changes:
- Changes do not become part of the artifact’s history nor are they persisted in the TIBCO ModelOps repository.
- Sandbox changes are preserved and remain local to the committing user.
- You can recommit the artifact (for example, after making additional edits to the artifact).