Deployment

Overview

The ModelOps product has these major infrastructure services installed in a Kubernetes cluster after a successful installation:

• Artifact management - storage, versioning, and design-time editing support for artifacts (flows, pipelines, data channels, models, etc.). Hosts ModelOps user interface.
• Chart repository - stores helm charts to instantiate scoring pipelines and data channels.
• CI/CD - provide build and deployment support for ModelOps installation, scoring pipelines, and data channels using Tekton.
• Data channel registry - runtime discovery of deployed data channels.
• Git repository - stores artifacts to make them available for scheduling.
• Log capture - ElasticSearch and Kibana to support log persistence and query access.
• Maven repository - stores Maven product dependencies required to instantiate ModelOps and deploy scoring pipelines and data channels.
• Metrics store - Prometheus to support metrics persistence.
• Real time metrics - metrics aggregation to support visualization.
• Scheduling - scheduling services for scoring pipelines and data channels.

In addition, container images built during ModelOps installation using the CI/CD service are stored in a container image registry provided by the cloud infrastructure.

Example scoring pipeline

TBD - add data flow description

Scoring lifecycle

Note detailed data channel interactions are skipped here.

1. The modelops user uses the UI to define the scoring pipeline (models + scoring flows + data channels)
2. The modelops user deploys the scoring pipeline in the form of a internal representation (IR)
3. The modelops user starts a scoring job
4. The Scheduling Server API is called to start the scoring job
5. A new tekton pipeline run is created for this scoring pipeline
6. The first task of the pipeline, install, is started
7. The unique job identifier is returned to the modelops server - this can be used to query or cancel the scoring job
8. The install task downloads the IR
9. Scoring pipeline helm chart values are calculated (also known as pre-processing)
10. The scoring pipeline helm chart is deployed, parameterized by the values file that represents the IR
11. The second task of the pipeline, wait, waits for the scoring to complete ( and pass back any errors found )
12. Helm chart templates are used to process the values into scoring flow and scoring service configurations ( in the form of kubernetes objects )
13. The creation of the scoring service kubernetes objects starts the scoring service(s)
14. Each scoring service downloads models it needs
15. The creation of the scoring flow kubernetes objects starts the scoring flow(s)
16. Each scoring flow waits for its associated scoring service to be up and running before starting
17. Once each scoring flow application is running, the scoring flow configuration is loaded and activated
18. The modelops server can use the job identifier to query the scoring job status
19. The scoring job status includes details such as state and pipeline task
20. Batch scoring jobs may determine that scoring is complete
21. Scheduling service is informed that scoring is complete
22. The pipeline wait task is cancelled - this triggers the pipeline to move to the finally task
23. The pipeline finally tasks is started
24. The helm chart is un-installed
25. The process of uninstalling the helm chart deletes the scoring flow kubernetes objects
26. The process of uninstalling the helm chart deletes the scoring service kubernetes objects

The pre-processing of the IR includes the following :

• Lookup sources and sinks in data channel
• Apply parameters to scoring flow
• Merge all the IR files into a single helm values file ( easier for the helm templates to use )
• Apply global parameters ( such as trace settings, location of git server etc )

Scoring lifecycle shutdown by Modelops server

TBD - add scoring lifecycle cancel description

Scoring lifecycle with namespaces

TBD - add scoring lifecycle namespaces description

Sub charts

The main modelops helm chart can import helm sub-charts via the normal maven dependency mechanism. The installation pipeline uses these sub-charts to build container images and deploy the built applications. Hence this is a useful way to extend the modelops helm chart.

The following sub-charts are included :

• Container tools
• Streaming runtime base
• Modelops server
• Scoring flow
• Scoring severs ( pmml, tensorflow, statistica and python )
• Data channel
• Scheduling server
• Test data source and sink

Each sub-chart may define :

• In values.yaml :
  • List of container build steps in values.yaml
  • Build steps that need to be built first
  • Any sub-chart specific settings ( for example scoring services define a listt of supported models )
• In templates directory :
  • Define any custom build steps as tekton task
  • Deployment configurations ( kubernetes Deployment, Service, ConfigMap etc )