Backup and Restore
This section defines how routine backups and restore are done to maintain the safety and integrity of the stored artifacts.
Overview
ModelOps stores all artifacts in an embedded database that is only accessible through the ModelOps UI. In general, this database does not require any user maintenance.
The database is stored external to the ModelOps Server Pod on a Persistent Volume Claim (PVC) named modelops-server. This PVC uses a ReadWriteOnce access mode with a size set during installation based on the configured system sizing.
// // Display details on modelops-server PVC // kubectl get pvc --namespace modelops modelops-server NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE modelops-server Bound pvc-7abeecd8-fb0c-4d80-894f-3560770411f2 20Gi RWO default 6d5h
The PVC is mounted at /var/opt/tibco/models/data in the ModelOps Server Pod.
Because the database is stored on a PVC, the ModelOps Server Pod can be safely restarted or removed without any data loss.
Backups
A backup is performed on the internal database every time the ModelOps Server Pod is restarted, for example because of a ModelOps product upgrade, or manually to force a backup.
Backups are exported as JSON and are stored in /var/opt/tibco/models/data/backups with this naming convention:
ModelOps-<backup-date>.json
The backup files are never automatically deleted. It is recommend that older backups be manually deleted, or moved to long-term archival storage if required for governance, to prevent exhausting the allocated PVC space. Standard Kubernetes tools can be used to access the running Pod to remove backup files, for example using kubectl:
// // Use the selector to display the ModelOps Server Pod // kubectl get pods --selector app=modelops-server --namespace modelops NAME READY STATUS RESTARTS AGE modelops-server-6c744659d7-ll7st 1/1 Running 0 2m13s // // Log in to ModelOps Server Pod // kubectl exec -it modelops-server-6c744659d7-ll7st --namespace modelops -- /bin/bash // // Change to backups directory // bash-4.4$ cd $MODELOPS_RUNTIME_HOME/data/backups bash-4.4$ ls ModelOps-2022-03-22-15-25-51.json ModelOps-2022-03-23-23-39-45.json ModelOps-2022-03-24-11-40-55.json
Monitoring
As more artifacts are added to the system, the size of the embedded database will grow. In addition, every backup will consume additional storage on the PVC until removed. To ensure that the allocated PVC storage is never exhausted, the utilization of the ModelOps Server PVC should be monitored. PVC utilization can be monitored using standard Kubernetes monitoring tools, for example kubedf:
kubedf PVC 1K-blocks Used Available Use% artifact-repository 20466256 2413160 18036712 12% elasticsearch-data-elasticsearch-es-default-0 51290592 441744 50832464 1% git-server 20466256 5616 20444256 0% modelops-server 20466256 17236 20432636 0%
Or Lens:
Restore
The embedded database can be restored from the backup files.
NOTE: Restoring the embedded database causes a service outage.
The steps to restore the database are:
- Log into the ModelOps Server Pod.
- Restore the database from a backup.
Once a restore has completed, the ModelOps server will automatically restart.
Accessing the ModelOps Pod
The ModelOps Pod name can be found with this command:
// // Use the selector to display the ModelOps Server Pod // kubectl get pods --selector app=modelops-server --namespace modelops NAME READY STATUS RESTARTS AGE modelops-server-6c744659d7-ll7st 1/1 Running 0 2m13s
Use the Pod name with this command to access a shell in the Pod:
kubectl exec -it modelops-server-6c744659d7-ll7st --namespace modelops -- /bin/bash Defaulted container "modelops-server" out of: modelops-server, wait (init) bash-4.4$
Restoring the Database
From the shell in the ModelOps Server Pod execute these commands to restore the database.
When the restore is complete the ModelOps Server Pod will be automatically restarted.
// // Change to the runtime directory // cd $MODELOPS_RUNTIME_HOME // // Choose the backup file to use for the restore // ls data/backups/ ModelOps-2022-03-26-00-20-59.json ModelOps-2022-03-26-00-22-40.json ModelOps-2022-03-26-14-24-45.json // // Execute the restore // ./restore data/backups/ModelOps-2022-03-26-14-24-45.json Moving database to data/database-restored Restoring database from data/backups/ModelOps-2022-03-26-14-24-45.json... ... Restore complete from data/backups/ModelOps-2022-03-26-14-24-45.json Will shutdown server in 10 seconds Interrupt to avoid shutdown... Shutting down ModelOps server bash-4.4$ command terminated with exit code 137
After the restore completes, the ModelOps server will be restarted in a new Pod, using the data from the backup file.