Model Management Server Installation on RedHat Openshift

ADDITIONAL VALUES REQUIRED FOR THIS STEP

• ELASTIC_PW
• GIT_SERVER_PW
• NEXUS_SERVER_PW
• PROMETHEUS_PW
• SCORING_ADMIN_PW
• ARTIFACT_MANAGEMENT_PW
• Oauth2 server details
• External DNS details

In order to avoid clear text passwords, Kubernetes provides a Secrets facility. So prior to installation, Kubernetes Secrets have to be created to contain the passwords required by MMS.

Assign each of the names above to some value, then issue the following commands to set up the secrets.

NOTE: Without secrets installed, the [helm installation] (#install-helm-chart) step will be in a wait mode and eventually time out.

Clear out any old secrets that may exist with this series of delete secret commands:

    kubectl delete secret git-server --namespace mms
    kubectl delete secret nexus-server --namespace mms
    kubectl delete secret prometheus-server --namespace mms
    kubectl delete secret scoring-admin --namespace mms
    kubectl delete secret oauth2 --namespace mms
    kubectl delete secret externaldns-config --namespace mms
    kubectl delete secret artifact-management --namespace mms
    kubectl delete secret artifact-management-env --namespace mms

Create new secrets:

    kubectl create secret generic elasticsearch-es-elastic-user \
        --from-literal=elastic=$ELASTIC_PW \
        --namespace mms --dry-run=client --output=yaml 2>/dev/null > secret.yaml
    kubectl apply --filename secret.yaml
    kubectl create secret generic git-server \
        --from-literal=mms=$GIT_SERVER_PW \
        --namespace mms
    kubectl create secret generic nexus-server \
        --from-literal=admin=$NEXUS_SERVER_PW \
        --namespace mms
    kubectl create secret generic prometheus-server \
        --from-literal=admin=$PROMETHEUS_PW \
        --namespace mms
    kubectl create secret generic scoring-admin \
        --from-literal=admin=$SCORING_ADMIN_PW \
        --namespace mms
    kubectl create secret generic artifact-management \
        --from-literal=admin=$ARTIFACT_MANAGEMENT_PW \
        --namespace mms
    kubectl create secret generic artifact-management-env --namespace mms

NOTE: The Elasticsearch password is limited to alphanumeric, “.”, “_”, “~”, and “-” characters, i.e. it must conform to this regular expression (‘^[a-zA-Z0-9._~-]+$’)).

NOTE: The prometheus-server secret is only used if you are not using OAuth2 authentication; if using OAuth2, this secret need not be created. It should consist of name value pairs where each name is a user name, and each value is the corresponding password. These are used to secure the Prometheus ingress with basic authentication.

Auto update of public DNS is possible by enabling the (external DNS)[https://github.com/kubernetes-sigs/external-dns] component - without this the DNS will need to be updated manually. External DNS on azure requires a secret containing azure credentials that has permissions to update DNS (tenant id, subscription id, dns resource group and client secret.

kubectl create secret generic externaldns-config --from-literal=azure.json="{ \"tenantId\": \"$TENANT_ID\", \"subscriptionId\": \"$SUBSCRIPTION_ID\", \"resourceGroup\": \"$DNS_RESOURCE_GROUP\", \"aadClientId\": \"$SERVICE_PRINCIPAL_ID\", \"aadClientSecret\": \"$CLIENT_SECRET_KEY\" }" --namespace mms

Openshift on Azure

Create a secret for accessing the fileshare in azure

        kubectl create secret generic mms-storage --from-literal=azurestorageaccountname=$STORAGE_ACCOUNT \
            --from-literal=azurestorageaccountkey=$STORAGE_ACCOUNT_KEY \
            --namespace mms

        helm upgrade \
            --install mms ${home}/mms/kubernetes-installer-1.4.0.tgz \
            --set cloud=openshift \
            --set openshift.oauth2=$OUTH2_SERVER_TYPE \
            --set openshift.externalDNS=azure \
            --set externalDNS.azure.clientID=$CLIENT_ID \
            --set externalDNS.azure.subscriptionID=$SUBSCRIPTION_ID \
            --set externalDNS.azure.tenantID=$TENANT_ID \
            --set externalDNS.azure.hostedZoneName=$DNS_ZONE_NAME  \
            --set externalDNS.azure.resourceGroupName=$DNS_RESOURCE_GROUP \
            --set openshift.networkExposure=ingress \
            --set size=large \
            --set openshift.ingressDomain=$(kubectl get ingresses.config/cluster -o jsonpath={.spec.domain}) \
            --set openshift.containerRegistry=$IMAGE_STREAM \
            --namespace mms \
            --set artifactManagement.storageClass=azurefile-csi \
            --set cloudProvider=azure \
            --set artifactManagement.azure.fileshare.fileshareName=$FILESHARE_NAME
            --set artifactManagement.azure.fileshare.resourceGroup=$FILESHARE_RESOURCE_GROUP

OpenShift on AWS

For EFS filesystem follow the steps mentioned in the (documentation)[https://cloud.redhat.com/experts/rosa/aws-efs/] Create a secret to access the EFS, and pass the $ROLE created to access the EFS as per the above documentation.

        cat << EOF | oc apply -f -
            apiVersion: v1
            kind: Secret
            metadata:
                name: aws-efs-cloud-credentials
                namespace: openshift-cluster-csi-drivers
            stringData:
            credentials: |-
                [default]
                role_arn = $ROLE
                web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
        EOF

        helm upgrade \
            --install mms ${home}/mms/kubernetes-installer-1.4.0.tgz \
            --set pullpolicy=Always \
            --set openshift.oauth2=$OUTH2_SERVER_TYPE \
            --set cloud=openshift \
            --set openshift.ingressDomain=$DOMAIN \
            --set openshift.containerRegistry=$IMAGE_STREAM \
            --set openshift.networkExposure=ingress \
            --namespace mms \
            --set openshift.externalDNS=aws \
            --set externalDNS.aws.eksRoleArn=$ROLE_CERT_MANAGER \
            --set externalDNS.aws.region=$REGION \
            --set externalDNS.aws.hostedZoneID=$HOSTED_ZONE_ID \
            --set openshift.containerRegistryPrefix=mms \
	        --set cloudProvider=aws \
            --set artifactManagement.aws.efs.efsId=$EFS_ID \
            --set artifactManagement.storageClass=efs-sc 

OpenShift on GCP

Create a secret to update DNS

        kubectl create secret generic clouddns-dns01-solver-svc-acct --from-file=key.json=${keyJsonfile} -n mms

        helm upgrade \
            --install mms ${home}/ep-kubernetes-installer-1.4.0.tgz \
            --set pullpolicy=Always \
            --set cloud=openshift \
            --set openshift.ingressDomain=$DOMAIN \
            --set openshift.containerRegistry=$IMAGE_STREAM \
            --set openshift.networkExposure=ingress \
            --namespace mms \
            --set openshift.externalDNS=gcp \
            --set openshift.containerRegistryPrefix=$PREFIX \
            --set artifactManagement.storageClass=filestore-sc \
            --set openshift.clusterType=internal \
            --set artifactManagement.gcp.filestore.instance=$FILESTORE_IP \
            --set artifactManagement.gcp.filestore.zone=$FILESTORE_ZONE \
            --set artifactManagement.gcp.filestore.fileshare=$FILESTORE_NAME \
            --set artifactManagement.gcp.filestore.mountPath=$MOUNT_PATH \
            --set cloudProvider=gcp \
            --set gcp.projectID=$GCP_PROJECT

Delete all completed TaskRuns in the namespace

tkn tr list --namespace mms -o name | sed 's#taskrun.tekton.dev/##' | xargs -I {} tkn tr delete {} --namespace mms -f

Delete all completed jobs in the namespace

kubectl delete jobs --all --namespace mms

If you want to keep some tasks or jobs for debugging purposes, you can use the following command to keep specific TaskRuns & jobs:

List all TaskRuns in the specified namespace, filter out the one to keep, and delete the rest

tkn tr list --namespace mms -o name | sed 's#taskrun.tekton.dev/##' | grep -v -E "$taskrun1_to_keep|taskrun2_to_keep" | xargs -I {} tkn tr delete {} --namespace mms -f

Replace ‘job1-to-keep’ and ‘job2-to-keep’ with the names of the jobs you want to keep

kubectl get jobs --namespace mms -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep -v -E '^(job1-to-keep|job2-to-keep)$' | xargs -I {} kubectl delete job --namespace mms {}