Working with Data Channels

Contents

• Overview
• Adding a Data Channel
• Configuring Data Channels
• Configuration Reference
  • File Data Channels
  • JDBC Data Channels
  • Kafka® Data Channels
  • REST Data Channels
• Approving Data Channels
• Deploying Data Channels
• Viewing Data Channels

Overview

A data channel is a configurable and deployable component that maps between an external protocol and scoring flows. A data sink is a data channel that consumes output data with a known schema and a standard serialization format. It defines the format of the end result. On the other hand, a data source is a data channel that provides input data with a known schema and a standard serialization format.

Adding a Data Channel

1. In the Project Explorer pane, select Data Channels.
2. Click Create a data channel to create a new data channel. You can also click the Add one option to add a new data channel if there are none present.
3. On the Create a Data Channel page, select the project, for which you wish to create the data channel, from the drop-down list.
4. Add data channel name and description, if needed.
5. Click Create.

   

Configuring Data Channels

This section shows how to configure a data channel.

1. In the Project Explorer pane, select Data Channels.
2. Select the data channel, which you wish to configure, from the list.
3. On the Data Channel Type tab, select the data channel type from the drop-down list.

   

4. Provide the required configuration for the data channel type. See the configuration reference for the data channel type being configured.

5. On the Schema Definition tab, you get the following option to add a schema definition.
   • Enter it manually: Builds schema from scratch.
   • Get from CSV file: Extracts schema from a CSV file. You need to select a csv file if you choose this option.
   • Get from Data Channel: Adds schema from an existing data channel. You need to select a data channel if you choose this option.
   • Get from Model: Extracts schema from a model. You need to select a model if you choose this option.
   • Get from Scoring Flow: Extracts schema from a scoring flow. You need to select a scoring flow if you choose this option.
6. Once done, click Save.

Data Channel Configuration Reference

This is the configuration reference for these data channels:

• File Data Channels
• JDBC Data Channels
• Kafka® Data Channels
• REST Data Channels

Each data channel requires different configuration which is summarized in the sections below.

kubectl create secret generic <jdbc-channel-secret-name>  \
            --from-literal=jdbc-database-username=<username> \
            --from-literal=jdbc-database-password=<password> \
            --namespace <data-channel-namespace>

kubectl create secret generic <jdbc-channel-secret-name>  \
            --from-literal=jdbc-database-username=<username> \
            --from-literal=jdbc-database-password=<password> \
            --namespace <data-channel-namespace>

test.com:9093[security.protocol=SASL_SSL,sasl.mechanism=PLAIN|sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=“admin” password=“********”;]
test2.com:9093[security.protocol=SASL_SSL|sasl.mechanism=SCRAM-SHA-256|sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=“admin” password=“********”;]

https://<subdomains>.<modelops-kubernetes-hostname>/<endpoint-path-prefix>

https://rest.data.channels.example.tibco.com/taxi-source-data

https://rest.data.channels.example.tibco.com/taxi-source-data

https://rest.data.channels.example.tibco.com/train-source-data

Approving Data Channels

1. In the Project Explorer pane, select Data Channels.
2. Select the data channel that needs to be approved.
3. Click Approve present in the right panel.

   

4. Click Approve present at the bottom of the list of data channels.

5. Select the desired environments by toggling the switch and close the pop-up window.

   

This data channel is now approved for deployment to the selected environments.

Note: Make sure to approve the data channels to an additional environment where scoring pipeline will be deployed

• Consider a scenario where a data channel is deployed to Data-Channels environment.
• Now, when you attempt to deploy pipeline to the Development environment, you might encounter a Pipeline deployment failure, since its data channel is not accessible from the Development environment (only Data-Channels).

Deploying Data Channels

1. In the Project Explorer pane, select Deployments.
2. Click Deploy new and select Data channel from the drop-down list.
3. Add deployment name and description in the respective field.
4. Select the data channel and the space from the drop-down list.
5. Select Data-Channels environment from the drop-down list.
6. Check off the additional environments that you need to expose the data channel to.
7. Select when you need to schedule the job from the given options (Immediate, Future, or Periodic).
8. Select the duration for which you need to run the job. You can run the job forever or add the duration as per your needs.
9. Click Deploy.

Note: You might see failure in pipeline deployment if the data channel is not accessible from the environment in which the scoring pipeline is being deployed.

• Consider a scenario where a Data channel is deployed to Data-Channels environment with Test and Production as external environments.
• Now, when you attempt to deploy pipeline to the Development environment, you might encounter a Pipeline deployment failure, since its data channel is not accessible from the Development environment (only Data-Channels, Test, and Production)
• Make sure to approve and expose the data channel to the same environment as that of the scoring pipeline.

Viewing Deployed Data Channels

You can view status for deployed data channels by:

1. In the Project Explorer pane, select Deployments.
   • A list of deployed scoring pipelines and data channels appears here. Select Data channels filter to see the list of only data channels.
   • The list is sorted based on the deployed time in ascending order.
   • The list shows information such as time when the data channel was deployed, status of the deployed data channel, user by whom the data channel was deployed, deployment name, project, artifact, type of deployment, and environment.
   • Additional information is displayed on the right pane after clicking the individual data channel instance.