Tibco

Installing Data Channels

TIBCO® ModelOps supports two kinds of data channels: sources and sinks. ModelOps currently supports two types of data channels: File and Kafka channels. The Data Channels page provides detailed information about data channels. This page will provide further details about installing data channels and any prerequisite work involved.

Contents

Creating Data Channels

A data channel configuration describing the data channel has to be created before a data channel can be deployed. The ModelOps UI allows users to create data channel configurations. Data channel configuration includes data channel type, data channel schema, and other information specific to data channel type (topic and broker information for Kafka channels, file-encoding and other parser options for File channels). See Kafka Channels and File Channels for details.

Data channel creation example

Data channel configurations have to be approved for use in environments before data channels can be successfully deployed.

Data channel approval example

Deploying Data Channels

After creation and approval, data channel configurations can be deployed using the Deployments page. The Deployments page allows the user to: * provide a name for the deployment * select the configuration to be deployed * select environments for deployment * the schedule and duration for the deployment * data channel specific information (like Volume Name for File channels)

Data channel deployment example

Kafka Channels

Kafka channels connect to running instances of Apache Kafka® brokers and a specific topic to read or write messages. The broker and topic information must be provided on the channel configuration page where the data channel schema is also specified.

ModelOps supports the following broker string specifications based on the authentication type. Examples for each authentication type are provided below.

No Authentication

This is a very simple no authentication broker string containing the server URL including the port number.

example.com:9092
omnibus-01.srvs.cloudkafka.com:9094

Note: The ModelOps brokers option supports multiple broker strings placed one below the other.

PLAIN

A simple username/password authentication mechanism i.e. PLAIN, or SASL/PLAIN, is typically used with TLS for encryption to implement secure authentication. More details are available here, Configuring PLAIN.

An example of a PLAIN authentication type broker string for ModelOps is shown below.

test.com:9094[security.protocol=SASL_PLAINTEXT|sasl.mechanism=PLAIN|sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="********";]

SCRAM

Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, addresses the security concerns with traditional PLAIN user/password authentication mechanism. More details are available here, Configuring SCRAM.

An example of a SCRAM authentication type broker string for ModelOps is shown below.

test.com:9094[security.protocol=SASL_SSL|sasl.mechanism=SCRAM-SHA-256|sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="********";]

Note: The ModelOps broker strings for both PLAIN and SCRAM type authentication contains a list of pipe separated authentication strings to match the underlying Kafka Adapter Brokers property. Please refer to the Kafka Adapter properties for more details, Kafka Consumer Adapter

File Channels

File channels read data from, and write data to, CSV files. File data source/sink channel configurations contain information in addition to the common data channel properties like type and schema information.

For File Sources, the following additional properties are required:

  • File Encoding: encoding to be used for file read operations
  • Delimiter: character to be used as delimiter while parsing CSV records
  • Has Header Row: flag indicating the presence of a header row

For File Sinks, the following additional properties are required:

  • File Encoding: encoding to be used for file read operations
  • Quote Mode: defines quoting behavior when writing CSV records

File sources and sinks need “Volume Name” information to be provided during deployment. Persistent Volume Claims available in a given namespace are provided as options for the “Volume Name” field.

Persistent Volume Claims (PVCs) represent storage requests from a user. PVCs allow usage of abstract storage that a cluster administrator might have provisioned. PVCs consume storage resources through Persistent Volumes (PVs). PVs allow abstraction of storage provising and consumption.

PVs and PVCs together provide a mechanism that let File Sources and Sinks read from, and write to, files shares in various kinds of storage (including cloud-provider-specific storage systems). A File Source deployment instance will read data from a folder named <file-source-deployment-name>/<user-name> in the file share (through the specified PVC) and stream data to scoring pipelines deployed by <user-name> that connect to <file-source-deployment-name>. A File Sink deployment will write data being sent from scoring pipelines deployed by <user-name> that connect to <file-sink-deployment-name>.

Example: PV and PVC Setup

There are several ways to create PVs and PVCs for use with File Source and Sink deployments. The following examples demonstrates one way of creating them using Azure File Shares. The following example assumes that the following are available or true:

Step-1: Creating a Kubernetes Secret to access storage

The Azure storage account name and key can be stored as a Kubernetes Secret for use in the creation of persistent volumes. Create the following configuration file.

create-storage-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: <storage-secret-name>
type: Opaque
data:
  azurestorageaccountname: <base-64-encoded-account-name>
  azurestorageaccountkey: <base-64-encoded-account-key>

To use the create-storage-secret.yaml file, run the following command:

kubectl apply -f create-storage-secret.yaml --namespace datachannels

Step-2: Creating a persistent volume that maps to a file share

The next step is to create Kubernetes Persistent Volumes that map to the file shares created in Azure storage. Create the following configuration file (adapted from this example).

create-persistent-volume.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-for-source
  labels:
    usage: pv-for-source
  annotations:
    description: "A persistent volume for file sources"
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteMany
  azureFile:
    secretName: storage-secret-name
    shareName: data-file-share
    readOnly: false

To use the create-persistent-volume.yaml file, run the following command:

kubectl apply -f create-persistent-volume.yaml

Step-3: Creating a persistent volume claim that uses a persistent volume

The next step is to create Kubernetes Persistent Volume Claims that use Persistent Volumes. Create the following configuration file (adapted from this example).

create-persistent-volume-claim.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-for-source
  annotations:
    description: "A persistent volume claim for file sources"
    volume.beta.kubernetes.io/storage-class: ""
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  selector:
    matchLabels:
        usage: pv-for-source

To use the create-persistent-volume-claim.yaml file, run the following command:

kubectl apply -f create-persistent-volume-claim.yaml --namespace datachannels

pv-for-source will now become available as an option for the Volume Name field in Data Channel Deployments page.