Using the Apache Kafka Schema Registry

You can use Schema Registry to validate Kafka messages against registered schema. Schema Registry ensures data consistency and increases data quality. Schema Registry supports schema versioning so that the different versions of the schema can be used simultaneously without causing compatibility issues. TIBCO BusinessEvents supports Confluent Schema Registry and TIBCO Schema Registry.

Before you begin

Configuring the Schema Registry

For information about configuring the Schema Registry, see Kafka Destination Configuration Properties.

Subject Naming Strategies

Confluent Schema Registry

When the schema is registered, the name of the subject is decided by subject name strategy. Subject name strategy can be set using Kafka destination properties. By default the subject name strategy is TopicNameStrategy. Subject name strategy can be configured by using the following property:

be.channel.kafka/Channels/<channelname>/<destinationname>.value.subject.name.strategy=<Subject Name Strategy Class>

You can set one of the following subject name strategy classes:

  • io.confluent.kafka.serializers.subject.TopicNameStrategy

  • io.confluent.kafka.serializers.subject.RecordNameStrategy

  • io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

TIBCO Schema Registry

For the TIBCO Schema Registry, you can set the subject name by using the property value.subject.name with a few "%" template variables. Default value for this property is %t-value.

The application automatically replaces the special characters %t, %r, and %% with topic, name of schema, and literal % character respectively.

You can set the value for the value.subject.name as TIBCO BusinessEvents property for each destination.

Example:

be.channel.kafka/Channels/KafkaChannel/orderDest.value.subject.name=%r-%t-value

The preceding property configures value.subject.name for "orderDest" destination. The subject name is "RecordName-TopicName-Value".

For more information, see Apache Avro Client Library Configuration Options.

Schema Compatibility

When schemas evolve and change, you must add (register) different schema versions. Before adding a version of a schema, the Schema Registry checks whether the new schema is compatible with the previously registered schema under the subject. This compatibility is managed through the Schema Registry compatibility parameter. For more information about setting schema compatibility, see Confluent documentation.

Schema Registration Scenarios

Refer to the following table for different scenarios about how schema is registered in the Schema Registry based on whether Subject and Schema fields are specified or not.

Note: You must set appropriate schema compatibility type before registering new schema.
Subject field specified? Schema field specified? How Schema is registered?
Yes Yes

If the schema subject is valid, TIBCO BusinessEvents fetches the latest schema from the Schema Registry associated with the subject. If the subject name is not as per the subject name strategy, then TIBCO BusinessEvents registers the latest schema under subject that is as per the subject name strategy.

If the schema subject is not valid, TIBCO BusinessEvents registers the schema configured in the Kafka destination properties with the subject name as per the subject name strategy. If schema is already registered, then TIBCO BusinessEvents updates the schema in the Schema Registry.
No Yes Registers the schema configured in the Kafka destination properties. If schema is already registered then TIBCO BusinessEvents updates the schema in the Schema Registry.
Yes No

If the schema subject is valid, TIBCO BusinessEvents fetches the latest schema from the Schema Registry associated with the subject. If an associated subject name is not as per the subject name strategy, then TIBCO BusinessEvents registers the latest schema under subject that is as per the subject name strategy.

If schema subject is not valid, TIBCO BusinessEvents creates schema using event properties and payload associated with the destination and registers it as per the subject name strategy
No No If schema subject is not valid, TIBCO BusinessEvents creates schema by using event properties and payload associated with the destination and registers it as per the subject name strategy.
Note: Confluent Schema Registry always sets subject name as per subject name strategy. TIBCO Schema Registry sets subject as per % template variables provided in value.subject.name.
Note: If the Update Schema in Schema Registry checkbox is selected, then Schema and Subject field scenarios are not considered. In this case, TIBCO BusinessEvents creates the schema by using event properties and the payload associated with the destination and registers it as per the subject name strategy.