Using the Apache Kafka Schema Registry
You can use Schema Registry to validate Kafka messages against registered schema. Schema Registry ensures data consistency and increases data quality. Schema Registry supports schema versioning so that the different versions of the schema can be used simultaneously without causing compatibility issues. TIBCO BusinessEvents supports Confluent Schema Registry and TIBCO Schema Registry.
-
For information about installation and concepts of Schema Registry, see the following documentation:
Configuring the Schema Registry
For information about configuring the Schema Registry, see Kafka Destination Configuration Properties.
Subject Naming Strategies
Confluent Schema Registry
When the schema is registered, the name of the subject is decided by subject name strategy. Subject name strategy can be set using Kafka destination properties. By default the subject name strategy is TopicNameStrategy. Subject name strategy can be configured by using the following property:
be.channel.kafka/Channels/<channelname>/<destinationname>.value.subject.name.strategy=<Subject Name Strategy Class>
You can set one of the following subject name strategy classes:
-
io.confluent.kafka.serializers.subject.TopicNameStrategy
-
io.confluent.kafka.serializers.subject.RecordNameStrategy
-
io.confluent.kafka.serializers.subject.TopicRecordNameStrategy
TIBCO Schema Registry
For the TIBCO Schema Registry, you can set the subject name by using the property value.subject.name
with a few "%" template variables. Default value for this property is %t-value
.
The application automatically replaces the special characters %t, %r, and %% with topic, name of schema, and literal % character respectively.
You can set the value for the value.subject.name
as TIBCO BusinessEvents property for each destination.
Example:
be.channel.kafka/Channels/KafkaChannel/orderDest.value.subject.name=%r-%t-value
The preceding property configures value.subject.name
for "orderDest" destination. The subject name is "RecordName-TopicName-Value".
For more information, see Apache Avro Client Library Configuration Options.
Schema Compatibility
When schemas evolve and change, you must add (register) different schema versions. Before adding a version of a schema, the Schema Registry checks whether the new schema is compatible with the previously registered schema under the subject. This compatibility is managed through the Schema Registry compatibility parameter. For more information about setting schema compatibility, see Confluent documentation.
Schema Registration Scenarios
Refer to the following table for different scenarios about how schema is registered in the Schema Registry based on whether Subject and Schema fields are specified or not.
Subject field specified? | Schema field specified? | How Schema is registered? |
---|---|---|
Yes | Yes |
If the schema subject is valid, TIBCO BusinessEvents fetches the latest schema from the Schema Registry associated with the subject. If the subject name is not as per the subject name strategy, then TIBCO BusinessEvents registers the latest schema under subject that is as per the subject name strategy. If the schema subject is not valid, TIBCO BusinessEvents registers the schema configured in the Kafka destination properties with the subject name as per the subject name strategy. If schema is already registered, then TIBCO BusinessEvents updates the schema in the Schema Registry. |
No | Yes | Registers the schema configured in the Kafka destination properties. If schema is already registered then TIBCO BusinessEvents updates the schema in the Schema Registry. |
Yes | No |
If the schema subject is valid, TIBCO BusinessEvents fetches the latest schema from the Schema Registry associated with the subject. If an associated subject name is not as per the subject name strategy, then TIBCO BusinessEvents registers the latest schema under subject that is as per the subject name strategy. If schema subject is not valid, TIBCO BusinessEvents creates schema using event properties and payload associated with the destination and registers it as per the subject name strategy |
No | No | If schema subject is not valid, TIBCO BusinessEvents creates schema by using event properties and payload associated with the destination and registers it as per the subject name strategy. |
value.subject.name.