Kafka and the Kafka Streams Channel
To send and receive messages from a Kafka broker, use the TIBCO BusinessEvents Kafka channel. The Kafka channel converts the incoming Kafka messages to BusinessEvents events and transforms BusinessEvents events as outgoing Kafka messages.
Before You Begin
Before setting up the Kafka or the Kafka Streams channel, go through the Kafka documentation for information (concepts, architecture, demos, APIs, and so on) about them. The following table lists the URLs that you can see for information about Kafka and Kafka Streams.
Channel Type | Documentation URL |
---|---|
Kafka | Kafka Documentation |
Kafka Streams | Kafka Streams Documentation |
Kafka Channel Serializers
- KafkaMapSerializer
- The KafkaMapSerializer serializer (com.tibco.cep.driver.kafka.serializer.KafkaMapSerializer) serializes and deserializes a BusinessEvents event along with its payload into (or from) HashMap. The KafkaMapSerializer is used to send and receive events between BusinessEvents instances. For incoming messages the serializer converts the bytes sequence to event and its payload. For outgoing events the serializer converts the event and its payloads into HashMap as bytes.
- KafkaJsonSerializer
- The KafkaJsonSerializer serializer (com.tibco.cep.driver.kafka.serializer.KafkaJsonSerializer) serializes and deserializes a BusinessEvents event along with its payload into (or from) JSON. For incoming messages, the KafkaXmlSerializer decodes the text from the message as an JSON string and deserializes it to an event. For outgoing events, the serializer converts the event and its payloads into an JSON string. The KafkaXmlSerializer serializer is useful for processing or sending messages between BusinessEvents and external systems.
- KafkaXmlSerializer
- The KafkaXmlSerializer serializer (com.tibco.cep.driver.kafka.serializer.KafkaXmlSerializer) serializes and deserializes a BusinessEvents event along with its payload into (or from) XML. For incoming messages, the KafkaXmlSerializer decodes the text from the message as an XML string and deserializes it to an event. For outgoing events, the serializer converts the event and its payloads into an XML string. The KafkaXmlSerializer serializer is useful for processing or sending messages between BusinessEvents and external systems.
Kafka Streams Channel Serializers
The Kafka Streams channel provides all the serializers that are present in the Kafka channel plus the KeyValueSerializer.
- KeyValueSerializer
- KeyValueSerializer (com.tibco.cep.driver.kafkastreams.serializer.KeyValueSerializer) deserializes a Kafka Streams record (with the primitive key and value types) into a SimpleEvent (set as Default Event). The event must have two properties with names RECORD_KEY and RECORD_VALUE with the data type matching the key and value of the stream record. For example, output of the Count transformation can be deserialized into an event with RECORD_KEY as String and RECORD_VALUE as Long. The KeyValueSerializer is used to receive events in the TIBCO BusinessEvents instances.
Adding a New Channel
Sample TIBCO BusinessEvents Applications
The following sample TIBCO BusinessEvents applications are provided with TIBCO BusinessEvents for Kafka and Kafka Streams channels to help you understand these channels better:
Catalog Functions
You can send and receive events with payload in Kafka channel using Event.sendEvent(), and Event.routeTo(). Currently, Kafka does not support synchronous request-reply of messages; thus, the functions Event.requestEvent() and Event.replyEvent() are not supported.
Kafka and Kafka Streams Properties
You can specify Kafka and Kafka Streams properties in the CDD file by adding appropriate prefixes to the channel. By using the same prefix, you can also override an existing Kafka and Kafka Streams property, which has been exposed through the TIBCO BusinessEvents Studio channel editor. For example, for specifying the default.replication.factor Kafka property, use be.channel.kafka.default.replication.factor.Channel Type | Prefix | Description |
---|---|---|
Kafka | be.channel.kafka | Used for Kafka properties that are applicable for the Kafka channel and all its destinations. |
be.channel.kafka.<destination_name> | Used for Kafka properties that are applicable for only the specified destination of the Kafka channel. | |
Kafka Streams | be.channel.kafka.streams | Used for Kafka Streams properties that are applicable for the Kafka Streams channel and all its destinations. |
be.channel.kafka.streams.<destination_name> | Used for Kafka Streams properties that are applicable for the specified destination of the Kafka Streams channel. |
- Adding a Kafka Channel in TIBCO BusinessEvents Application
Configure channel properties and destination properties for Kafka channel to successfully add the Kafka channel to the BusinessEvents application. - Adding a Kafka Streams Channel in TIBCO BusinessEvents Application
To process incoming Kafka Streams records in your TIBCO BusinessEvents application, set up a Kafka Streams channel in your application. - Setting Up Authentication for Kafka
You can set up authentication and SSL to improve security between BusinessEvents and Kafka broker.