Open topic with navigation

TIBCO BusinessEvents® Enterprise Edition 6.3.1

# Kafka Channel

To send and receive messages from a Kafka broker, use the TIBCO BusinessEvents Kafka channel. The Kafka channel converts the incoming Kafka messages to BusinessEvents events and transforms BusinessEvents events as outgoing Kafka messages.

## Before You Begin

Before setting up the Kafka channel, go through the Kafka documentation for information (concepts, architecture, demos, APIs, and so on) about them. The following table lists the URLs that you can see for information about Kafka.

Channel Type | Documentation URL
--- | ---
Kafka | Kafka Documentation

## Kafka Channel Serializers

The Kafka channel provides the following serializers to handle payloads:

KafkaMapSerializer
The KafkaMapSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaMapSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) HashMap. The KafkaMapSerializer is used to send and receive events between BusinessEvents instances. For incoming messages, the serializer converts the byte sequence to the event and its payload. For outgoing events, the serializer converts the event and its payloads into HashMap as bytes.
KafkaJsonSerializer
The KafkaJsonSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaJsonSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) JSON. For incoming messages, the KafkaXmlSerializer decodes the text from the message as a JSON string and deserializes it to an event. For outgoing events, the serializer converts the event and its payloads into a JSON string. The KafkaXmlSerializer serializer is useful for processing or sending messages between BusinessEvents and external systems.
KafkaXmlSerializer
The KafkaXmlSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaXmlSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) a JSON string that does not have any qualifiers. For incoming messages, the KafkaXmlSerializer decodes the text from the message as an XML string and deserializes it to an event. For outgoing events, the serializer converts the event and its payloads into an XML string. The KafkaXmlSerializer serializer is useful for processing or sending messages between BusinessEvents and external systems.
KafkaTextPayloadSerializer
The KafkaTextPayloadSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaTextPayloadSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) text ignoring event properties. This serializer ignores the event properties even if they are defined. For incoming messages, the KafkaTextPayloadSerializer reads the text from the message and deserializes it to an event. For outgoing events, the serializer converts the event payload into a Kafka message value.
Note: If you are migrating a project from an earlier version that contains `KafkaStringPayloadSerializer` then the project still works in this version. However, the `KafkaStringPayloadSerializer` serializer is not available for a new Kafka channel.
KafkaAvroSerializer
The KafkaAvroSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaAvroSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) Avro data format. For incoming messages, the KafkaAvroSerializer reads the data from the Avro message and deserializes it to an event. For outgoing events, the serializer converts the event payload into an Avro message value.
KafkaProtobufSerializer
The KafkaProtobufSerializer serializer (`com.tibco.cep.driver.kafka.serializer.KafkaProtobufSerializer`) serializes and deserializes a BusinessEvents event along with its payload into (or from) Protobuf data format. For incoming messages, the KafkaProtobufSerializer reads the data from the message in Protobuf format and deserializes it to an event. For outgoing events, the serializer converts the event payload into a Protobuf message value.
Note: The Confluent schema registry supports the Protobuf data format. However, the TIBCO Schema Registry does not support this. For details about the Kafka Schema Registry, see Apache Kafka Schema Registry.

## Adding a New Channel

See the following topics for more information about the new Kafka channel:

- Adding a Kafka Channel in TIBCO BusinessEvents Application

## Sample TIBCO BusinessEvents Applications

The following sample TIBCO BusinessEvents application is provided with TIBCO BusinessEvents for Kafka channel to help you understand the channel better:

Channel Type | Sample TIBCO BusinessEvents Application
--- | ---
Kafka | BE_HOME/examples/standard/KafkaChannel

## Catalog Functions

You can send and receive events with payload in the Kafka channel using Event.sendEvent(), and Event.routeTo(). Currently, Kafka does not support synchronous request-reply of messages. Thus, the functions Event.requestEvent() and Event.replyEvent() are not supported.

## Kafka Channel Advanced Properties

You can specify Kafka channel advanced properties in the CDD file by adding appropriate prefixes to the channel. By using the same prefix, you can also override an existing Kafka channel property, which has been exposed through the TIBCO BusinessEvents Studio channel editor. For example, for specifying the `default.replication.factor` Kafka property, use `be.channel.kafka.default.replication.factor`.

Channel Type | Prefix | Description
--- | --- | ---
Kafka | `be.channel.kafka` | Used for Kafka properties that are applicable for the Kafka channel and all its destinations.
`be.channel.kafka.<destination_name>` | Used for Kafka properties that are applicable for only the specified destination of the Kafka channel.
`be.channel.kafka.<Channel URI>.<Kafka Property>=<value>` | Used to configure a multiple Kafka channels property. For example, To configure multiple Kafka Client JAAS configurations in a single be-engine, use the below syntax: `be.channel.kafka/Channels/KafkaConsumer.sasl.jaas.config=<value1>` `be.channel.kafka/Channels/KafkaProducer.sasl.jaas.config=<value2>` To set a property `default.replication.factor`, only for a particular destination, you can include a Destination URI instead of a Channel URI. `be.channel.kafka/Channels/KafkaChannel_1/dest.default.replication.factor=value1`

## Support for Kafka Message Parameters

The Kafka message parameters, such as key and offset, can be accessed through the TIBCO BusinessEvents event properties. You can define event properties for each Kafka parameter and use them in a rule or rulefunction. The deserializer fills the values in these event property fields when creating a TIBCO BusinessEvents event instance for a received Kafka message. For details, see Event Property and Rules and Functions.
The following properties can be added to events for Kafka message parameters.

Event Property | Description
--- | ---
`kafka_message_key` | Placeholder for a key of the Kafka message.
`kafka_message_offset` | Placeholder for an offset of the Kafka message.
`kafka_message_partition` | Placeholder for a partition number of the Kafka message.
`kafka_message_timestamp` | Placeholder for a time stamp of the Kafka message.
`kafka_message_timestamptype` | Placeholder for a type of the time stamp field of the Kafka message.

Subtopics

Did you find this helpful?

YesNo
Great!
Thanks for taking the time to give us some feedback.
Sorry about that

Why wasn't this helpful? (check all that apply)

Could not find what I was looking for
Instructions confusing or unclear
Instructions did not work

Want to tell us more?

Please select at least one option or provide details about the issue!!

Error occurred while submitting feedback.

Error2 occurred while submitting feedback.

TIBCO BusinessEvents® Enterprise Edition

Copyright © 2004-2024. Cloud Software Group, Inc. All Rights Reserved.