Kafka Consumer Trigger

Apache Kafka Consumer Trigger receives records from specified topic in the Apache Kafka cluster.

Trigger Settings

On the Settings tab, you can define the Apache Kafka connection and its details as given in the following table:

Condition Applicable Field Description
N/A Apache Kafka Client Configuration Apache Kafka client configuration to be used.
N/A Topic The topic where Apache Kafka cluster stores streams of record.
Note: If you are using multiple topics, separate them using commas.
N/A Topic Whitelist (Optional) A whitelist of topics you can subscribe to for consuming messages from multiple topics. You can specify the group using a regex pattern. For example: 
mytopic*
N/A Consumer Group ID The group ID for the consumer group.
N/A Value Deserializer Select the type of record value to be received from the drop-down list:
  • String
  • JSON
  • Avro
Applicable only when Avro is selected in the Value Serializer field. Subject

A list of all registered subjects in your schema registry. A subject refers to the name under which a schema is registered.

Select the subject to be used.

Applicable only when Avro is selected in the Value Serializer field. Version Version of the subject (registered name of schema) registered.

Select the version of the subject to be used.

N/A Commit Interval The time interval in which a consumer offset commits to Apache Kafka.

Default Value: 5000 milliseconds

N/A Initial Offset Select one of the following options:
  • Newest: To start receiving published records since the consumer is started
  • Oldest: To start receiving records since the last commit
N/A Fetch Min Bytes Minimum size of data that server sends on fetch request.
N/A Fetch Max Wait

The maximum amount of time that the server would block before answering a fetch request if there is not sufficient data to immediately satisfy the requirement that you have configured in the Fetch Min Bytes field.

N/A Heartbeat Interval

Time in milliseconds to send heartbeats to consumer. Heartbeats are used to ensure that the consumer's session remains active and to facilitate rebalancing when consumers join or leave a group.

Note: Heartbeat interval must not be more than one-third of the session time.
N/A Session Timeout The consumer sends periodic heartbeats to server indicating about its liveness to the broker. If no heartbeats are received by a broker before the session times out, the broker removes this consumer from the group and initiates a rebalance.

Output Settings

Condition Applicable Field Description
N/A Headers Record headers to be received. Only String datatype value is supported.
Note: Headers are supported in the Apache Kafka version 0.11.0 and later.
Applicable only when JSON is selected in the Value Serializer field on the Trigger Settings tab. Schema for JSON value The JSON schema for the Apache Kafka record value
Applicable only when Avro is selected in the Value Serializer field on the Trigger Settings tab. Schema for Avro Value

The Avro schema for the Apache Kafka record value. Depending on the Subject and Version selected, the schema is displayed here.

Note: This field is read-only if Use Schema Registry in the Apache Kafka Client Configuration dialog box is set to True. Otherwise, you can provide the schema using this editor.

Output

Condition Applicable Field Description
Applicable only when JSON is selected in the Value Serializer field. jsonValue Data structure based on JSON schema that you have configured in the Output Settings section.
Applicable only when Avro is selected in the Value Serializer field. avroData Data structure based on Avro schema that you have configured in the Output Settings section.
N/A partition Partition number of the record
N/A offset Offset of the record
N/A topic Name of the topic
N/A key Key value
Applicable only when String is selected in the Value Deserializer field on the Settings tab. stringValue String value to be received
N/A headers Header value to be received