Kafka Consumer Trigger

Apache Kafka Consumer Trigger receives records from specified topic in the Apache Kafka cluster.

Settings

On the Settings tab, you can define the Apache Kafka connection and its details as given in the following table:

Condition Applicable Field Description
N/A Apache Kafka Client Configuration Apache Kafka client configuration to be used.
N/A Topic The topic where Apache Kafka cluster stores streams of record.
Note: If you are using multiple topics, separate them using commas.
N/A Topic Whitelist (Optional) A whitelist of topics you can subscribe to for consuming messages from multiple topics. You can specify the group using a regex pattern. For example: 
mytopic*
N/A Consumer Group ID The group ID for the consumer group.
N/A Value Deserializer Select the type of record value to be received from the drop-down list: String or JSON
N/A Commit Interval The time interval in which a consumer offset commits to Apache Kafka.

Default Value: 5000 milliseconds

N/A Initial Offset Select one of the following options:
  • Newest: To start receiving published records since the consumer is started
  • Oldest: To start receiving records since the last commit
N/A Fetch Min Bytes Minimum size of data that server sends on fetch request.
N/A Fetch Max Wait

The maximum amount of time that the server would block before answering a fetch request if there is not sufficient data to immediately satisfy the requirement that you have configured in the Fetch Min Bytes field.

N/A Heartbeat Interval

Time in milliseconds to send heartbeats to consumer. Heartbeats are used to ensure that the consumer's session remains active and to facilitate rebalancing when consumers join or leave a group.

Note: Heartbeat interval must not be more than one-third of the session time.
N/A Session Timeout The consumer sends periodic heartbeats to server indicating about its liveness to the broker. If no heartbeats are received by a broker before the session times out, the broker removes this consumer from the group and initiates a rebalance.

Output Settings

Condition Applicable Field Description
N/A Headers Record headers to be received. Only String datatype value is supported.
Note: Headers are supported in the Apache Kafka version 0.11.0 and later.
Applicable only when JSON is selected in the Value Serializer field on the Settings tab. Schema for JSON value The JSON schema for the Apache Kafka record value

Output

Condition Applicable Field Description
Applicable only when JSON is selected in the Value Serializer field. jsonValue Complex data structure based on JSON schema that you have configured in the Output Settings section.
N/A partition Partition number of the record
N/A offset Offset of the record
N/A topic Name of the topic
N/A key Key value
Applicable only when String is selected in the Value Deserializer field on the Settings tab. stringValue String value to be received
N/A headers Header value to be received