Kafka Listener

Apache Kafka is a distributed streaming platform that is highly scalable and secure, and it can:

• Consume and publish messages/event streams, similar to an enterprise messaging system.
• Store the messages for as long as you want.
• Process the messages/event streams as they occur.

Creating Kafka Listener

Follow the steps below to create Kafka listener:

1. Click Configure > EVENTS > Kafka Listener.
   
2. Click Create Kafka Listener.
3. In Create Kafka Listener window, do the following:
   
   
   
   
   1. In the Name and Description fields, enter the name and description respectively for the Kafka listener.
      

   2. In the Kafka Account field, select the Kafka account. You can create a new Kafka account by clicking .
      

      Click Test to check if you are able to connect with Kafka server using this Kafka account. 

   3. In the Topic(s) field, select the topic to be used. 

      You can select multiple topics.
   4. In the Consumer Group field, enter the string that uniquely identifies the group of consumer processes to which this consumer belongs. By setting the same consumer group multiple processes indicate that they are all part of the same consumer group.
   5. In the Key(s) to Filter Messages field, enter the Name of the message key to process. You can use comma to provide multiple message keys.
   6. In the Consumer Count field, enter the number of consumers that connect to Kafka server.
   7. In the Process Flow Name field, select the process flow.
   8. In the Auto Offset Reset field, select the value to define the offset when there is no initial offset in ZooKeeper or if an offset is out of range:
      
      • EARLIEST : Automatically reset the offset to the earliest offset
      • LATEST : Automatically reset the offset to the latest offset fail: throw exception to the consumer.
      • NONE
   9. Select the Enable Message Aggregator check box to define the message aggregator properties:
      1. In the Number of Messages for Aggregation field, enter the number of message should be aggregated.
      2. In the Aggregation Timespan field, enter the time in seconds, messages received within the provided timespan will be aggregated.

      3. In the Message Type field, select the message format.
         

         The supported message formats are JSON, XML, and PLAINTEXT.

      4. In the Message Joiner field, enter the delimiter based on which the messages will be aggregated.
         It depends on the Message Type.
         
      • • XML : It will be an XPath.
        • PLAINTEXT : It will be a delimiter.
        • JSON : It can be empty or an element name.
   10. In the Seek To field, select if the Kafka Consumer will read the message from beginning or end on startup.
       
   11. In the Heartbeat Interval field, enter the heartbeat interval.
       The expected time between heartbeats to the consumer coordinator when using Kafka group management facilities. Heartbeats are used to ensure that the consumers session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session timeout, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.
   12. In the Session Timeout field, enter the time interval for session timeout.
       The timeout used to detect failures when using Kafka group management facilities.

   13. Expand Advanced Properties, and enter the email id(s) separated by commas in the Notification Receiver Email-ID(s) field to receive notifications in case of Kafka Listener failure.

       If you want the Kafka Listener to be deactivated after its failure for a specified number of retries, you need to set the value for the property abpm.event.abort.retryCount available in the server-configure.properties file to that specific number. For example, if you want the Kafka Listener to be deactivated after thirteen (13) retries, set the value for the property abpm.event.abort.retryCount to 13. The user gets an additional notification about the deactivation of the Kafka Listener on the email id set in the Notification Receiver Email-ID(s) field.

   14. In the Select Project field, select the project.

   15. Click Save.

       • Kafka Listener and Target created from the manage page are available to be used only in the process flow. Every time you create a template with Kafka as a source, you need to define the properties for the Kafka listener. The same holds true when you create a template with Kafka as a destination. 
       • After you create a Kafka Listener, you may need to configure the property abpm.services.kafka.messages.location to specify the location where you want to save the Kafka messages. Defining this property is a one time activity – you need not define this property every time you create a Listener.

Activating or deactivating the Kafka Listener

You can activate or deactivate a Kafka Listener by clicking  on the Kafka listener manage page.

• Once activated, the listener will start consuming the messages from Kafka Server and trigger the process flow for the received message.
• Once deactivated, the listener will stop receiving Kafka messages from server.