Page Comparison

Kafka messages sometimes remain unprocessed by a JMS source for unexplained reasons. To ensure that all the Kafka messages are consumed and published successfully, Adeptia Connect integrates Apache Kafka using which the application components can create, send, receive, and read messages using reliable communication.

To use Apache Kafka, you need to create a Kafka account and define a Kafka listener and a target. Kafka messages can be consumed and published by a template, transaction, or process flow. 

...

1. Click Configure > EVENTS > Kafka Listener.
   
2. Click Create Kafka Listener.
3. In Create Kafka Listener window, do the following:
   
   
   
   
   1. In the Name and Description fields, enter the name and description respectively for the Kafka listener.
      

   2. In the Kafka Account field, select the Kafka account. You can create a new Kafka account by clicking .
      

      Note
      Click Test to check if you are able to connect with Kafka server using this Kafka account.

      
      

   3. Anchor
      StepC StepC

      In the Topic(s) field, select the topic to be used. 

      Tip
      You can select multiple topics.

      
      

   4. In the Consumer Group field, enter the string that uniquely identifies the group of consumer processes to which this consumer belongs. By setting the same consumer group multiple processes indicate that they are all part of the same consumer group.
   5. In the Key(s) to Filter Messages field, enter the Name of the message key to process. You can use comma to provide multiple message keys.
   6. In the Consumer Count field, enter the number of consumers that connect to Kafka server.
   7. In the Process Flow Name field, select the process flow.
   8. In the Auto Offset Reset field, select the value to define the offset when there is no initial offset in ZooKeeper or if an offset is out of range:
      
      • EARLIEST : Automatically reset the offset to the earliest offset
      • LATEST : Automatically reset the offset to the latest offset fail: throw exception to the consumer.
      • NONE
   9. Select the Enable Message Aggregator check box to define the message aggregator properties:
      1. In the Number of Messages for Aggregation field, enter the number of message should be aggregated.
      2. In the Aggregation Timespan field, enter the time in seconds, messages received within the provided timespan will be aggregated.

      3. In the Message Type field, select the message format.
         

         Tip
         The supported message formats are JSON, XML, and PLAINTEXT.

         
         

      4. In the Message Joiner field, enter the delimiter based on which the messages will be aggregated.
         It depends on the Message Type.
         
      • • XML : It will be an XPath.
        • PLAINTEXT : It will be a delimiter.
        • JSON : It can be empty or an element name.
   10. In the Seek To field, select if the Kafka Consumer will read the message from beginning or end on startup.
       
   11. In the Heartbeat Interval field, enter the heartbeat interval.
       The expected time between heartbeats to the consumer coordinator when using Kafka group management facilities. Heartbeats are used to ensure that the consumers session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session timeout, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.
   12. In the Session Timeout field, enter the time interval for session timeout.
       The timeout used to detect failures when using Kafka group management facilities.
   13. In the Select Project field, select the project.

   14. Click Save.

       Note
       Kafka Listener and Target created from the manage page are available to be used only in the process flow. Every time you create a template with Kafka as a source, you need to define the properties for the Kafka listener. The same holds true when you create a template with Kafka as a destination.

       
       

Activating or deactivating the Kafka Listener

You can activate or deactivate a Kafka Listener by clicking      Image Added on the Kafka listener manage page.

• Once activated, the listener will start consuming the messages from Kafka Server and trigger the process flow for the received message.
• Once deactivated, the listener will stop receiving Kafka messages from server. 

Anchor
Creating Kafka Target Creating Kafka Target

Creating Kafka Target

...

1. Select Kafka as a source application.
2. In the Which event should trigger the data transfer? field, select New Message and click Next.
3. In the Provide Kafka account information field, select the Kafka account and click Next.
4. In the Kafka Settings page, follow the instructions from the step c through step l from the section Creating Kafka Listener.
5. Click Next to complete the steps that follow to create the template. 
   

Publishing Kafka message to a Kafka server

...

1. Select Kafka as a destination application.
2. In the Provide Kafka account information field, select the Kafka account and click Next.
3. In the How the data should be delivered to destination? field, select Send Message and click Next.
4. In the Kafka Settings page, follow the instructions from the step c through step l from the section Creating Kafka Target.
5. Click Next to complete the steps that follow to create the template.