Apache Kafka is a distributed streaming platform that is highly scalable and secure, and it can:
...
Click Configure > ACCOUNTS > Kafka.
- Click Create Kafka account.
- In Create Kafka Account window, do the following:
- In the Name and Description fields, enter the name and description respectively for the new Kafka account.
- In the Broker field, enter the URL of the Kafka brokers, foe example, host1:port1,host2:port2.
In the Security Protocol field, enter the protocol used to communicate with brokers.
Tip The supported protocols are PLAINTEXT and SSL.
For SSL:
In case you are using SSL for client authentication, select SSL in the Security Protocol field and then select a Keystore in the Select Keystore field.
In case you are using SSL for server authentication, you must import the Kafka server certificate in Adeptia Connect Truststore. To import the certificate, click here.
In the Addon Configuration window, enter the Add On Configurations to be used in back end for performing operation on Kafka server (e.g. param1=value,param2=value..).
Note Here you can use properties which are not listed on this Kafka account creation interface, for example, you may need to use a property called Serializer that converts the data to bytes. You can find the properties available at this location. - In the Select Project field, select the project.
- Click Save.
Anchor
...
Follow the steps below to create Kafka listener:
...
In the Kafka Account field, select the Kafka account. You can create a new Kafka account by clicking 
.
| Note |
|---|
| Click Test to check if you are able to connect with Kafka server using this Kafka account. |
...
| Tip |
|---|
| You can select multiple topics. |
...
- EARLIEST : Automatically reset the offset to the earliest offset
- LATEST : Automatically reset the offset to the latest offset fail: throw exception to the consumer.
- NONE
...
In the Message Type field, select the message format.
...
...
- XML : It will be an XPath.
- PLAINTEXT : It will be a delimiter.
- JSON : It can be empty or an element name.
...
Click Save.
| Note |
|---|
|
...
You can activate or deactivate a Kafka Listener by clicking 
on the Kafka listener manage page.
| Note |
|---|
| Activate or Deactivate options will be available only for the Kafka Listeners that have been created from the manage page and published as a process flow. |
- Once activated, the listener will start consuming the messages from Kafka Server and trigger the process flow for the received message.
- Once deactivated, the listener will stop receiving Kafka messages from server.
...
Follow the steps below to create Kafka target:
...
In the Kafka Account field, select the Kafka account.
...
In the Message Type field, select the message format.
| Tip |
|---|
The supported message formats are JSON, XML, and PLAINTEXT. |
...
In the Message Splitter field, enter the delimiter based on which the messages will be splitted.
It depends on the Message Type.
- XML : It will be an element name of the record.
- PLAINTEXT : It will be a delimiter.
- JSON: It can be empty or an element name.
...
In the Request Required Acks field, select the number of acknowledgments the producer requires the leader to have received before considering a request complete.
This controls the durability of records that are sent. The following settings are common:
if set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.
if set to 1 then this will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.
if set to all then this means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee.
...
In the Request Timeout field, enter the amount of time the broker will wait trying to meet the RequestRequiredAcks requirement before sending back an error to the client.
...
In the Number of Retries field, enter the value.
Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two records are sent to a single partition, and the first fails and is retried but the second succeeds, then the second record may appear first.
...
In the Select Project field, select the project
...
Click Save.
Anchor
Configuring Kafka properties Configuring Kafka properties
Configuring Kafka properties
| Configuring Kafka properties | |
| Configuring Kafka properties |
To enable proper processing of messages in Kafka, you may need to set the relevant properties. There are two properties that govern the storage and splitting of the Kafka messages. To update the properties, follow the steps given below:
- Click Account > Settings.
- Expand the Server Node Settings in the left panel.
- Select the server node.
- Click Edit.
- Click
to expand Kafka Configuration. - Edit the following properties.
abpm.services.kafka.messages.location
Enter the location where you want to save the Kafka messages.Info In case of cluster environment the property abpm.services.kafka.messages.location needs to be updated to be able to read data from Kafka. The property value has to be the absolute path of shared drive till Attachments folder inside shared drive. - abpm.services.kafka.target.special.characters
Enter the set of special characters based on which the message will be split into separate records in Kafka.
- Click Save.
Using SASL security protocol in Kafka
Adeptia Connect supports Simple Authentication and Security Layer (SASL) security protocol in Kafka. SASL security protocol can be used by defining a set of properties in the Addon Configuration field. These properties are set based on the protocol (PLAINTEXT or SSL) that you have selected in the Security Protocol field.
Follow the steps below to use SASL protocol in Kafka:
- Select the security protocol in the Security Protocol field based on your Kafka server setting.
Note: The supported protocols are PLAINTEXT and SSL. - Based on the protocol you have selected, define the SASL properties in the Addon Configuration field as explained in the example below:
Note: Define each property in a new line.
This is an example in case you have selected PLAINTEXT as a security protocol.
saslMechanism=PLAIN
securityProtocol=SASL_PLAINTEXT
saslJaasConfig= org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
The table below describes the properties and their values:
Property key | Description |
saslMechanism | The SASL mechanism used. For the valid values, click here. |
securityProtocol | Protocol used to communicate with brokers. SASL_PLAINTEXT (In case you have selected PLAINTEXT as a security protocol) or SASL_SSL (In case you have selected SSL as a security protocol). |
saslJaasConfig | Expose the Kafka sasl.jaas.config parameter, for example, org.apache.kafka.common.security.plain.PlainLoginModule required username=USERNAME password=PASSWORD; |
Using Kafka in a process flow
When you use a process flow in a Kafka listener with a process flow, the the File Source service that you have in the process flow starts receiving the messages when the listener is activated. The only condition to be met here is that the Event Context Enabled property for that source service should be enabled while creating the process flow.
| Note |
|---|
| If you want to send a message to Kafka, you can use a Kafka target in the process flow. |
Perform the following steps to enable the Event Context Enabled property in a process flow:
- Select the Source service and then click View Properties option from the context pad menu.
- In the Activity Properties panel, under the GENERAL tab, enable the Event Context Enabled property.
- Save the process flow.
- Activate the process flow.
Once you save the process flow with this setting and activate it, you can choose it in the Process Flow Name field while creating a Kafka listener service.
| Tip |
|---|
| Ensure that you have activated the Kafka listener service. Click here to know how to activate it. |
...
- Select Kafka as a source application.
- In the Which event should trigger the data transfer? field, select New Message and click Next.
- In the Provide Kafka account information field, select the Kafka account and click Next.
- In the Kafka Settings page, follow the instructions from the step c through step i from the section Creating Kafka Listener.
- Click Next to complete the steps that follow to create the template.
Publishing Kafka message to a Kafka server
...
- Select Kafka as a destination application.
- In the Provide Kafka account information field, select the Kafka account and click Next.
- In the How the data should be delivered to destination? field, select Send Message and click Next.
- In the Kafka Settings page, follow the instructions from the step c through step i from the section Creating Kafka Target.
- Click Next to complete the steps that follow to create the template.