Kafka Connect Standalone

Synopsis

This quickstart guide will show you how to set up the OData Business Events Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka, extracting business event data from a SAP system. The publicly available business event object type used for this scenario is Sales Order. The corresponding BOR object type is BUS2032.

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy and extract the Business Events Source Connector package into the Kafka Connect plugins directory.

Connector Configuration

1. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers=localhost:9092
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000
   plugin.path=<KAFKA_ROOT>/plugins,
   

   Note

   Make sure the plugin path exists.

2. Extract the Business Events Source Connector properties with a minimal configuration and copy it to <KAFKA_ROOT>/config/busevent-source-connector.properties. Remember to include your license key.

   name = busevent-source-connector
   connector.class = org.init.ohja.kafka.connect.odata.busevent.source.ODataBusinessEventsSourceConnector
   tasks.max = 1
   sap.busevent.license.key = "Your license key here"
   sap.odata.host.address = services.odata.org
   sap.odata.host.port = 443
   sap.odata.host.protocol = https
   sap.odata.user.name = anonymous
   sap.odata.user.pwd = anonymous
   sap.busevent.subscriber.code = KBES
   sap.busevent#00.object.type = SalesOrder
   sap.busevent#00.object.task-code = Deleted,Created,Changed
   sap.busevent#00.topic = Order_Details
   sap.busevent#00.event.priority = 1
   sap.busevent#00.object.resolve = 0
   sap.busevent#00.service = /sap/opu/odata/sap/API_SALES_ORDER_SRV
   sap.busevent#00.entityset = A_SalesOrder
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Maintain subscriber code KBES in your SAP system, as described in section SAP subscriber configuration (in the doc/readme.md).

2. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

3. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

4. Start a local Kafka server instance.

   bin\windows\kafka-server-start.bat config\kraft\server.properties
   

5. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Order_Details --from-beginning
   

6. In another shell start a local standalone Kafka Connect instance and execute the Business Events Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\busevent-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

7. Execute transaction code SWUE in your SAP system and create a business event for object type BUS2032.

Switch to the Kafka consumer shell and if everything goes right you should see a JSON representation of the sales order business event messages together with its schema printed to the standard console output.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt