Kafka Connect Standalone

Synopsis

This quickstart will show how to set up the IDoc Source Connector against an existing SAP NetWeaver® system and run it locally in Kafka Connect standalone mode using Apache Kafka. We will use a non encrypted communication channel with basic SAP® username/password authentication. In productive environments, it is recommended to use a SAP Secure Network Communication (SNC) setup with optional Single Sign On (SSO).

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy the IDoc source connector jar into the Kafka Connect plugins directory.
3. Get a copy of SAP® JCo v3.1.11 (sapjco3.jar and native lib sapjco3.dll or sapjco3.so) and copy it to the plugins directory.
4. Get a copy of SAP® Java IDoc library v3.1.3 (sapidoc3.jar) and copy it to the plugins directory.

Connector Configuration

1. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers=localhost:9092
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000
   plugin.path=<KAFKA_ROOT>/plugins
   

   Note

   Ensure that the plugin path exists.

2. Extract the properties with a minimal configuration for the IDoc Source Connector and copy this template to <KAFKA_ROOT>/config/idoc-source-connector.properties. Remember to add in your license key.

   # The first few settings are required for all connectors: a name, the connector class to run, and the maximum number of tasks to create:
   name = idoc-source-connector
   connector.class = org.init.ohja.kafka.connect.idoc.source.IDocSourceConnector
   tasks.max = 1
   sap.idoc.license.key = "Your license key here"
   # Execution interval in seconds for extraction of new IDoc files
   sap.idoc#00.exec-period = 900
   

3. Get in contact with your administration team for the connection properties of your SAP NetWeaver® installation and maintain the following minimum connector properties:

   # SAP Netweaver application host DNS or IP
   jco.client.ashost = 127.0.0.1
   # SAP system number
   jco.client.sysnr = 20
   # SAP client number
   jco.client.client = 100
   # SAP RFC user
   jco.client.user = user
   # SAP user password
   jco.client.passwd = password
   

   Note

   Make sure the SAP® user has enough privileges to access RFC-enabled function modules and the SAP NetWeaver® Gateway is configured to accept RFC connections from your host.

4. Maintain the target Kafka topic name, and the information for the IDoc of your choice:

   # SAP application server file path to use for querying generated IDoc files (flat file or XML)
   sap.idoc#00.file.path = /usr/sap/data/out
   # Regular expression filter applied to the file names
   sap.idoc#00.file.filter.regex = *.xml
   # IDoc file type (0=XML,1=flat file)
   sap.idoc#00.file.type = 0
   # Kafka output topic name
   sap.idoc#00.topic=IDOCTEST
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

2. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

3. Start a local Kafka server instance.

   bin\windows\kafka-server-start.bat config\kraft\server.properties
   

4. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic IDOCTEST --from-beginning
   

5. In another shell start a local standalone Kafka Connect instance and execute the IDoc Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\idoc-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt