Kafka Connect Standalone

Synopsis

This quickstart guide will show how to set up the ODP Source Connector against an existing SAP NetWeaver® system and run it locally in Kafka Connect standalone mode using Apache Kafka. We will use a non encrypted communication channel with basic SAP® username/password authentication. In productive environments it is recommended to use a SAP® Secure Network Communication (SNC) setup with optional Single Sign On (SSO).

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy the ODP Source Connector jar into the Kafka Connect plugins directory.
3. Get a copy of SAP® JCo v3.1.11 (sapjco3.jar and native lib sapjco3.dll or sapjco3.so) and copy it to the plugins directory.

Connector Configuration

1. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers=localhost:9092
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000
   plugin.path=<KAFKA_ROOT>/plugins
   

   Note

   Ensure that the plugin path exists.

2. Extract the properties template for the ODP Source Connector with a minimal configuration and copy it to <KAFKA_ROOT>/config/odp-source-connector.properties. Remember to add in your license key.

   name = odp-source-connector
   connector.class = org.init.ohja.kafka.connect.odp.source.ODPSourceConnector
   tasks.max = 1
   sap.odp.license.key = "Your license key here"
   sap.odp.subscriber-name = OhJaODPKafkaConnector
   sap.odp#00.exec-period = 900
   sap.odp#00.package-size = 52428800
   

3. Get in contact with your administration team for the connection properties of your SAP NetWeaver® installation and maintain the following minimum connector properties.

   # SAP Netweaver application host DNS or IP
   jco.client.ashost = 127.0.0.1
   # SAP system number
   jco.client.sysnr = 20
   # SAP client number
   jco.client.client = 100
   # SAP RFC user
   jco.client.user = user
   # SAP user password
   jco.client.passwd = password
   

   Note

   Make sure the SAP® user has enough privileges to access RFC-enabled function modules and the SAP NetWeaver® Gateway is configured to accept RFC connections from your host.

4. Maintain the output Kafka topic name, and the following connector configs according to the delta enabled ODP data source of your choice in SAP®.

   sap.odp#00.name = Test
   sap.odp#00.context = SAPI
   # Delta-Initialization without data transfer (0=false,1=true(default))
   sap.odp#00.init-simulation = 0
   # Kafka output topic name
   sap.odp#00.topic = ODPSAPITEST
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

2. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

3. Start a local Kafka server instance.
   bash bin\windows\kafka-server-start.bat config\kraft\server.properties

4. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ODPSAPITEST --from-beginning
   

5. In another shell start a local standalone Kafka Connect instance and execute the ODP Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\odp-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

To view the JSON representations of the ODP data messages along with their schema, switch to the Kafka consumer shell. If everything is set up correctly, you should see the messages being printed out to the console output in JSON format.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt