Kafka Connect Standalone

Synopsis

This quickstart will show how to set up the Webservice Data Source Sink Connector against an existing SAP Netweaver system and run it locally in Kafka Connect standalone mode using Apache Kafka. We will use a non encrypted communication channel with basic SAP username/password authentication. In productive environments, it is recommended to use a SAP® Secure Network Communication (SNC) setup with optional Single Sign On (SSO).

Preliminary Setup

Download and extract Apache Kafka.
Copy the WebDS sink connector jar into the Kafka Connect plugins directory.
Get a copy of SAP JCo v3.1.11 (sapjco3.jar and native lib sapjco3.dll or sapjco3.so) and copy it to the plugins directory.

Connector Configuration

Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=<KAFKA_ROOT>/plugins

Note

Ensure that the plugin path exists.

Extract the WebDS Sink Connector properties with a minimal configuration and copy it to <KAFKA_ROOT>/config/webds-sink-connector.properties. Remember to include your license key.

name = webds-sink-connector
connector.class = org.init.ohja.kafka.connect.webds.sink.WebDSSinkConnector
tasks.max = 1
sap.webds.license.key = "Your license key here"
topics = ZWEBDSTEST

Get in contact with your administration team for the connection properties of your SAP Netweaver installation and maintain the following minimum connector properties.
```
# SAP Netweaver application host DNS or IP
jco.client.ashost = 127.0.0.1
# SAP system number
jco.client.sysnr = 20
# SAP client number
jco.client.client = 100
# SAP RFC user
jco.client.user = user
# SAP user password
jco.client.passwd = password
```
Note

Make sure the SAP user has enough privileges to access RFC-enabled function modules and the SAP Netweaver Gateway is configured to accept RFC connections from your host.
Maintain the input Kafka topic name, and the following connector configs according to the webservice data source of your choice in SAP.
```
sap.webds#00.system = WEBDS
sap.webds#00.name = ZTEST
# Kafka input topic name
sap.webds#00.topic = ZWEBDSTEST
```

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

cd <KAFKA_ROOT>
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties

Format the Kafka storage directory from the shell using a random cluster ID.

bin\windows\kafka-storage.bat random-uuid

bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties

Start a local Kafka server instance.
bash bin\windows\kafka-server-start.bat config\kraft\server.properties
In another shell start a local standalone Kafka Connect instance and execute the WebDS Sink Connector.
```
bin\windows\connect-standalone.bat config\connect-standalone.properties config\webds-sink-connector.properties > log.txt 2>&1
```
The logging outputs will be written to the file log.txt.

Start a standalone Kafka producer from another shell.

bin\windows\kafka-console-producer.bat --bootstrap-server localhost:9092 --topic ZWEBDSTEST

Now you can enter messages that will be put into topic ZWEBDSTEST as source topic for the connector. You only have to enter the value part of the messages, e.g.

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"ZCONTRACT","field":"_BIC_ZCONTRACT"},{"type":"int32","optional":true,"name":"org.apache.kafka.connect.data.Date","version":1,"doc":"ZKEYDATE","field":"_BIC_ZKEYDATE"},{"type":"bytes","optional":false,"name":"org.apache.kafka.connect.data.Decimal","version":1,"doc":"ZCONTRVAL","field":"_BIC_ZCONTRVAL"},{"type":"string","optional":false,"doc":"Currency Key","field":"CURRENCY"}],"name":"ES_DATA"},"payload":{"_BIC_ZCONTRACT":"100","_BIC_ZKEYDATE":16647,"_BIC_ZCONTRVAL":4923.8,"CURRENCY":"EUR"}}

Switch to the SAP GUI and monitor the webservice data source to check for new data, see Monitoring. If no data arrives at SAP you should check log.txt for any errors.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt

1.3.6