Confluent Platform

Synopsis

This section shows how to launch the Webservice Data Source Sink Connector on a Confluent Platform running locally within a docker environment.

Preliminary Setup

1. Make sure to have Docker Engine and Docker Compose installed on the machine where you want to run the Confluent Platform.
   1. Docker Desktop available for Mac and Windows includes both.
2. Download and launch a ready-to-go Confluent Platform Docker image as described in Confluent Platform Quickstart Guide.
3. Ensure that the machine where the Confluent Platform is running on has a connection to the SAP source.
4. Ensure that you have a licensed version of the SAP Java Connector 3.1 SDK available.

Connector Installation

The connector can be installed either manually or through the Confluent Hub Client.

Manual Installation

1. Unzip the zipped connector init-kafka-connect-webds-x.x.x.zip.
2. Get a copy of SAP Java Connector v3.1.11 SDK and move it to the lib/ folder inside the unzipped connector folder. SAP JCo consists of sapjco3.jar, and the native libraries like sapjco3.dll for Windows OS or sapjco3.so for Unix. Include e.g. the native lib sapjco3.so next to the sapjco3.jar.
3. Move the unzipped connector folder into the configured CONNECT_PLUGIN_PATH of the Kafka Connect service.

4. Within the directory where the docker-compose.yml of the Confluent Platform, is located you can start the Confluent Platform using Docker Compose.

   docker-compose up -d
   

Confluent Hub Client

Install the zipped connector init-kafka-connect-webds-x.x.x.zip using the Confluent Hub Client from outside the Kafka Connect docker container.

docker-compose exec connect confluent-hub install {PATH_TO_ZIPPED_CONNECTOR}/init-kafka-connect-webds-x.xx-x.x.x.zip

Connector Configuration

The connector can be configured and launched using the control-center service of the Confluent Platform.

1. In the control-center (default: localhost:9091) select a Connect Cluster in the Connect tab.

   

2. Click the button “Add connector” and select WebDSSinkConnector.

   

3. Enter a name and further required configuration for the connector.

To import data to the SAP® source using the Webservice Data Source Sink Connector, you need to follow these steps:

1. Transfer the minimal properties to the Confluent Control Center user interface. Remember to include your license key.

   name = webds-sink-connector
   connector.class = org.init.ohja.kafka.connect.webds.sink.WebDSSinkConnector
   tasks.max = 1
   sap.webds.license.key = "Your license key here"
   topics = ZWEBDSTEST
   sap.webds#00.system = TEST
   sap.webds#00.name = ZWEBDSTEST
   sap.webds#00.topic = ZWEBDSTEST
   jco.client.ashost = 127.0.0.1
   jco.client.sysnr = 20
   jco.client.client = 100
   jco.client.user = user
   jco.client.passwd = password
   

2. Launch the sink connector.
3. To test the sink connector, you need sample data.

Starting a Connector via REST Call

1. Save the example configuration JSON file into a local directory, e.g. named as sink.webds.json. Remember to include your license key.

   {
       "name": "webds-sink-connector",
       "config": {
           "connector.class": "org.init.ohja.kafka.connect.webds.sink.WebDSSinkConnector",
           "tasks.max": "1",
           "sap.webds.license.key": "Your license key here",
           "topics": "ZWEBDSTEST",
           "sap.webds#00.system": "TEST",
           "sap.webds#00.name": "ZWEBDSTEST",
           "sap.webds#00.topic": "ZWEBDSTEST",
           "jco.client.ashost": "127.0.0.1",
           "jco.client.sysnr": "20",
           "jco.client.client": "100",
           "jco.client.user": "user",
           "jco.client.passwd": "password"
       }
   }
   

2. Once the configuration JSON is prepared, you can start the connector by sending it via a REST call to the Kafka Connect REST API. Use the following command to send a POST request:

   curl -X POST http://localhost:8083/connectors \
   -H "Content-Type:application/json" \
   -H "Accept:application/json" \
   -d @sink.webds.json
   

3. Once the connector is started successfully, the Kafka Connect REST API will return a response in JSON format with details about the connector, including its status and any potential errors. You can verify that the connector is running by checking its status:

   curl -X GET http://localhost:8083/connectors/webds-sink-connector/status
   

   This will return a JSON object indicating whether the connector is running, its tasks, and any associated errors.