Kafka Connect Standalone

Synopsis

This quickstart guide shows you how to set up the OData V4 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka.

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy and extract the OData V4 connectors package into the Kafka Connect plugins directory.

Connector Configuration

Set up the OData V4 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka to extract data from an existing OData V4 service. The publicly available OData service used for this scenario is the TripPin service. It does not require authentication for read operations.

OData V4 Source Connector

1. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers=localhost:9092
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000
   plugin.path=<KAFKA_ROOT>/plugins,
   

   Note

   Make sure the plugin path exists.

2. Open TripPin service in a browser to obtain a temporary service instance URL that allows write operations. The URL will include a temporary service ID that replaces the S(readwrite) part of the URL.

3. Extract the properties with a minimal configuration and copy it to <KAFKA_ROOT>/config/odatav4-source-connector.properties. Remember to include your license key.

   name = odatav4-source-connector
   connector.class = org.init.ohja.kafka.connect.odatav4.source.OData4SourceConnector
   tasks.max = 1
   sap.odata.license.key = "Your license key here"
   sap.odata.host.address = services.odata.org
   sap.odata.host.port = 443
   sap.odata.host.protocol = https
   sap.odata#00.service = /V4/{TEMPORARY_SERVICE_ID}/TripPinServiceRW
   sap.odata#00.entityset = People
   sap.odata#00.topic = People
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

2. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

3. Start a local Kafka server instance.

   bin\windows\kafka-server-start.bat config\kraft\server.properties
   

4. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic People --from-beginning
   

5. In another shell start a local standalone Kafka Connect instance and execute the OData V4 Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\odatav4-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

Switch to the Kafka consumer shell and if everything goes right you should see JSON representations of the TripPin People details messages together with their schema printed to the standard console output.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt