Kafka Connect Standalone

Synopsis

This quickstart guide shows you how to set up the OData V2 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka.

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy and extract the OData V2 connectors package into the Kafka Connect plugins directory.

Connector Configuration

Set up the OData V2 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka to extract data from an existing OData V2 service. The publicly available OData service used for this scenario is the Northwind V2 service. It does not require authentication for read operations.

OData V2 Source Connector

1. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers = localhost:9092
   key.converter = org.apache.kafka.connect.json.JsonConverter
   value.converter = org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable = true
   value.converter.schemas.enable = true
   offset.storage.file.filename = /tmp/connect.offsets
   offset.flush.interval.ms = 10000
   plugin.path = <KAFKA_ROOT>/plugins,
   

   Note

   Make sure the plugin path exists.

2. Extract the properties with a minimal configuration and copy it to <KAFKA_ROOT>/config/odatav2-source-connector.properties. Remember to include your license key.

   name = odatav2-source-connector
   connector.class = org.init.ohja.kafka.connect.odatav2.source.OData2SourceConnector
   tasks.max = 1
   sap.odata.license.key = "Your license key here"
   sap.odata.host.address = services.odata.org
   sap.odata.host.port = 443
   sap.odata.host.protocol = https
   sap.odata#00.service = /V2/Northwind/Northwind.svc
   sap.odata#00.entityset = Order_Details
   sap.odata#00.topic = Order_Details
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

2. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

3. Start a local Kafka server instance.

   bin\windows\kafka-server-start.bat config\kraft\server.properties
   

4. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Order_Details --from-beginning
   

5. In another shell start a local standalone Kafka Connect instance and execute the OData V2 Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\odatav2-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

Switch to the Kafka consumer shell and if everything is successful, you should see JSON representations of the Northwind OData Order_Details messages together with their schema printed to the standard console output.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt