Kafka Connect Standalone

Synopsis

This quickstart guide shows you how to set up the OData V2 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka.

Preliminary Setup

1. Download and extract Apache Kafka.
2. Copy and extract the OData V2 connectors package into the Kafka Connect plugins directory.

Connector Configuration

Set up the OData V2 Source Connector on a local Kafka Connect instance in standalone mode using Apache Kafka to extract data from an existing OData V2 service. The publicly available OData service used for this scenario is the OData V2 service. It does not require authentication for read operations.

OData V2 Source Connector

1. Open OData V2 service in a browser to obtain a temporary service instance URL that allows write operations. The URL will include a temporary service ID that replaces the S(readwrite) part of the URL.

2. Edit the contents of file <KAFKA_ROOT>/config/connect-standalone.properties like this:

   bootstrap.servers = localhost:9092
   key.converter = org.apache.kafka.connect.json.JsonConverter
   value.converter = org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable = true
   value.converter.schemas.enable = true
   offset.storage.file.filename = /tmp/connect.offsets
   offset.flush.interval.ms = 10000
   plugin.path = <KAFKA_ROOT>/plugins,
   

   Note

   Make sure the plugin path exists.

3. Extract the properties with a minimal configuration and copy it to <KAFKA_ROOT>/config/odatav2-source-connector.properties. Remember to include the temporary service ID as well as your license key.

   name = odatav2-source-connector
   connector.class = org.init.ohja.kafka.connect.odatav2.source.OData2SourceConnector
   tasks.max = 1
   sap.odata.license.key = "Your license key here"
   sap.odata.host.address = services.odata.org
   sap.odata.host.port = 443
   sap.odata.host.protocol = https
   sap.odata#00.service = /{ENTER SERVICE ID HERE}/V2/OData/OData.svc/
   sap.odata#00.entityset = Suppliers
   sap.odata#00.topic = Suppliers
   

Execution

The following steps are intended for users running Windows OS. Please make sure to use the appropriate paths and commands for your environment. If you are using a different operating system, adapt the commands accordingly.

1. Use a custom logging configuration file located at <KAFKA_ROOT>/config/tools-log4j.properties.

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   

2. Format the Kafka storage directory from the shell using a random cluster ID.

   bin\windows\kafka-storage.bat random-uuid
   

   bin\windows\kafka-storage.bat format --standalone -t <KAFKA_CLUSTER_ID> -c config/kraft/server.properties
   

3. Start a local Kafka server instance.

   bin\windows\kafka-server-start.bat config\kraft\server.properties
   

4. Start a simple standalone Kafka consumer from another shell.

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Suppliers --from-beginning
   

5. In another shell start a local standalone Kafka Connect instance and execute the OData V2 Source Connector.

   bin\windows\connect-standalone.bat config\connect-standalone.properties config\odatav2-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt.

Switch to the Kafka consumer shell and if everything is successful, you should see JSON representations of the OData Service Suppliers messages together with their schema printed to the standard console output.

Logging

Check the log outputs by opening file log.txt in an editor of your choice. For Windows OS just type:

type log.txt