Quickstart Guide

This section offers two types of guides: one for the OData v4 Source Connector on a local Kafka Connect instance in standalone mode and another one for both OData v4 connectors (source and sink) running on a Confluent platform inside docker.

Kafka Connect Standalone

Preliminary setup

1. Download and extract Apache Kafka
2. Copy and extract the OData v4 connectors package into the Kafka Connect plugins directory

Configuration

1. Edit the contents of file /config/connect-standalone.properties like this:

   bootstrap.servers=localhost:9092
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000
   plugin.path=/kafka_2.12-2.0.0/plugins,
   

   Note

   Make sure the plugin path exists

2. Extract the OData v4 Source Connector property template etc/odatav4-source-connector.properties from the connector package and copy it to /config/odatav4-source-connector.properties

Execution

1. Start a local Zookeeper instance from the shell, e.g. for Windows OS type:

   cd <KAFKA_ROOT>
   set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:<KAFKA_ROOT>/config/tools-log4j.properties
   bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
   

2. Start a local Kafka server instance from another shell, e.g. for Windows OS type:

   bin\windows\kafka-server-start.bat .\config\server.properties
   

3. Start a simple kafka consumer from another shell, e.g. for Windows OS type:

   bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Order_Details --from-beginning
   

4. Start a local standalone Kafka Connect instance and execute the OData v4 source connector from another shell, e.g. for Windows OS type:

   bin\windows\connect-standalone.bat .\config\connect-standalone.properties .\config\odata-source-connector.properties > log.txt 2>&1
   

   The logging outputs will be written to the file log.txt

Switch to the Kafka consumer shell and if everything goes right you should see json representations of the Northwind OData order details messages together with their schema printed to the standard console output.

Logging

Check the log outputs by opening file log.txt in an editor of your choice or for Windows OS just type:

type log.txt

Confluent Platform

Synopsis

This section provides instructions on how to deploy the OData v4 connectors on a local Confluent Platform instance, using a Docker environment.

Preliminary setup

1. Make sure to have Docker Engine and Docker Compose installed on the machine where you want to run the Confluent Platform
   1. Docker Desktop available for Mac and Windows includes both
2. Download and launch a ready-to-go Confluent Platform Docker image as described in Confluent Platform Quickstart Guide
3. Ensure that the machine where the Confluent Platform is running on has a network connection to the publicly available (read only) Northwind V4 and (read/write) OData V4 services.

Connector Package Installation

The OData v4 connectors can either be installed manually or through the Confluent Hub Client.

Manual Installation

1. Unzip the zipped connector package init-kafka-connect-odatav4-x.x.x.zip
2. Move the unzipped connector folder into the configured CONNECT_PLUGIN_PATH of the Kafka Connect service
3. Within the directory where the docker-compose.yml of the Confluent Platform is located, you can start the Confluent Platform using Docker Compose

   docker-compose up -d
   

Confluent CLI

Install the zipped connector package init-kafka-connect-odatav4-x.x.x.zip using the Confluent Hub Client from outside the Kafka Connect docker container

  docker-compose exec connect confluent-hub install {PATH_TO_ZIPPED_CONNECTOR}/init-kafka-connect-odatav4-x.x.x.zip

Further information on the Confluent CLI can be found in the Confluent CLI Command Reference

Configuration

The OData v4 connectors can be configured and launched using the control-center service of the Confluent Platform.

1. In the control-center (default: localhost:9091) select a Connect Cluster in the Connect tab
2. Click the button “Add connector” and select OData4SourceConnector or OData4SinkConnector
3. Enter a name and further required configuration for the connector

OData v4 Source Connector

To enable the OData v4 Source Connector to extract data from the Northwind V4 service, you can transfer the properties from the OData v4 Source Connector property template etc/odatav4-source-connector.properties located in the connector package to the control-center user interface. After that, you can launch it.

OData v4 Sink Connector

For the OData v4 Sink Connector to export data to the OData V4 service, you need to follow these steps:

1. To get a URL for a temporary service instance that allows write operations open OData V4 in a browser. The S(readwrite) part in the URL will be automatically replaced with a temporary service ID.
2. Transfer the properties from the OData v4 Sink Connector property template etc/odatav4-source-connector.properties in the connector package to the control-center user interface. Make sure to include the service id to the service path.
3. Launch the sink connector.
4. To test the sink connector, sample data is required. The target OData service instance also provides sample data for all entities. This data can be imported into Kafka using the OData v4 Source Connector.