Product Brief

This document contains a brief description of the Kafka Connect ODP source connector.

Contact

Company
INIT Individuelle Softwareentwicklung & Beratung GmbH
Rotebühlplatz 21
70178 Stuttgart - Deutschland
Amtsgericht Stuttgart, HRB 224214
Geschäftsführer: Thomas Winkelsdorf
https://init-software.de/

Business and technical contact
Ronald Schertle
connector-contact@init-software.de
+49 173 81 49 709

Description

The ODP Source connector is an Apache Kafka Connect source connector to extract data out of SAP® ODP delta enabled data sources.

SAP® Operational Data Provisioning provides a technical infrastructure for data extraction and replication out of various SAP® products. ODP can be used to directly extract data out of specific ODP sources in full mode, extract data in delta mode using a so-called Operational Data Queue with potentially multiple independent subscribers and extract data in real time using Real-Time Data Acquisition. In general, it can be called a publish subscribe system with the following benefits:

  • The ODP infrastructure provides services such as monitoring data requests for delta enabled ODP sources
  • Data is stored in a compressed state in the ODQ of the source system
  • Data changes in the ODQ can be consumed by multiple subscribers
  • Data in the ODQ is retained for a specified time period
  • Data in the ODQ can be extracted multiple times, e.g. for recovery purposes

The connector uses the offset storage management provided by Kafka Connect and implements exactly-once delivery semantics as meant by Kafka Connect.

Initial extraction runs for a new SAP® ODP data source include an initialization. Delta initialization can be done by either extracting all data up to the current point in time or by telling the source system to skip all the data and just deliver delta updates beginning from the first execution of the connector. Full initialization mode is a special kind of initialization giving you the option to apply complex filters to the data extracted during initialization. The initialization behavior can be customized on ODP level using configuration property “delta-initialization”.
Direct full extractions are provided by the ODP API, but not supported by the ODP source connector yet.

ODPs provide data in a tabular format. Nesting of data types is not supported. Most of the SAP® data types are supported by the ODP source connector. The metadata for these fields being extracted is provided by the SAP® system and the connector use this information to build a rich Kafka Connect schema provided with each generated SourceRecord.

Versioning

ODP source connector

  • Connector name: ODP Source Connector
  • Version: 1.3.5
  • Type: Kafka Connect source connector

ODP source system

The ODP Source Connector uses the SAP® ODP API v2 (see SAP® Note 1931427), which is available in the following Netweaver versions:

  • PI_BASIS 730 Support Package 14 (Part of SAP® NetWeaver 7.30 Support Package 14)
  • PI_BASIS 731 Support Package 16 (Part of SAP® NetWeaver 7.03 Support Package 16 und SAP® NetWeaver 7.31 Support Package 16)
  • PI_BASIS 740 Support Package 11 (Part of SAP® NetWeaver 7.40 Support Package 11)
  • SAP_BW 750 Support Package 0 (including earlier PI_BASIS-Packages)
  • DW4CORE 100 SP 0 (including earlier PI_BASIS-Packages)
  • DW4CORE 200 SP 0 (including earlier PI_BASIS-Packages)

Kafka

  • Tested on Confluent Platform v5.5.1, v6.0.1, v7.0.1, v7.3.0, v7.5.0 and v7.6.0
  • Tested with Apache Kafka and Connect API v2.5, v2.6, v2.8, v3.3, v3.5 and v3.6

Dependencies

The connector makes use of SAP® Java Connector (JCo) v3.1.8 released by SAP®. According to the license of SAP® JCo it is not allowed to bundle it with other software. Therefore, JCo needs to be obtained separately from the SAP® Marketplace, see SAP JCo.

Configuration

The package for the ODP source connector contains expressive configuration documentation in file configuration.html. A minimal configuration example is also included as ohja-odp-source-connector.properties file. The connector makes heavy use of validators and recommenders and has been tested with the Confluent Control Center UI.

Command line tool

The connector package includes a command line tool to test parameters for the SAP® connection, get a list of available ODP data sources and query details about ODP sources.
For further details see README.md included in the connector package.

1.3.5