Key Features

Extraction and Initialization Modes

You can choose between four extraction modes using the sap.odp#00.init-simulation setting, which controls how data is initialized and extracted:

1. Delta Initialization with Data Transfer (Mode 0): Extracts all available data up to the time of the first extraction request.

2. Delta Initialization without Data Transfer (Mode 1) (also known as delta simulation): Only new delta records are extracted after the initial extraction request. Existing data before this point is not included.

3. Full Initialization with Filters (Mode 2): Allows custom filters to be applied during the first extraction, reducing the amount of data. If the full extraction process fails or is interrupted, the connector will create a new initialization request upon restart. However, this might break exactly-once semantics, so failed messages should be deleted manually if exactly-once delivery is required. Note: Failed extractions cannot be recovered with SAP® ODP.

4. Full Extraction with Filters (Mode 3): Used for ODP sources that don’t support delta extractions. A full data snapshot is requested each time, either based on a set schedule or cron configuration. Be aware that multiple full extractions can create duplicate entries in the source system.

Delivery Semantics

The ODP Source Connector supports exactly-once delivery, ensuring no duplicates when processing data. You can enable this feature using the exactly.once.support property and setting transaction.boundary=poll. For more information, refer to KIP-618.

Exactly-once delivery applies to ODP data extractions that are configured for either delta initialization or one-time full extractions.

The ODP API includes features to commit delta requests and track them per subscriber. If there’s an issue, the connector uses Kafka Connect’s offset to restart. While the source system might extract data multiple times, the connector filters out duplicates using offset data before passing it to Kafka producers.

How to Enable Exactly-Once in Kafka

To enable exactly-once support, update the Kafka worker setting exactly.once.source.support to either preparing or enabled. Then, configure the connector’s exactly.once.support property:

• Required: Guarantees exactly-once delivery with a pre-check. If the connector can’t support exactly-once, it will fail.
• Requested (default): No pre-check is performed, but it attempts exactly-once delivery.

Offset Handling

Each time the connector runs poll(), it creates ODP delta requests, collecting data from the last poll until the current time. Data is broken into packages based on the package-size setting, which defines the maximum bytes per package.

Each package has unique identifiers: request ID, package number, and record number, which are used to manage offsets. The connector commits data at the request level. If any issues occur, it retrieves the latest offsets and re-extracts requests to prevent data loss. Any duplicated records are filtered out using offset data to ensure no duplicates are sent to Kafka.