Key Features

Delivery Semantics of the Source Connector

Change Data Capture was officially introduced with the OASIS OData V4 protocol specification. With the Apache Olingo V4 library the source connector is capable of implementing delta mode based on the specification. This includes offset recovery and exactly-one delivery semantics.

While both the source and sink connectors offer at-least-once delivery semantics, the source connector ensures exactly-once delivery in delta mode or full mode when execution period is set to -1 (one-time polls).

Enabling Exactly-Once in Kafka

To enable exactly-once support, update the Kafka worker configuration property exactly.once.source.support to preparing or enabled. Refer to KIP-618 for details.

When deploying a source connector, configure the exactly.once.support property as follows:

  • Required: Ensures exactly-once delivery, with a preflight check. The connector fails if it can’t support exactly-once delivery.
  • Requested (default): No preflight check is performed.

Operation Modes of the Source Connector

The source connector operates in two modes: full mode and delta mode. You can set the mode via the track-changes property to either 0 (full mode) or 1 (delta mode).

Operation Modes

  • Full Mode: Periodically requests data from an OData V4 service’s entity set.
  • Delta Mode: Uses change-tracking to extract only new, changed, or deleted records after an initial data request. Deleted records include key properties and a RECORDMODE or ODQ_CHANGEMODE field set to “D”, with a DELETED_AT timestamp.

After each extraction, the source connector waits for the time specified in exec-period before the next extraction.

Offset Handling

Each message’s offset contains:

  • The URL from which it was requested,
  • A record number identifying the message’s position in the response,
  • Optionally, a second URL for server-side paging (next link) or change tracking (delta link).

If issues arise, the connector retrieves the latest offsets from Kafka Connect, resuming from the last processed request. This ensures no data loss, and duplicate messages are filtered out. In delta mode, at-least-once semantics are maintained, while in full mode, the source system must guarantee data consistency across calls.

Delivery Semantics of the Sink Connector

The sink connector makes use of Kafka’s message commit functionality and is therefore able to guarantee at-least-once semantics.

How It Works

  • The connector stores consumed records in a task cache and delivers them at intervals specified by the connect worker configuration property: offset.flush.interval.ms.
  • You can trigger deliveries outside of this interval based on the number of records in the cache using the sap.odata.flush.trigger.size property.

Operation Modes of the Sink Connector

The sink connector supports three operation modes: Insert, Update, and Dynamic, set by the configuration property sap.odata.operation.mode. Currently, only OData V4 CRUD operations are supported, and there is no Upsert mode. However, upsert functionality can be implemented by the OData service provider.

Operation Modes

  1. Insert Mode:

    • Uses HTTP POST to export entries.
    • Entries already existing in the target system may result in errors, depending on the OData service.
    • Supports deep inserts, allowing insertion of nested business objects and their associations. Learn more about deep inserts here.

  2. Update Mode:

    • Uses HTTP PUT to update entities.
    • Fails if the entry doesn’t exist in the target system.
    • Requires all fields to be present for the update; partial updates (HTTP MERGE) are not supported.
    • Deep updates are not supported by OData and will result in exceptions.

  3. Dynamic Mode:

    • The HTTP method (POST or PUT) is determined by the record field value specified in sap.odata.operation.mode.property:
      • ‘I’ = HTTP POST (Insert)
      • ‘U’ = HTTP PUT (Update)

Delete Operation

  • Enable the sap.odata.enable.delete property to allow deleting records in the target system when a Kafka tombstone record (key with null value) is consumed.
  • Tombstone records are converted into HTTP DELETE requests, or ignored if this property is disabled.
  • Ensure all key properties are included in the record key, which must be a struct.
1.3.6