Data Management
Data Serialization
The OData Business Events Source Connector supports the various data converters, such as the JSON Converter as well as the AVRO Converter.
Supported Data Types
The following table provides an overview of the OData V2 data type mapping from OData V2 data types to Kafka Connect data types applied by the source connector:
| OData V2 | Kafka Connect Schema Type | Java Data Type |
|---|---|---|
| Bit | INT8 | java.lang.Byte |
| Uint7 | INT8 | java.lang.Byte |
| EdmSByte | INT8 | java.lang.Byte |
| EdmByte | INT16 | java.lang.Short |
| EdmGuid | STRING | java.lang.String |
| EdmTime | TIME | java.util.Date |
| EdmInt16 | INT16 | java.lang.Short |
| EdmInt32 | INT32 | java.lang.Integer |
| EdmInt64 | INT64 | java.lang.Long |
| EdmBinary | BYTES | java.nio.ByteBuffer |
| EdmDouble | FLOAT64 | java.lang.Double |
| EdmSingle | FLOAT32 | java.lang.Float |
| EdmString | STRING | java.lang.String |
| EdmBoolean | BOOL | java.lang.Boolean |
| EdmDecimal | DECIMAL | java.math.BigDecimal |
| EdmDateTime | TIMESTAMP | java.util.Date |
| EdmDateTimeOffset | TIMESTAMP | java.util.Date |
| EdmStructuralType | STRUCT | java.util.Map[String, Object] |
Kafka Headers
The OData Business Event Source Connector supports inserting metadata information into the Kafka message header by setting sap.odata.headers.enable = 1 (default is 0). The following header fields are supported:
| name (string) | value | value type |
|---|---|---|
| connector.version | The version of the connector | String |
| GUID | Global unique identifier | String |
| subscriber.code | The subscriber code; the default is KBES | String |
| Event Data Service URL | OData Businessevent URL | String |
| oauth.token.endpoint | OData oauth token endpoint | String |
| oauth.client.id | OData oauth client id | String |
| oauth.redirect | OData oauth redirect url | String |
Single Message Transforms (SMTs)
Single Message Transforms (SMTs) allow for lightweight, real-time modifications of data as it passes through Kafka Connect. SMTs can be applied in source connectors before writing data to Kafka topics. The OData Business Events Source Connector supports SMT and has been successfully tested with a concatenation of two SMTs.
Use Cases for SMTs
- Data Filtering: Remove unnecessary fields or records based on specific conditions, focusing only on relevant data.
- Field Manipulation: Modify fields by renaming, masking sensitive data, or changing formats, ensuring consistency and compatibility.
- Field Enrichment: Add metadata or default values to provide more context to messages.
- Transformation Chains: Combine multiple SMTs for complex transformations.
- Routing and Partitioning: Dynamically change the Kafka topic or partition for records based on their contents.
The OData Business Events Source Connector supports the chaining of multiple SMTs, allowing for flexible and powerful data processing, further enhancing the capabilities of data pipelines in Kafka Connect.