Data Management
Supported Data Types and Formats
ODP Serialization Format
The ODP API and connector offer several options for serializing data when transmitting it from the ODQ system via RFC. Selecting the appropriate format is crucial because it affects the load on the SAP® system during translation, the volume of data transferred, and overall replication performance.
For most cases, we recommend using the default format, which is GZipped BasXML, for storing delta initialization data in ODQ. However, you also have other options, including:
- BAS_XML
- Column
- ABAP_XML
- GZipped BAS_XML
- GZipped Json
- Automatic
- Json
- String
To choose your preferred serialization format, use the sap.odp.serialization property.
Connector Serialization
The ODP Connector supports the Confluent JSON Converter as well as the AVRO Converter. If you choose Avro for serialization, the ODP Source Connector will automatically translate field names from the ODP source into valid Avro names, replacing any illegal characters with an underscore (_).
Data Type Mapping
SAP® JCo defines internal data types in the com.sap.conn.jco.JCoMetaData class, each corresponding to one of SAP® ABAP’s built-in types. The ODP Source Connector only supports flat tables containing the following SAP® basic data types, which are mapped to Kafka Connect’s org.apache.kafka.connect.data types:
| JCo | Kafka Connect Schema Type | Java Data Type |
|---|---|---|
| TYPE_CHAR | STRING | java.lang.String |
| TYPE_DECF16 | Decimal | java.math.BigDecimal |
| TYPE_DECF34 | Decimal | java.math.BigDecimal |
| TYPE_DATE | Date | java.util.Date |
| TYPE_BCD | Decimal | java.math.BigDecimal |
| TYPE_FLOAT | FLOAT64 | java.lang.Double |
| TYPE_INT1 | INT16 | java.lang.Short |
| TYPE_INT2 | INT16 | java.lang.Short |
| TYPE_INT | INT32 | java.lang.Integer |
| TYPE_INT8 | INT64 | java.lang.Long |
| TYPE_BYTE | BYTES | java.nio.ByteBuffer |
| TYPE_NUM | STRING | java.lang.String |
| TYPE_XSTRING | STRING | java.lang.String |
| TYPE_TIME | Time | java.util.Date |
| TYPE_STRING | STRING | java.lang.String |
| TYPE_UTCLONG | INT64 | java.lang.Long |
| TYPE_UTCMINUTE | INT64 | java.lang.Long |
| TYPE_UTCSECOND | INT64 | java.lang.Long |
| TYPE_DTDAY | INT32 | java.lang.Integer |
| TYPE_DTWEEK | INT32 | java.lang.Integer |
| TYPE_DTMONTH | INT32 | java.lang.Integer |
| TYPE_TSECOND | INT32 | java.lang.Integer |
| TYPE_TMINUTE | INT16 | java.lang.Short |
| TYPE_CDAY | INT32 | java.lang.Integer |
Meta Data
Delta Descriptive Fields
ODP sources provide two fields that describe the delta type (CRUD) of a record: ODQ_CHANGEMODE and ODQ_ENTITYCNTR (found in the ODQ_S_DELTA_APPEND structure).
Below are the possible field values and their descriptions:
| ODQ_CHANGEMODE | ODQ_ENTITYCNTR | Delta Type | Description |
|---|---|---|---|
| C | 1 | New | Describes the state of an entity after creation. Similar to an after delta record without a preceding before delta record. |
| U | -1 | Before | Describes the state of an entity before a change. Summable non-key fields will have a reversed sign. |
| U | 1 | After | Describes the new state of an entity after a change. |
| D | -1 | Reverse | Type of a delete record with contents similar to the before record. In contrast to the before record no subsequent after record will follow. |
| D | 0 | Delete | Similar to a Kafka tombstone message. Only key fields may be specified. |
| U | 0 | Additive | Describes the new state of an entity, but in contrast to an after image as difference to the previous state of the entity. Logically, it aggregates a before and after record for summable non-key fields. The rest of the non-key fields are equal to an after record. |
Kafka Headers
The ODP source connector supports inserting metadata information into the kafka message header by setting sap.odp.headers.enable = 1 (default is 0). The following header fields are supported:
| Name (String) | Value | Value Type |
|---|---|---|
| connector.version | The version of the connector | String |
| odp.name | ODP name | String |
| odp.context | ODP context | String |
| odp.subscriber | Subscriber name | String |
| odp.request | ODP request ID (TSN) | String |
| odp.package | ODP request package number | Integer |
| odp.record | ODP request record number (starting with 0) | Integer |
| numeric.mapping | numeric.mapping configuration | String |
| decimal.mapping | decimal.mapping configuration | String |
| auth-id | The authentication identifier (user, alias_user, snc_myname, or x509cert); may be empty for some authentication mechanisms. | String |
| jco.version | The version of jco | String |
| jco.client.ashost | The SAP application server. | String |
| jco.client.sysnr | The SAP system number. | String |
| jco.client.mshost | The SAP message server. | String |
| jco.client.msserv | The SAP message service. | String |
| jco.client.r3name | The R/3 name. | String |
- The ODP request ID follows the pattern:
yyyyMMddHHmmss.XXXXXXXXX. For more details about the mapping between RID, TSN, and pointer, see transaction ODQMON and tables RODPS_REPL_TID and RODPS_REPL_RID. - The package number starts at
0000000and is combined with the ODP request ID to form a unique package ID. The package number equals the ODQ_UNITNO from ODQMON minus one. - The record number starts at
0000000000and is combined with the ODP request ID and package number to form a unique record ID. This record number corresponds to ODQ_RECORDNO from ODQMON minus one.
ODQ Row Identifiers
In addition to the delta descriptive fields, the ODP Connector can add three additional ODQ row identifiers to the Kafka message value. These fields are part of the ODQ_S_DELTA_APPEND structure. To enable them explicitly, set the sap.odp#XX.identifiers.enable property to 1 (default 0). The following three identifier fields are added:
| name (string) | value | value type |
|---|---|---|
| ODQ_TSN | Transaction Sequence Number (TSN) | String |
| ODQ_UNITNO | Unit Number | Integer |
| ODQ_RECORDNO | Sequence Number of the Data Record | Integer |
The TSN string corresponds to the following pattern: yyyyMMddHHmmss.XXXXXXXXX.
SAP’s handling of ODQ row identifiers varies based on the serialization format: - String: SAP does not support adding ODQ row identifiers in combination with the String serialization format. - Automatic: SAP only populates ODQ row identifiers in the delta responses after the initial load. - For all other serialization formats SAP supports adding ODQ row identifiers.
Optional Data Conversion Settings
sap.odp.decimal.mapping: Use this property to convert DECIMAL types to other types. Settingsap.odp.decimal.mapping = primitivewill convert decimals to double or long, depending on the scale.sap.odp.currency.conversion: SAP® stores amounts in a specific format (e.g.,1 yen = 0.01). You can adjust this with thesap.odp.currency.conversionproperty.sap.odp.numeric.mapping: This property lets you transform ABAP types liked(date),t(time), andn(numeric varchar) to other formats (like strings) if needed.
Single Message Transforms (SMTs)
Single Message Transforms (SMTs) allow for lightweight, real-time modifications of data as it passes through Kafka Connect. SMTs can be applied in source connectors before writing data to Kafka topics. The ODP Source Connector supports SMTs and has been successfully tested with a concatenation of two SMTs.
Use Cases for SMTs
- Data Filtering: Remove unnecessary fields or records based on conditions, focusing only on relevant data.
- Field Manipulation: Modify fields by renaming, masking sensitive data, or changing formats, ensuring consistency and compatibility.
- Field Enrichment: Add metadata or default values to provide more context to messages.
- Transformation Chains: Combine multiple SMTs for complex transformations.
- Routing and Partitioning: Dynamically change the Kafka topic or partition for records based on their contents.
The ODP Source Connector supports the chaining of multiple SMTs, allowing for flexible and powerful data processing, enhancing the capabilities of data pipelines in Kafka Connect.
Schema Registry Integration
The connector is fully compatible with Confluent Schema Registry, allowing seamless integration of data with Kafka topics that use Avro, JSON Schema, or Protobuf formats. This compatibility ensures that all schema definitions for ODP messages are stored and managed centrally, simplifying the development and maintenance of applications that consume the data.
Advantages of Schema Registry Compatibility
- Data Consistency: Enforces a well-defined schema for ODP messages, preventing issues like missing fields or type mismatches.
- Schema Evolution: Supports backward and forward schema compatibility, allowing you to update ODP structures without breaking existing consumers.
- Reduced Data Size: Avro and Protobuf serialization minimize the payload size, improving data transmission efficiency.
- Centralized Schema Management: Simplifies handling multiple ODP message formats by storing all schema versions in one place.
This integration enhances data governance and ensures robust handling of ODP messages in Kafka.