Data Management
Supported Data Types and Formats
The IDoc Connectors support the Confluent JSON Converter as well as the AVRO Converter.
Data Type Mapping Source Connector
SAP® JCo defines internal data types in com.sap.conn.jco.JCoMetaData, each corresponding to one of the built-in types of SAP® ABAP®.
The IDoc Source Connector supports the following SAP® basic data types and mappings to Kafka Connect org.apache.kafka.connect.data data/schema types:
| JCo | Kafka Connect Schema Type | Java Data Type |
|---|---|---|
| TYPE_CHAR | STRING | java.lang.String |
| TYPE_DECF16 | Decimal | java.math.BigDecimal |
| TYPE_DECF34 | Decimal | java.math.BigDecimal |
| TYPE_DATE | Date | java.util.Date |
| TYPE_BCD | Decimal | java.math.BigDecimal |
| TYPE_FLOAT | FLOAT64 | java.lang.Double |
| TYPE_INT1 | INT16 | java.lang.Short |
| TYPE_INT2 | INT16 | java.lang.Short |
| TYPE_INT | INT32 | java.lang.Integer |
| TYPE_INT8 | INT64 | java.lang.Long |
| TYPE_BYTE | BYTES | java.nio.ByteBuffer |
| TYPE_NUM | STRING | java.lang.String |
| TYPE_XSTRING | STRING | java.lang.String |
| TYPE_TIME | Time | java.util.Date |
| TYPE_STRING | STRING | java.lang.String |
| TYPE_UTCLONG | INT64 | java.lang.Long |
| TYPE_UTCMINUTE | INT64 | java.lang.Long |
| TYPE_UTCSECOND | INT64 | java.lang.Long |
| TYPE_DTDAY | INT32 | java.lang.Integer |
| TYPE_DTWEEK | INT32 | java.lang.Integer |
| TYPE_DTMONTH | INT32 | java.lang.Integer |
| TYPE_TSECOND | INT32 | java.lang.Integer |
| TYPE_TMINUTE | INT16 | java.lang.Short |
| TYPE_CDAY | INT32 | java.lang.Integer |
Data Type Mapping Sink Connector
The IDoc Sink Connector supports flat-structured tables with the following SAP® basic data types and their mappings to Kafka Connect’s org.apache.kafka.connect.data data/schema types:
| JCo | Kafka Connect Schema Type | Restrictions |
|---|---|---|
| TYPE_UTCSECOND | INT8 | between 0 and 315538070400 |
| TYPE_UTCSECOND | INT16 | between 0 and 315538070400 |
| TYPE_UTCSECOND | INT32 | between 0 and 315538070400 |
| TYPE_UTCSECOND | INT64 | between 0 and 315538070400 |
| TYPE_UTCMINUTE | INT8 | between 0 and 525896784 |
| TYPE_UTCMINUTE | INT16 | between 0 and 525896784 |
| TYPE_UTCMINUTE | INT32 | between 0 and 525896784 |
| TYPE_UTCMINUTE | INT64 | between 0 and 525896784 |
| TYPE_UTCLONG | INT8 | between 0 and 3155380704000000000 |
| TYPE_UTCLONG | INT16 | between 0 and 3155380704000000000 |
| TYPE_UTCLONG | INT32 | between 0 and 3155380704000000000 |
| TYPE_UTCLONG | INT6 | between 0 and 3155380704000000000 |
| TYPE_BYTE | INT8 | INT8 interpreted as Byte in an array of length 1 |
| TYPE_BYTE | BYTES | |
| TYPE_BYTE | BYTES/logical type Decimal | java.math.BigDecimal |
| TYPE_TSECOND | INT8 | between 0 and 86401 |
| TYPE_TSECOND | INT16 | between 0 and 86401 |
| TYPE_TSECOND | INT32 | between 0 and 86401 |
| TYPE_TMINUTE | INT8 | between 0 and 1441 |
| TYPE_TMINUTE | INT16 | between 0 and 1441 |
| TYPE_DTMONTH | INT8 | between 0 and 119988 |
| TYPE_DTMONTH | INT16 | between 0 and 119988 |
| TYPE_DTMONTH | INT32 | between 0 and 119988 |
| TYPE_XSTRING | BOOLEAN | “X”=true, "“=false | | TYPE_XSTRING | STRING | | | TYPE_XSTRING | schema type not in (MAP,STRUCT,ARRAY) | | | TYPE_STRING | BOOLEAN | ”X“=true, ”"=false |
| TYPE_STRING | STRING | |
| TYPE_STRING | schema type not in (MAP,STRUCT,ARRAY) | |
| TYPE_DTWEEK | INT8 | between 0 and 521725 |
| TYPE_DTWEEK | INT16 | between 0 and 521725 |
| TYPE_DTWEEK | INT32 | between 0 and 521725 |
| TYPE_FLOAT | FLOAT32 | |
| TYPE_FLOAT | FLOAT64 | |
| TYPE_DTDAY | INT8 | between 0 and 3652061 |
| TYPE_DTDAY | INT16 | between 0 and 3652061 |
| TYPE_DTDAY | INT32 | between 0 and 3652061 |
| TYPE_TIME | INT32/logical type Time | java.lang.Integer or java.util.Date |
| TYPE_TIME | STRING | pattern HHmmss |
| TYPE_INT8 | INT8 | |
| TYPE_INT8 | INT16 | |
| TYPE_INT8 | INT32 | |
| TYPE_INT8 | INT64 | |
| TYPE_INT8 | INT64/logical type Timestamp | java.util.Date (milliseconds since Unix epoch) |
| TYPE_INT2 | INT8 | |
| TYPE_INT2 | INT16 | |
| TYPE_INT1 | INT8 | between 0 and 255 |
| TYPE_INT1 | INT16 | between 0 and 255 |
| TYPE_DATE | INT32/logical type Date | java.lang.Integer or java.util.Date |
| TYPE_DATE | STRING | pattern yyyyMMdd |
| TYPE_CHAR | BOOLEAN | ‘X’=true, ‘ ’=false |
| TYPE_CHAR(length) | STRING | string.length <= length |
| TYPE_CDAY | INT8 | between 0 and 366 |
| TYPE_CDAY | INT16 | between 0 and 366 |
| TYPE_BYTE | INT8 | INT8 interpreted as Byte |
| TYPE_NUM(length) | INT8 | INT8 > 0 and INT8.length <= length |
| TYPE_NUM(length) | INT16 | INT16 > 0 and INT16.length <= length |
| TYPE_NUM(length) | INT32 | INT32 > 0 and INT32.length <= length |
| TYPE_NUM(length) | INT64 | INT64 > 0 and INT64.length <= length |
| TYPE_NUM(length) | STRING | string.length <= length and string only contains digits |
| TYPE_NUM(length) | INT64/logical type Timestamp | java.util.Date (milliseconds since Unix epoch) and INT64.length <= length |
| TYPE_INT | INT8 | |
| TYPE_INT | INT16 | |
| TYPE_INT | INT32 | |
| TYPE_BCD | FLOAT32 | |
| TYPE_BCD | FLOAT64 | |
| TYPE_BCD | BYTES/logical type Decimal | b[] or java.math.BigDecimal |
| TYPE_DECF16 | FLOAT32 | |
| TYPE_DECF16 | FLOAT64 | |
| TYPE_DECF16 | BYTES/logical type Decimal | b[] or java.math.BigDecimal |
| TYPE_DECF34 | FLOAT32 | |
| TYPE_DECF34 | FLOAT64 | |
| TYPE_DECF34 | BYTES/logical type Decimal | b[] or java.math.BigDecimal |
Kafka Headers
The IDoc Source Connector supports inserting metadata information into the Kafka message header by setting sap.idoc.headers.enable = 1 (default is 0). The following header fields are supported:
| name (string) | value | value type |
|---|---|---|
| connector.version | The version of the connector | String |
| idoc.file.path | IDoc absolute file path and name | String |
| idoc.file.index | IDoc index in file, if file contains multiple IDocs | String |
| idoc.type | IDoc basic/compound type | String |
| idoc.number | IDoc number | String |
| idoc.receiver.partner.number | The receiver partner number | String |
| idoc.receiver.partner.type | The receiver partner type | String |
| idoc.receiver.port | The receiver port | String |
| auth-id | The authentication identifier (user, alias_user, snc_myname, or x509cert); may be empty for some authentication mechanisms. | String |
| jco.version | The version of jco | String |
| jco.client.ashost | The SAP application server. | String |
| jco.client.sysnr | The SAP system number. | String |
| jco.client.mshost | The SAP message server. | String |
| jco.client.msserv | The SAP message service. | String |
| jco.client.r3name | The R/3 name. | String |
Single Message Transforms (SMT)
Single Message Transforms (SMTs) allow for lightweight, real-time modifications of data as it passes through Kafka Connect. SMTs can be applied in source connectors before writing data to Kafka topics or in sink connectors before sending data to external systems.
Use Cases for SMTs
- Data Filtering: Remove unnecessary fields or records based on conditions, focusing only on relevant data.
- Field Manipulation: Modify fields by renaming, masking sensitive data, or changing formats, ensuring consistency and compatibility.
- Field Enrichment: Add metadata or default values to provide more context to messages.
- Transformation Chains: Combine multiple SMTs for complex transformations.
- Routing and Partitioning: Dynamically change the Kafka topic or partition for records based on their contents.
The IDoc Connectors support the chaining of multiple SMTs, allowing for flexible and powerful data processing, enhancing the capabilities of data pipelines in Kafka Connect.
Schema Registry Integration
The connector is fully compatible with Confluent Schema Registry, allowing seamless integration of IDoc data with Kafka topics that use Avro, JSON Schema, or Protobuf formats. This compatibility ensures that all schema definitions for IDoc messages are stored and managed centrally, simplifying the development and maintenance of applications that consume the data.
Advantages of a Schema Registry
- Data consistency: Enforces a well-defined schema for IDoc messages, preventing issues like missing fields or type mismatches.
- Schema evolution: Supports backward and forward schema compatibility, allowing you to update IDoc structures without breaking existing consumers.
- Reduced data size: Avro and Protobuf serialization minimize the payload size, improving data transmission efficiency.
- Centralized schema management: Simplifies handling multiple IDoc message formats by storing all schema versions in one place.
This integration enhances data governance and ensures robust handling of IDoc messages in Kafka.