Functionality
Source Connector Functionality
The IDoc Source Connector uses Remote Function Call (RFC) to extract IDoc files generated and stored in a SAP® ABAP environment. Each IDoc, including the root segment, subsegments, and control structure, is transferred to Kafka as a single message. The data types of the fields in each segment are accurately mapped to Kafka’s corresponding types, preserving all data type details from the SAP® system’s metadata.
IDoc File Processing Sequence
In each query cycle, the connector processes files in the path specified by the file.path configuration and whose names match the regular expression defined in file.filter.regex. Files are processed in parallel threads, and the number of files handled in a poll cycle can be restricted using the file.queue.size option. This helps manage the flow of data to Kafka, allowing control over backpressure.
The sequence in which files are processed is determined by the file.sort.order configuration. It is advisable to align this setting with the sorting in the IDoc outbound file port, as it ensures the correct sequential order of Kafka messages in the target topic partition. Ensure that subsequent files are created in ascending order according to the sort sequence; otherwise, they may be skipped. Additionally, avoid changing the sort sequence during operation, as it may result in processing errors and break sequential order.
The configuration option accepts multiple values, which are prioritized based on their importance.
Valid values are:
| Value | Description |
|---|---|
| NameAsc (default) | Files are processed in lexicographically ascending order by file name. |
| NameDesc | Files are processed in lexicographically descending order by file name. |
| LastModifiedAsc | Files are processed in ascending order by “last modified” timestamp in SAP®. |
| LastModifiedDesc | Files are processed in descending order by “last modified” timestamp in SAP®. |
| PartNameAsc | Adds a filter based on a regular expression match on the file name, with sorting equivalent to NameAsc. |
| PartNameDesc | Adds a filter based on a regular expression match on the file name, with sorting equivalent to NameDesc. |
| Serial | Files are sorted by the control structure’s serialization field in ascending order. |
Recommendations
- Unique folders for each IDoc type: It’s recommended to use separate folders for each IDoc type on the SAP® application server to keep files organized and avoid conflicts.
- Unique file names for each IDoc: Use a unique file name for each IDoc by appending a prefix in the “Directory” field during port configuration (_WE21_) and using a function module like
EDI_PATH_CREATE_DATE_TIME. This prevents files from being overwritten, avoiding data loss.
Sink Connector Functionality
The IDoc control structure data can be included as part of each Kafka message. If no control structure is provided in an incoming message, or if certain key fields need to be overridden, the following configuration parameters are available:
| Parameter | Description |
|---|---|
| control.structure | The version of the SAP® IDoc control structure (e.g., EDI_DC or EDI_DC40). |
| basic-type | The basic type of the IDoc, according to EDI_DC-DOCTYP or EDI_DC40-IDOCTYP fields. |
| extension-type | IDoc extension type, derived from the control structure’s CIMTYP field. |
| message-type | IDoc message type, based on the control structure field MESTYP. |
| sap-release | SAP® release version, based on the control structure field DOCREL. Use null to leave it empty and apply the most recent version released. |
| application-release | Application release version used when creating an IDoc. Use null to leave it empty and apply the most recent version. |
| client | IDoc client, taken from the control structure’s MANDT field. |
| sender.partner.number | Sender number, from the control structure field SNDPRN. |
| sender.partner.type | Sender type, from the control structure field SNDPRT. |
| sender.port | Sender port, from the control structure field SNDPOR. |
| sender.address.logical | Sender logical address, based on the control structure field SNDLAD. |
| receiver.partner.number | Receiver number, from the control structure field RCVPRN. |
| receiver.partner.type | Receiver type, from the control structure field RCVPRT. |
| receiver.port | Receiver port, from the control structure field RCVPOR. |