Source connector
The IDoc source connector uses RFC to extract IDoc files generated and stored in a SAP® ABAP environment. The IDoc root segment and all corresponding subsegments together with the control structure will be transferred as a single message per IDoc to Kafka. Data types for the fields contained in a segment are properly mapped to corresponding types in Kafka connect by preserving all data type details as provided by the corresponding SAP® system’s metadata.
Restrictions
- The source connector reads IDoc XML or flat files stored on a SAP® application server using RFC
Supported features
- The connector supports files generated using port definitions (transaction WE21) for ‘File’ and ‘XML File’.
With ‘File’ you may choose either ‘IDoc rec. types SAP® Release 2.0/2.1’, ‘IDoc rec. types SAP® Release 3.0/3.1’ or ‘IDoc record types SAP® Release 4.x’(recommended). Please make sure that ‘Unicode format’ is ticked.
With ‘XML file’ use may choose to use xml format ‘SAP® Release 46’ or ‘Unicode’(recommended).
Recommendations
- It is recommended to use unique folders for each idoc type on the SAP® application server
- It is recommended to generate unique file names for each single IDoc, e.g. by appending a file name prefix to “Directory” in port configuration (WE21) and using a function module like EDI_PATH_CREATE_DATE_TIME. Otherwise, IDoc files will get overwritten and data loss will be the consequence.
Configuration
Group of files and IDocs
The connector has built-in support for configuration of multiple groups of IDoc types and files. A group is defined by the configuration prefix sap.idoc#
IDoc files processing sequence
In each query cycle, the connector processes all files that are located in the path specified by file.path and whose file name matches the regular expression specified by file.filter.regex. The connector processes the files in individual threads. The number of files to be processed in a poll cycle can further be restricted using the configuration option file.queue.size. This can be used to influence the backpressure to Kafka.
Configuration option file.sort.order specifies the sequence of file processing by the source connector during a poll cycle. It is recommended to select the sorting according to the settings in the IDoc outbound file port. This setting therefore has an essential influence on the sequential order of the Kafka IDoc messages in the corresponding target topic partition. It is important to ensure that the files following the IDoc file currently being processed are created in ascending order according to the sort sequence, otherwise they will be permanently skipped. It is also recommended not to change the sorting sequence during operation, otherwise this may also lead to faults in the processing of the IDoc files and compliance with sequential sorting may be violated.
The configuration option accepts multiple values, sorted according to their priority.
Valid values are:
| value | description | |
|---|---|---|
| NameAsc (default) | File sequence according to the file name in lexicographically ascending order | |
| NameDesc | File sequence according to the file name in lexicographically descending order | |
| LastModifiedAsc | File sequence according to the “last modified timestamp” in SAP® in ascending order | |
| LastModifiedDesc | File sequence according to the “last modified timestamp” in SAP® in descending order | |
| PartNameAsc | Adds an additional option to filter based on a regular expression match on the file name. Besides that sorting equals NameAsc. | |
| PartNameDesc | Adds an additional option to filter based on a regular expression match on the file name. Besides that sorting equals NameDesc. | |
| Serial | The final IDoc list sorted according to the control structure serialization field in ascending order |
Headers
The connector supports inserting metadata information into the kafka message header by setting sap.idoc.headers.enable = 1 (default). The following header fields are supported:
| name (string) | value | value type |
|---|---|---|
| idoc.file.path | IDoc absolute file path and name | String |
| idoc.type | IDoc basic/compound type | String |
| idoc.number | IDoc number | String |
JMX metrics
The IDoc source connector supports all the connector and task metrics provided by Kafka Connect through Java Management Extensions (JMX). In addition, the IDoc source connector provides extra JMX metrics for accessing state managed by the connector.
MBean: org.init.ohja.kafka.connect:type=idoc-source-task-metrics,connector=([-.w]+),task=([d]+)
| Metric | Explanation |
|---|---|
| task-active-threads | Count of active threads that are in use by the connector for processing data. |
| thread-retries | Count of retries performed in a connector thread that is in retrying state. |
| thread-next-retry | Timestamp for next retry performed in a connector thread that is in retrying state. |
| jco-destination-pool-capacity | Maximum number of connections that will be held open for the JCo destination instance. |
| jco-destination-peak-limit | Maximum number of connections that can be used simultaneously with the JCo destination instance |
| jco-destination-pooled-connections | Count of connections that are currently held open for the JCo destination instance. |
| jco-destination-used-connections | Count of connections that are currently being used with the JCo destination instance. |
| ${configGroup}-file-path | File path configured for IDoc configured in the configuration group. |
| ${configGroup}-file-regexp | File name regular expression filter for IDoc configured in the configuration group. |
| ${configGroup}-files-queue | Files currently in processing queue for IDoc configured in the configuration group. |
| ${configGroup}-committed-record | Current committed Kafka offset for IDoc configured in the configuration group. |
| ${configGroup}-recent-poll | Timestamp for last time of extraction for IDoc configured in the configuration group. |