Advanced Features

Group of Files and IDocs

The connector has built-in support for configuring multiple groups of IDoc types and files. A group is defined by the configuration prefix sap.idoc#<num> and includes a file path as well as a file name regular expression filter for selecting IDoc files to be processed as part of the corresponding group. Each group will be processed independently and may be executed on a separate connector task. It is recommended to use a separate folder for each IDoc type, but at least one overlap-free file selection should be configured for proper connector execution.

Package Size Restriction

The size of a package of sink records sent to the target system as part of a single SAP® transaction is mainly determined by the batch of records handed over from the Kafka Connect consumer per topic.

The Configuration parameter sap.idoc.package.size can be used to further restrict the maximum number of sink records being part of a package.

If a send or conversion error occurs for this batch, a partial commit will be done.

• When error.tolerance is set to None, the last committed offset will be from the last successfully send record. It will stop when the first conversion error occurs.
• When error.tolerance is set to All, all records will be sent, despite the conversion errors. Even the records after the conversion error will be sent.

If batching records is disabled, each record read from a Kafka topic read by the Kafka Connect consumer in one poll cycle will be sent to the target system as part of exactly one SAP® transaction.

Timeout in Pre-commit

There will be a timeout if in pre-commit no data could be loaded. To wait longer please modify the parameter sap.odata.flush.wait.ms. You may have to raise the consumer.override.max.poll.interval.ms and add the connector.client.config.override.policy = All to the sink connectors properties file.

Scheduling

The connector is meant to run continuously according to streaming principles. Nevertheless, a clock period can be set using configuration property exec-period per single IDoc source. A data extraction will then take place only once in a clock period interval. As an alternative one can use configuration properties cron.expression and cron.security.time.sec to define a cron-like schedule, e.g. if a source system specifies strict timeframes for extraction processes.

Plugin Discovery

Plugin Discovery is the strategy the Connect worker uses to find plugin classes and make them available for configuration and execution. This process is controlled by the worker’s plugin.discovery setting.

By default, the plugin.discovery setting is HYBRID_WARN, which is compatible with all plugins and logs a warning if it encounters any plugins incompatible with the SERVICE_LOAD mode. The SERVICE_LOAD option, which uses the faster ServiceLoader mechanism, may improve performance during worker startup, but will not load incompatible plugins. See Connect Worker Configuration for all plugin.discovery values.

The IDoc Connectors support the ServiceLoader mechanism.