Kafka Connect - IDoc Connectors

Dependencies

The connectors depend on the SAP® Java Connector 3.1 SDK and SAP® Java IDoc Library to interact with SAP® systems. To run the connectors you need to provide a copy of SAP® JCo library v3.1.8 (jar and native library) and Java IDoc library v3.1.3 in the classpath or in the plugin path configured for Kafka Connect.

Note

JCo needs to be obtained separately from the SAP® Marketplace. For more detailed information about licensing terms and how to obtain a license visit the SAP® FAQ and the SAP® connectors’ homepage.

License

See licenses for INITs evaluation license (DE and EN) or for more information on the dependencies’ licenses.

Installation

Packaging

The IDoc connectors package name is: init-kafka-connect-idoc-<connector version>.zip

The zip archive includes one folder called init-kafka-connect-idoc-<connector version>, which itself contains the nested folders lib, etc, assets, sap and doc.

  • lib/ contains the uber Java archive that needs to be extracted into the plugin.path of Kafka Connect.
  • etc/ contains a sample properties file for the connector that can be supplied as an argument in the CLI during startup of Kafka Connect or by upload to the Confluent Control Center.
  • doc/ contains more detailed documentation about the connector like licenses, readme, configurations and so on.
  • assets/ contains media files like icons and company logos.
  • sap/ contains SAP® transports that can be installed (optionally) in the SAP® system to provide minor performance improvements.

Manual

Put the connector jars and library jars from directory /lib in the configured plugin path of Kafka Connect.

Confluent CLI

The Confluent platform provides a command line installation interface that can be used to install the connectors package zip file from a local file system, see Confluent CLI Command Reference

SAP® system connection

The IDoc connectors include configuration properties from SAP&reg JCo, which are identified by the prefix jco. For a detailed description of these properties, please refer to the Java documentation of the com.sap.conn.jco.ext.DestinationDataProvider interface. The JCo JavaDoc can be found within the sapjco3.jar package.

A basic configuration for a SAP® JCo client destination is as follows:

# SAP Netweaver application host DNS or IP
jco.client.ashost = 127.0.0.1
# SAP system number
jco.client.sysnr = 20
# SAP client number
jco.client.client = 100
# SAP RFC user
jco.client.user = user
# SAP user password
jco.client.passwd = password

Rather than specifying the application host directly using the jco.client.ashost property, one can use jco.client.mshost to utilize the Message Server for load balancing within SAP®.

To enable encryption and data integrity for communication between the connector and SAP®, one can use the SAP® Secure Network Connection (SNC) and its corresponding JCo configuration properties.

The connector supports various authentication types, including user/password, SNC Single Sign-On (SSO) using X509 certificates, and SAP® cookie v2 logon ticket.

For TCP/IP ports utilized by SAP® RFC, please refer to the TCP/IP ports of all SAP® products. The default ports for RFC communication are as follows:

  • 33<NN> (unencrypted)
  • 48<NN> (encrypted)

To be able to establish communication between the connector and the SAP® system and within the framework of authorization management, you need an SAP® user. The SAP® user must be of type Communications or Dialog. To save SAP® dialog resources, the Communications type is recommended.

Authorizations

The SAP® user configured for the connector needs at least the following authorizations:

  • Object S_RFC:
    ACTVT: 16
    RFC_NAME: RFCPING, SMLG_GET_DEFINED_GROUPS, RFC_GET_FUNCTION_INTERFACE, RFC_METADATA_GET, BAPI_USER_GET_DETAIL, EPS2_GET_DIRECTORY_LISTING, EPS_READ_BLOCK, EPS_CLOSE_FILE, EPS_DELETE_FILE, /INI/IDOC_GET_DTELS, /INI/IDOC_GET_CONTROLS, IDOCTYPE_READ_COMPLETE, DD_DTEL_GET, SYSTEM_RESET_RFC_SERVER, IDOCTYPES_LIST_WITH_MESSAGES, API_CREATE_TID, API_CLEAR_TID
    RFC_TYPE: FUNC

  • Object S_RFC:
    ACTVT: 16
    RFC_NAME: RFC1
    RFC_TYPE: FUGR

  • Object S_CTS_ADMI:
    CTS_ADMFCT: EPS1

  • Object S_DATASET:
    ACTVT: 33, 06
    FILENAME:
    PROGRAM: SAPLEPSF

  • Object S_IDOCDEFT:
    ACTVT: 03
    EDI_TCD: WE30

  • Object B_ALE_RECV:
    EDI_MES: <idoc message types, e.g. DEBMAS>

For more detailed information, see the following SAP® support notes:

Graceful backoff

If there are any connection or communication issues with the configured SAP® system, the connector will employ a retry backoff strategy. The maximum number of retry attempts can be configured using the sap.idoc.max.retries property. After each failed attempt, the connector will pause execution for a random number of milliseconds between the values set in sap.idoc.min.retry.backoff.ms and sap.idoc.max.retry.backoff.ms configuration properties.

Any exceptions for connection attempts and communication with SAP® are assigned an internal exception group. The list of exception groups for which the backoff retry strategy is applied can be configured using property sap.idoc.retry.exception.groups. A complete list of exception groups can be found in the JCo JavaDocs of com.sap.conn.jco.JCoException. BAPI return messages of any message class can be included, too.

Supported features

  • The connectors have built-in support for IDocs using control structure versions 0 (EDI_DC) and 40 (EDI_DC40)

SMT

The use of Single Message Transforms in an iDoc connector allows for each record to be passed through one or a chain of several simple transformations. The iDoc connectors support SMT and have been successfully tested with a concatenation of multiple SMTs.

1.3.5