Configuration

The WebDS Connector package includes detailed configuration documentation, which can be found in the following locations:

• Sink Configuration
• Examples

Additionally, the connector package includes a configuration template file in the etc/ subfolder:

• webds-sink-connector.properties

The connector utilizes extensive validators and recommenders and has been tested with the Confluent Control Center UI.

Benefits of Configuring through Confluent Control Center

The WebDS Connector fully supports configuration through the Confluent Control Center UI. Compared to using property files, configuring through this UI offers several advantages:

• Suggested property values
• A comprehensive list of recommended settings
• Incremental visibility into relevant configuration options
• A wide range of interactive validations
• And more

To maintain optimal performance, the Confluent Control Center UI displays a maximum of 1000 recommendations at a time.

Once the required fields for a service entity are filled out in the configuration UI, a new section will appear, allowing you to configure additional service entities.

To see how easy it is to set up the WebDS Connector using the Confluent Control Center, watch this screen recording.

SAP® System Connection

The WebDS Sink Connector includes configuration properties from SAP® JCo, which are identified by the prefix jco.. For a detailed description of these properties, please refer to the Java documentation of the com.sap.conn.jco.ext.DestinationDataProvider interface. The JCo JavaDoc can be found within the sapjco3.jar package.

A basic configuration for a SAP® JCo client destination is as follows:

# SAP Netweaver application host DNS or IP
jco.client.ashost = 127.0.0.1
# SAP system number
jco.client.sysnr = 20
# SAP client number
jco.client.client = 100
# SAP RFC user
jco.client.user = user
# SAP user password
jco.client.passwd = password

If you’re using SAP’s Message Server for load balancing, replace jco.client.ashost with jco.client.mshost.

To ensure encryption and data integrity, use SAP® Secure Network Connection (SNC) properties. The connector also supports various authentication types like user/password, SNC SSO, and SAP® cookie logon tickets.

SAP® Authorizations

To be able to establish communication between the connector and the SAP® system and within the framework of authorization management, you need an SAP® user. The SAP® user must be of type Communications or Dialog. To save SAP® dialog resources, the Communications type is recommended.

Minimum Required Authorizations

• Object S_RFC:
  ACTVT: 16
  RFC_NAME: RFCPING, SMLG_GET_DEFINED_GROUPS, RFC_GET_FUNCTION_INTERFACE, RFC_METADATA_GET, /BIC/CQ*
  RFC_TYPE: FUNC

• Object S_RFC:
  ACTVT: 16
  RFC_NAME: RFC1, RSDS_BAPI, RSDS_WS
  RFC_TYPE: FUGR

WebDS Sink Configuration

A minimal connector sink configuration looks like this:

# A SAP BI webservice data source is identified by a primary key combination 
# containing a system name and a data source name.
sap.webds#00.system = MYSELF
sap.webds#00.name = ZJCOTESTVERT1
sap.webds#00.topic = ZDO1VERT

topic defines the Kafka input topic from which the connector will consume data.

Available Webservice data sources in the SAP sink system can be looked up by executing transaction RSA1, opening the Modeling - Source Systems - Web Service perspective and clicking on the available system. This will guide you to a list of data sources available for the selected system.

SAP BI Webservice data sources offer real-time data acquisition, allowing source systems to push data sets of any batch size or single events to the webservice data source. These datasets are grouped into requests by the BI system. A request is closed either when the customizable limits for time or size in the corresponding infopackage are exceeded, or if an RDA daemon actively closes the request. DTPs for RDA and daemon jobs can then be used to automatically transform and write the data to targets like data store objects.

Configuration Support with our CLI Tool

The webservice sink connector package includes a command line interface (CLI) to help with various tasks such as:

• Validating connector properties
• Testing SAP® connections
• Listing available webservice data sources
• Querying details of a webservice data source

Since the CLI is written in Scala you can execute it in either Scala or Java.

Prerequisites

To run the CLI you need to provide the following dependencies in the CLASSPATH:

1. Scala runtime libraries
2. kafka-clients
3. kafka-connect-webds connector libraries
4. SAP Java Connector 3.1 SDK

Since a Java runtime is provided by the Confluent Platform and the Scala runtime libraries are part of the connector package, executing the CLI with Java would not require installing Scala.

Java

copyjava -cp <kafka-connect-webds>:<kafka-clients>:\
<kafka-avro-serializer>:<sapjco3> \
org.init.ohja.kafka.connect.webds.sink.WebDSDetailsApp \
<command> <options>

Scala

copyscala -cp <kafka-connect-webds>:<kafka-clients>:\
<kafka-avro-serializer>:<sapjco3> \
org.init.ohja.kafka.connect.webds.sink.WebDSDetailsApp \
<command> <options>

The required Kafka libraries are usually located in:

• /usr/share/java/kafka/
• /usr/share/java/kafka-serde-tools/

Available Commands

• ping: Test the SAP® connection.
• list-webds: List all available sources from the SAP system.
• webds-details: Print meta information of a source.
• extract-schema: Print the avro schema of a source. Extracting Avro schemas requires the presence of io.confluent.kafka-connect-avro-converter in the classpath.

usage:
   WebDSDetailsApp <command> <options>

commands:
   ping
   list-webds
   extract-schema -s <source system> -w <data source name>  
   webds-details -s <source system> -w <data source name>

mandatory options:
   -p <path to connector properties file>