Configuration
The ODP Source Connector package includes detailed configuration documentation, which can be found in the following locations:
Additionally, the connector package includes a configuration template file in the etc/
subfolder:
odp-source-connector.properties
The connector utilizes extensive validators and recommenders and has been tested with the Confluent Control Center UI.
Benefits of Configuring through Confluent Control Center
The ODP Source Connector fully supports configuration through the Confluent Control Center UI. Compared to using property files, configuring through this UI offers several advantages:
- Suggested property values
- A comprehensive list of recommended settings
- Incremental visibility into relevant configuration options
- A wide range of interactive validations
- And more
To maintain optimal performance, the Confluent Control Center UI displays a maximum of 1000 recommendations at a time. Once the required fields for a service entity are filled out in the configuration UI, a new section will appear, allowing you to configure additional service entities.
To see how easy it is to set up the ODP Connector using the Confluent Control Center, watch the screen recordings.
Installation
Configuration
SAP® System Connection
The ODP Connector includes configuration properties from SAP® JCo (prefixed with jco.
). For a detailed description of these properties, please refer to the Java documentation of the com.sap.conn.jco.ext.DestinationDataProvider
interface. The JCo JavaDoc can be found within the sapjco3.jar
package.
A basic configuration for a SAP® JCo client destination is as follows:
# SAP Netweaver application host DNS or IP
jco.client.ashost = 127.0.0.1
# SAP system number
jco.client.sysnr = 20
# SAP client number
jco.client.client = 100
# SAP RFC user
jco.client.user = user
# SAP user password
jco.client.passwd = password
If you’re using SAP’s Message Server for load balancing, replace jco.client.ashost
with jco.client.mshost
.
To ensure encryption and data integrity, use SAP® Secure Network Connection (SNC) properties. The connector also supports various authentication types like user/password, SNC SSO, and SAP® cookie logon tickets.
SAP® RFC Communication Ports
Refer to the TCP/IP ports for SAP® products. Common ports:
33<NN>
(unencrypted)48<NN>
(encrypted)
SAP® Authorizations
To be able to establish communication between the connector and the SAP® system and within the framework of authorization management, you need an SAP® user. The SAP® user must be of type Communications or Dialog. To save SAP® dialog resources, the Communications type is recommended.
Minimum Required Authorizations
-
Object S_RFC:
ACTVT: 16
RFC_NAME: RFCPING, SMLG_GET_DEFINED_GROUPS, RFC_GET_FUNCTION_INTERFACE, RFC_METADATA_GET
RFC_TYPE: FUNC -
Object S_RFC:
ACTVT: 16
RFC_NAME: RFC1, RODPS_REPL, BUS1090
RFC_TYPE: FUGR -
Object S_BTCH_JOB: JOBACTION: RELE
JOBGROUP: * -
Object S_RO_OSOA:
ACTVT: 03
OLTPSOURCE: [odp names]
OSOAAPCO: *
OSOAPART: DATA, DEFINITION -
Object S_TABU_NAM:
ACTVT: 03
TABLE: V_CURC
For more detailed information, see the following SAP® support notes:
SAP® System Requirements
For a comprehensive overview of ODP and a list of supported ODP extractors, please refer to the ODP Introduction and ODP enabled extractors pages.
Here are some crucial SAP® support notes to consider:
- 1931427 - ODP Data Replication API 2.0
- 2481315 - Operational Data Provisioning (ODP)
- 2232584 - Release of SAP extractors for ODP replication
- 3198662 - Supplement of SAP Note 2232584 DataSource Release for ODP
- 2520897 - ODP package size and data load performance
- 2632829 - Switch ODQ Delta data Transfer uncompressed to compressed
- 2195306 - ODP: Visibility of ADSOs in the BW context
ODP Contexts
A context represents a source of ODPs. Context identifiers exist for all SAP® technologies whose analytical views can be exposed as ODPs. Please refer to the relevant links and support notes for information on the specific prerequisites and configuration procedures for your ODP context. Currently, the following ODP-contexts are available (depending on release):
- ABAP_CDS: ABAP Core Data Services Views enabled for analytical data extraction
- BW: SAP® NetWeaver Business Warehouse (for data mart scenario)
- SAPI: SAP® Service Application Programming Interface (S-API) for SAP® DataSources / Extractors without Enterprise Search (ESH)
- HANA: SAP HANA® Information View
- SLT~: SAP® Landscape Transformation Replication Server, see also SLT configuration for ODP
- ESH: Search and operational analytics
- ODP_SELF: Full details about the structure and semantics of ODPs and their associations
ODP Suffixes
For instance, when being used to extract data from SAP® BW InfoProviders, ODP data sources receive a suffix describing the type of data:
Suffix | Meaning |
---|---|
$F | transaction fata/facts |
$P | time-independant master data/attributes |
$Q | time-dependent master data/attributes |
$T | texts |
$H | hierarchies |
$M | joined time-(in)dependant master data view |
ODP Source Configuration
ODP being a publish/subscribe oriented component with potentially multiple subscribers, each subscriber must provide a unique name, referred to as sap.odp.subscriber-name
, specifically for delta extractions. This name serves as the primary key used for tracking pointers that indicate which delta data has already been successfully extracted.
The name
and context
properties must correspond to the ODP name and ODP context configured in SAP®. The available contexts vary depending on the source system. To get a dropdown list of valid values in the Confluent Control Center, a prefix must be entered. A list of possible contexts can be found above in section ODP contexts.
topic
defines the Kafka output topic the connector producer will use to publish extracted data.
# The remaining configs are specific to the ODP source connector.
# Unique subscriber name used to subscribe to ODP sources in SAP.
# This identifier will be used to calculate delta requests.
sap.odp.subscriber-name = OhJaODPKafkaConnector
sap.odp#00.name = Test
sap.odp#00.context = SAPI
sap.odp#00.topic = ODPSAPITEST
Configuration Support with our CLI Tool
The ODP connector’s package includes a command line interface (CLI) to help with various tasks such as:
- Validating connector properties
- Testing SAP® connections
- Listing and extracting available ODPs as well as query details of an ODP
- Extracting schemas for ODP sources
Prerequisites
To run the CLI you need to provide the following dependencies in the CLASSPATH:
- Scala runtime libraries
- kafka-clients
- kafka-connect-odp connector libraries
- SAP® Java Connector 3.1 SDK
Since a Java runtime is provided by the Confluent Platform and the Scala runtime libraries are part of the connector package, executing the CLI with Java would not require installing Scala.
- Java
-
java -cp <kafka-connect-odp>:<kafka-clients>:\ <kafka-avro-serializer>:<sapjco3> \ org.init.ohja.kafka.connect.odp.source.ODPDetailsApp \ <command> <options>
- Scala
-
scala -cp <kafka-connect-odp>:<kafka-clients>:\ <kafka-avro-serializer>:<sapjco3> \ org.init.ohja.kafka.connect.odp.source.ODPDetailsApp \ <command> <options>
The required Kafka libraries are usually located in:
/usr/share/java/kafka/
/usr/share/java/kafka-serde-tools/
Available Commands
ping
: Test the SAP® connection.list-odp
: List all available ODP sources from the SAP system.odp-details
: Print meta information of an ODP source.extract-schema
: Print the avro schema of an ODP source. Extracting Avro schemas requires the presence ofio.confluent.kafka-connect-avro-converter
in the classpath.
usage:
ODPDetailsApp <command> <options>
commands:
ping
list-odp
odp-details -n <object name> -c <context name>
extract-schema -n <object name> -c <context name>
mandatory options:
-p <path to connector properties file>
Avro schemas may differ if the Single Message Transform(s) in the connector configuration are used.