Installing the TIBCO® Metadata Agent

Installation and configuration

TIBCO® Metadata agent is a Java application provided as a jar file and configured using a properties file. Its main purpose is to establish interactions between the source system instance (through specific adapter) installed on a server and the EBX® Metadata Management application. One agent can be connected to one instance of a source system. The EBX® Metadata Management application supports connections to multiple agents that manage different data sources.

An agent can have one or multiple adapters to support metadata extraction from different data sources. An adapter is a separate Java application that ensures one or several of the metadata capabilities for a given data source type(s):

• Data source management

• Metadata harvesting

• Data sampling

• Metadata provisioning

• Lineage extraction.

Prior to agent configuration, you must create a technical user from the Administration perspective. For more details, see Technical Users.

Note: the technical user created is automatically prefixed with ebx-.

Prerequisites

The TIBCO® Metadata Agent requires:

• JRE 11 minimum

If the TIBCO® TDV Metadata Adapter is used (see TIBCO TDV Metadata Adapter installation):

• TIBCO® Data Virtualization version 8.3 minimum

Security and Cloud Access

In order to access to EBX® (CIC Cloud), and to allow the EBX® Metadata Management application to access to the TIBCO® Metadata Agent, you must take into account the following infrastructure considerations:

• TIBCO® Metadata Agent hosting server must be opened to the Internet.

• It is recommended to define a DNS Name for the TIBCO® Metadata Agent hosting server, in order to ease certificate management.

• TIBCO® Metadata Agent hosting server must be reachable via HTTPs (HTTP over SSL), using a CA-signed certificate (no self-signed certificate).

• TIBCO® Metadata Agent external port must be port 443 (default HTTPs port), using a reverse-proxy (IIS, Apache, Ngninx, …). (ma.http.ws-url has to be set to the reverse-proxied URL to the agent, reachable from Internet).

• TIBCO® Metadata Agent internal port can be configured on any available port on the machine (see ma.http.port in Agent configuration properties), and use in the reverse-proxy configuration to expose the TIBCO® Metadata Agent to Internet.

Installation process

The following steps describe how to get the TIBCO® Metadata Agent running on your system:

1. Download TIBCO® Metadata Agent package from the Metadata Administration perspective.

2. Unpack it on your system.

3. Fill in properties template as described below in the Properties configuration section. Please note that for some properties it is required to log into EBX® and/or obtain some information from a EBX® administrator.

4. Download and install the EBX® Metadata Management application Adapters required for your source systems.

5. Fill in properties template for each adapter.

6. Activate required adapters in the main Agent properties template.

7. Launch TIBCO® Metadata Agent as described below in the Execution section.

Packaging

The agent is available as a standalone Java application (jar file). You can download it from the Metadata Administration perspective.

Properties files

The main packaging of the TIBCO® Metadata Agent includes following properties files:

• application-metadata-agent.properties for configuration of Agent

• application-datasources.properties for configuration of the supported data source types per adapter

• application-tdv.properties (optional) for configuration of the TIBCO® TDV Metadata Adapter.

TIBCO® Metadata Agent properties template

######################################################
###################################################### 
##### ##### ##### Metadata connection ##### ##### #####
######################################################
######################################################
ma.metadata.context=[[METADATA BASE URL]] 
ma.metadata.username=[[METADATA TECHNICAL USER NAME]]
ma.metadata.password=[[METADATA TECHNICAL USER PASSWORD MAY BE ENCRYPTED]] 
ma.metadata.public-key=[[METADATA TECHNICAL USER JWT]]
ma.metadata.label=[[LABEL FOR THIS AGENT]] 
#### 
#### Uncomment this line if you want to use encryption for agent passwords. 
#### In this case, you will have to encrypt password using TIBCO Metadata Agent Tools 
#### Read the documentation for more information 
#### 
#ma.metadata.useEncryption=true

######################################################
###################################################### 
##### ##### ##### Agent properties ##### ##### #####
######################################################
###################################################### 
ma.http.port=[[AGENT PORT]]
ma.http.ws-url=[[AGENT CONTEXT URL]]

######################################################
###################################################### 
##### ##### ##### Agent log properties ##### ##### #####
######################################################
###################################################### 
#Folder where the logs will be stored 
ma.logging.path=./logs 
#Log level for technical logs (Spring framework, external libraries). 
ma.logging.level.root=INFO 
#Log level for the Agent's functional logs. 
ma.logging.level.agent=INFO 
#Log level for the SOAP messages. Set to TRACE to see full request body. 
ma.logging.level.soap.sent=INFO
#Log level for the SOAP messages. Set to TRACE to see full response body.
ma.logging.level.soap.received=INFO

######################################################
###################################################### 
##### ##### ##### Agent proxy properties ##### ##### #####
######################################################
###################################################### 
#### 
#### Uncomment this line if you want to use a proxy configuration for the agent 
####
#ma.proxy.useProxy=true ma.proxy.proxyHost=[[PROXY HOST]]
ma.proxy.proxyPort=[[PROXY PORT]] 
ma.proxy.proxyAuthentication=[[PROXY
AUTHENTICATION ?]] 
ma.proxy.proxyUsername=[[PROXY USERNAME]]
ma.proxy.proxyPassword=[[PROXY PASSWORD]]

######################################################
###################################################### 
##### ##### ##### Agent Adapter properties ##### ##### #####
######################################################
###################################################### 
#Comma-separated list of packages to scan for adapters classes
agent-adapter.base.package=com.tibco.ca.tabula.agent.tdv 
#Comma separated list of profiles (corresponding to adapter technical names)
agent-adapter.active-profiles=tdv

Properties configuration

Prior to launching the TIBCO® Metadata Agent, you need to complete the information in the properties file to check the connection to the EBX® Metadata Management application.

Users are strongly discouraged to modify properties that are not described in the table below but are present in the properties file, especially specific URLs configured for EBX® Metadata Management application operation or the H2 database JDBC URL. A good practice is to remove old specific properties from your prior-to-version 4 agent, in order to keep only specific properties in your configuration file.

Once these properties are configured, you can launch the agent as a Java application, using command line or any other suitable tool (see below on this page).

The configuration file should be named application.properties.

Upon the first launch, the agent will query the EBX® Metadata Management application REST API for registration and will obtain a GUID to be used in all information exchanges between the two applications. Check that in the agent’s log file after startup you see the message "Started MetadataAgentApplication in xx seconds." This indicates that the start up was correct and the agent is running. If you don’t see this message, your need to check the log file for error messages explaining the issue preventing agent to start.

Once registered, the agent will be displayed in the EBX® Metadata Administration perspective as described TIBCO® Metadata Agents.

Data source types properties file

The properties file application-datasources.properties allows the agent to make connection between supported data source types, metadata management functions and the adapters to use.

Properties in this file must follow the pattern: datasource.operation_name.data_source_type=adapter_profile Supported operation names are as follows:

• dsmanagement

• harvesting

• sampling

• provisioning

• lineage.

Data source type name must be formatted as alphanumeric name all in lower case without any special characters. You can also define a label for data source type that will be displayed in the EBX® Metadata Management application. Only one adapter can be defined for operation - data source type.

Below is the configuration pattern:

datasource.label.<data source type>=
datasource.dsmanagement.<data source type>=
datasource.harvesting.<data source type>=
datasource.sampling.<data source type>=
datasource.provisioning.<data source type>=
datasource.lineage.<data source type>=

By default this file includes configuration for the data source types supported by TIBCO TDV Metadata Adapter installation.

EBX® Metadata Management application Connection Encryption

If ma.metadata.useEncryption is activated, then ma.tdv.datasource.password, ma.tdv.soap.password, ma.metadata.password and ma.proxy.proxyPassword must be encrypted into application.properties before starting the agent:

• As a prerequisite, ma.metadata.public-key must be defined in application.properties according to the corresponding EBX® Metadata Management application Technical User.

• Use TIBCO® Metadata Agent Tools to encrypt the original password. Launch in a terminal:

  java -jar metadata-agent-tools.jar MY_PLAIN_PASSWORD

  See the output sample below.

• Copy-paste the encrypted values into application.properties for each corresponding password property.

The output provided is as follows:

`**************************`
`Value to encrypt is MY_PLAIN_PASSWORD`
`**************************`
`Encrypted value is : I7UOEPOHtF+J4EYPTPRxR0++QG4hoXFe87uSPluI1lbi`
`**************************`

Caution: with this feature enabled, an encrypted password must be re-encrypted on each renewal of JWT key.

Agent execution (as a Standalone Java Application)

You have to configure a launcher to execute the agent, adding the configuration files into a config directory under the directory where is placed the agent.

logs and storage directories are created by the agent itself directly under the structure.

Under storage, two files are created, one for agent local database (configDB.mv.db), one for scheduling ( quartzDB.mv.db). If you have previously used different names for these files in your previous configuration, please rename these files accordingly in your file system, after stopping the current agent.

To launch the agent, you can create a shell file as follows (in this example the jars are placed in a /metadata-agent directory):

You have to define the default encoding as UTF-8 for the JVM using -Dfile.encoding=UTF-8, if not already set.

Example on Linux (with TDV adapter activated): java -Dloader.path=metadata-agent-tdv-adapter.jar,metadata-agent-tdv-driver-jdbc.jar -Dspring.profiles.active=tdv -jar metadata-agent.jar >> logs/metadata-agent.log 2>&1 &

Example on Windows (with TDV adapter activated): java -Dfile.encoding=UTF-8 -Dloader.path=metadata-agent-tdv-adapter.jar,metadata-agent-tdv-driver-jdbc.jar -Dspring.profiles.active=tdv -jar metadata-agent.jar > /logs/metadata-agent.log 2>&1

Once started, an application.pid file is created containing the process id of the agent. You can stop the agent by killing this process. kill $(cat application.pid)

The agent can be also shut down using a HTTP POST request on a dedicated endpoint: curl -X POST http://<agent_host>:<agent-port>/manage/shutdown

Troubleshooting

The following table provides causes and resolutions to two possible errors: