Configuration Properties for Apache Spark

By default, the Apache Spark configuration properties are defined in the Configurator to use the big data import functionality.

Change the values of the following properties as per your requirement:
Property Name Description Default Value
Spark Import App ( com.tibco.loader.sparkappname) Application name to import records through Apache Spark cim_spark_App.
Spark Master URL ( com.tibco.loader.sparkmasterurl) The URL for the Apache Spark master node spark://localhost:7077.
Partitions in SparkApp( com.tibco.loader.sparkpartitions) Number of partitions required for the Apache Spark import application 500.
HDFS URL ( com.tibco.loader.hdfsurl) URL to the Hadoop Distributed File System (HDFS) that provides high-throughput access to application data hdfs://localhost:9000.
Database Host Name (com.tibco.cim.database.host) Database host name. Default value: localhost.

Change to database host IP.

Database Port Number (com.tibco.cim.database.port) Database port number. Default value: 5432.

Change to database port number.

Copy the required JAR files to the $MQ_HOME/lib/external directory.

Optional: If TIBCO MDM and Apache Spark are installed on different machines, enable the discovery of remote nodes in the IgniteMember.xml file from $MQ_HOME/config.

Even after configuration of Apache Spark with TIBCO MDM, if you face any issues when performing the big data import operation, see the "Big Data Import Errors" section in TIBCO MDM System Administration to resolve the issues.