Spark Cluster Connection Prerequisites

The following Spark cluster configuration must be completed for the New Workflow.

The TIBCO® DV and the nodes of the Spark cluster must have a shared file system with a unique and consistent name. Only the system administrator can make these changes.

The shared file system can be HDFS, NFS drive, or Amazon S3 bucket. If you are using HDFS shared drive, you must have access to the shared file system for changing the configuration.

For example, if the name of a NFS shared file system is tds, then you can interactively mount as follows:

sudo mkdir -p /mnt/tds
sudo mount some.host.net:/path/to/volume /mnt/tds
sudo chown -R someOtherUser /mnt/tds

Here, the root is the only Linux user allowed to perform the mount (and unmount). Therefore, you must change the ownership of the mount point someOtherUser after the mount is complete. The someOtherUser is the Linux username (or user ID) that owns the Spark worker process (or the Hadoop YARN node manager process). If the owner is not changed, then the TIBCO Data Virtualization reader/writer components cannot access the mount point.