Export to File Storage

Writes data from upstream operators to file storage (for example, to AWS S3).

Information at a Glance

Note: This operator can only be used with TIBCO® Data Virtualization and Apache Spark 3.2 or later.

Category Tools
Data source type TIBCO® Data Virtualization
Send output to other operators No
Data processing tool TIBCO® DV, Spark

Algorithm

This operator provides the ability to write data from upstream operators to file storage (for example, to AWS S3) in Parquet format. The default directory path of the file storage is the path configured in the TIBCO Data Science - Team Studio configuration file (Alpine.config). The resulting table is published in TIBCO® DV to the schema specified.

The operator is limited by the cluster resources and the size of the Spark data frame.

Input

A single tabular dataset.

Configuration

Parameter Description
Notes Notes or helpful information about this operator's parameter settings. When you enter content in the Notes field, a yellow asterisk appears on the operator.
Output Schema The TIBCO® DV schema the dataset is published to. Users can access the data in "Output Schema"."Result Table Name".
Result Table Name The name of the dataset stored in file storage. No white space is allowed in the file name.

Output

Visual Output
Output: The dataset to write.
Summary: Displays the full path to the file storage where the data is written and the TIBCO Data Virtualization table schema to which the data is published.
Data Output to Successive Operators
A database table to use in a downstream operator.

Example

Description

This example applies row filter operators on the dataset and uses the Export to File Storage (TDV) operator to write the data to AWS S3.

example workflow for Export to File Storage (TDV)

Data

  • Crabs: A dataset with crab species and demographic data.

    • 173 rows

    • 7 columns

Parameters

  • Output Schema: Compute_S3

  • Result Table Name: alp@user_id@flow_id@....
Results: Output

output panel for Export to File Storage (TDV) example

Summary

Writing to s3://tdv-test/output/alp_206_Export_to_File_Storage_TDV__OP2.parquet

Publishing to "Compute_S3"."alp3_206_Export_to_File_Storage__TDV__OP2"