Row Filter
This operator sets the criteria for filtering data set rows. Only the rows that meet the criteria remain in the output data sets.
Information at a Glance
Note: This operator can only be used with TIBCO® Data Virtualization and Apache Spark 3.2 or later.
|
Parameter |
Description |
|---|---|
| Category | Transform |
| Data source type | TIBCO® Data Virtualization |
| Send output to other operators | Yes |
| Data processing tool | TIBCO® DV, Apache Spark 3.2 or later |
You can specify row filters in the following modes:
- Simple mode: Use a simple template to define the filter, choosing a column, an inequality (for example, ">" or "between"), and a value (for example, a literal value or a column expression).
- Script mode: Specify any set of filters by using a script.
Note: Column names in the script mode must be enclosed by a backtick (` `) if the column name contains special characters or multi-byte characters such as Chinese, Japanese, or Korean characters. Do not use double quotes to enclose the column names.
Input
An input is a single tabular data set.
Bad or Missing Values
Null values are not allowed and result in an error.
Configuration
| Parameter | Description |
|---|---|
| Notes | Notes or helpful information about this operator's parameter settings. When you enter content in the Notes field, a yellow asterisk appears on the operator. |
| Filter | Specify the filters for the operator. See the Define Filter dialog
for more information. Note: The SQL
expression must adhere to the SQL syntax supported by Spark since this is Spark SQL.
|
| Output Schema | Specify the schema for the output table or view. |
| Output Table | Specify the table path and name where the output of the results is generated. By default, this is a unique table name based on your user ID, workflow ID, and operator. |
| Store Results | When set to Yes, the operator saves the results. If set to No, the operator does not save the results. |
Output
Visual Output
A table that displays the output (data rows) of a data set.
Data Output
A tabular data set of the newly created table or a view.
Example
The following example displays the data set rows created from data fields of the given data set, forming a new table using the Row Filter operator.
Data
golf: This data set contains the following information:
- Multiple columns namely outlook, temperature, wind, humidity, and play.
- Multiple rows (14 rows).
Parameter Setting
The parameter settings for the golf data set are as follows:
-
Filter: All conditions must be met
-
outlook, like, '%nny'
-
temperature, >, 80
-
-
Store Results: Yes
Output
The following figure displays the output for the parameter settings for the golf data set.
