Replace Outliers (HD)
Reduces the range of values for numeric columns.
Information at a Glance
For more information about how the Replace Outliers operator works, see Outliers in Numerical Data.
Input
This operator works for tabular data sets on HDFS. The transformation function can be applied only to numeric columns, and the type of the numeric input columns is preserved in the output.
- Bad or Missing Values
- Any row that contains dirty data, such as a string in a numeric column, is removed as the data is read in. After the data is read in, the operator filters out all rows that contain null values in the selected numeric columns. Rows that have null values in any of the columns not selected are not removed. The rows removed are reported in the Summary tab. If the value of Write Null Data to File Parameter is set to yes, then the rows removed because they have null data are written to an external file (the location of which is reported in the Summary tab).
Restrictions
Any data set with numeric columns can be used. This operator slows down as the number of columns selected and the cardinality of the columns increases.
Configuration
Parameter | Description |
---|---|
Notes | Any notes or helpful information about this operator's parameter settings. When you enter content in the Notes field, a yellow asterisk is displayed on the operator. |
Columns | Numeric columns to transform. |
Lower Boundary (%) | A double that represents the percentage of values in the left tail of the distribution (on the low end of the range in each column) to replace. |
Upper Boundary (%) | A double that represents the percentage of values in the right tail of the original distribution for each column (the high end of the range in each column) to replace. |
Write Null Data To File
*required |
Rows with null values are removed from the analysis. This parameter allows you to specify that the data with null values be written to a file.
The file is written to the same directory as the rest of the output. The filename is suffixed with _baddata.
|
Storage Format | Select the format in which to store the results. The storage format is determined by your type of operator.
Typical formats are Avro, CSV, TSV, or Parquet. |
Compression | Select the type of compression for the output.
Available Avro compression options. |
Output Directory | The location to store the output files. |
Output Name | The name to contain the results. |
Overwrite Output | Specifies whether to delete existing data at that path. |
Advanced Spark Settings Automatic Optimization |
|
Output
- Visual Output
- The operator has two tabs of output. The first is the output data, which is passed on to the next operator. The second is a summary that explains which parameters were selected, how much null data was removed, and where the results were written.
- Data Output
- The operator outputs the same tabular data set as the input data, but with some of the values in the selected numeric columns replaced. See Outliers in Numerical Data for more information. This output should work as input for any Hadoop operator that expects tabular data.