Convert to Spark/Revert to Non-Spark
If you have a workflow that you created before version 6.4, and if operators in the workflow now can use Spark (node fusion operators), you can set the operators all to use Spark, without setting the option individually for each operator. Likewise, if you want to revert a workflow to its pre-version 6.4 status, where no operator uses Spark, you can do that without setting the option individually for each operator.
By default, existing workflow operators have the option to use Spark set to
no, which ensures no upgrade impact. New operators in existing or new workflows have the default for
Use Spark set to
yes.
- After your update, if you want to update all of the available node fusion operators to
Use Spark = yes, then from the top-right menu in the workflow, click
. When you set this option, you are ready to run your existing workflow using Spark Node Fusion.
Note: For the Row Filter and Variable operators, advanced syntax uses Spark SQL, rather than Pig, if Use Spark is set to yes.
- To revert to non-Spark versions of all operators, from the top-right Workflow menu, click . Doing so reverts the workflow to its pre-6.4 state.
Related concepts
Copyright © 2021. Cloud Software Group, Inc. All Rights Reserved.