Convert to Spark/Revert to Non-Spark
If you have a workflow that you created before version 6.4, and if operators in the workflow now can use Spark (node fusion operators), you can set the operators all to use Spark, without setting the option individually for each operator. Likewise, if you want to revert a workflow to its pre-version 6.4 status, where no operator uses Spark, you can do that without setting the option individually for each operator.
By default, existing workflow operators have the option to use Spark set to
no, which ensures no upgrade impact. New operators in existing or new workflows have the default for
Use Spark set to
yes.
Before you beginTo access the Actions menu, you must have a workflow open on the workflow canvas.
You must have updated your workflows to 6.4 or later.
- To update all of the available node fusion operators to
Use Spark = yes:
From the Actions drop-down menu, click Convert to Spark. When you set this option, you are ready to run your existing workflow using Spark Node Fusion.Note: For the Row Filter (DB) and Variable (DB) operators, advanced syntax uses Spark SQL, rather than Pig, if Use Spark is set to yes. - To revert to non-Spark versions of all operators:
From the Actions drop-down menu, click Revert to Non-Spark. Doing so reverts the workflow to its pre-6.4 state.