Building Your First Custom Operator in Scala

Use the Custom Operator framework to create new functionality for your Team Studio workflows. We have provided a set of samples in Scala and Java to showcase some of the things you can do and describe the format an operator takes.

To get started writing your own custom operator in Scala, perform the following tasks.

• Install and configure the Required Tools.
• Review the structure and build instructions for a simple example called Column Filter.

You can find the SDK with examples on the Team Studio Github page. For more in-depth information about the available API methods, see the relevant Scaladoc in Installing the Custom Sample Operator for your Version.

1. Setting Up Your Environment
   You can use the IntelliJ IDEA to create a custom operator for Team Studio. Prepare the development environment to hold your project.
2. Creating the Signature Class
   After you set up the development environment, write the code for the operator. A custom operator must implement three classes: the signature, the GUI node, and the runtime. Each class defines behavior and information that the Team Studio workflow engine uses to execute the operator.
3. Creating the Constants Object
   Create a Constants object to keep some values. You can refer to variables across the operator while making it easier to maintain and update.
4. Creating the GUI Node Class
   Implement the GUI Node, which represents the design-time configuration. The GUI Node defines configuration parameters that are displayed to the user, describes input and output schemas, and can be used to design visualizations for the results.
5. Creating the onPlacement Method
   Regardless of whether you are using the Spark data frame template, you must override the onPlacement() method.
6. Building the Operator Dialog Box
   We use the operatorDialog object to define parameters for our operator. These parameters can be text boxes, radio buttons, or any other basic form element.
7. Defining the Output Schema
   To ensure our output displays in a consistent fashion, we define the output schema for our data frame.
8. Creating a Runtime Class
   Now that you have created a signature and a GUI node, it is time to create the runtime. This is where the bulk of the operator's action occurs; you define the Spark job that performs the data transformation here. When the operator is executed, a data frame with the selected columns is returned from the Spark job. After the job finishes, the SparkDataFrameJob class saves the results as a file using the storage parameters defined in the onPlacement() method. If you did not specify, this defaults to storing on HDFS. This information is then passed to the Team Studio plugin engine so that your results can be visualized.
9. Preparing Your Operator to Build
   After you complete your code, there are only a few things left to do before you upload it to the Team Studio web application. You must edit two more files - pom.xml and plugins.xml. You can also customize the icon that is displayed for your operator at this point.