Creating a Runtime Class

Finally, you must define what happens when the user clicks Run from the application.

In our case, we just want to submit a Spark job. Because of this, our Runtime class appears to be empty, but the base class that we extend from does the work of launching the Spark job.

Prerequisites

You must have created the onInputOrParameterChange method.

Procedure

  • Add the following code: 
    class SimpleDatasetGeneratorRuntime extends SparkRuntimeWithIOTypedJob[SimpleDatasetGeneratorJob,
  IONone, HdfsTabularDataset] {}

    Let's break down what all of this means. Our class, called SimpleDatasetGeneratorRuntime, extends SparkRuntimeWithIOTypedJob. This base class allows us to launch a Spark job without having to write any extra code to handle that. This makes it very easy to integrate your own Spark code into the Team Studio platform. As type parameters, you'll pass three things:

    • SimpleDatasetGeneratorJob - The name of your Spark job, which we implement below.
    • IONone - The input type. Because we are building a source operator, it has no input.
    • HdfsTabularDataset - The output type. Our operator returns a file on HDFS with the new data it generates.
    Note: The data type of the input and output match what we defined in the GUI Node class, and both of them extend IOBase.
Next topic: Creating a Spark Job
Previous topic: Creating the onInputOrParameterChange Method