File Manipulation

Several built-in statements enable you to manipulate files and directories on the Driver or Engines. These can be used to move input and output files to and from the Engine.

The copy statement copies a file. Typically, the prejob block is used to copy input files from the Driver machine to the staging directory on the Manager, which can be referenced as $DSSTAGEDIR. In the task or pretask block, one can then copy that file from the staging directory to the Engine’s local filesystem. Output files can be copied in the reverse direction.

You can remove files from the Driver or Engine using the delete statement. The wildcard * is supported.

Similarly, the mkdir and rmdir statements create or remove directories. To use rmdir, the directory must be empty.

The file manipulation statements are not OS-dependent and pathnames are automatically translated to work on the appropriate platform. For example, mkdir "sample/log1" creates sample\log1 on Windows systems.

The following script uses files for standard input, output, and errors. It essentially copies everything from the input file to the output file.

job simplefile
prejob
    copy "hello.txt" $DSSTAGEDIR
end
task 1
    copy "$DSSTAGEDIR/hello.txt" $DSWORKDIR
    execute stdin="$DSWORKDIR/hello.txt" stdout="$DSWORKDIR/output.txt" stderr="error.txt" 
        "sort"
    copy "$DSWORKDIR/output.txt" $DSSTAGEDIR
end
end job