Running Asynchronously

While it might be simple to run PDriver synchronously, waiting for the results to return might be impractical in a production scenario, especially if each task takes time to complete. An alternative is to run the PDS script asynchronously, using the bsub flag. This runs the entire script except for post-job processing. Instead of waiting for tasks to complete, it immediately returns a batch ID, which can later be used to fetch results.

Let’s modify the above simplejob.pds to add a postjob block with a hypothetical command. Here, it is another simple log command, but in a real job, it might be a command that copies, collates, or otherwise post-processes task results into a final result. Let us call this script simplejob2.pds:

job simplejob
    
task 10
    log "hello, world"
end
postjob
    log "job processed."
end
end job

Run this script by typing the following command:

pdriver -bsub simplejob2.pds

In the Driver logs displayed on the screen, there are lines that look similar to the following lines:

03/02 14:54:16:265 Info: Batch job has been submitted
03/02 14:54:16:265 Info: Collect outputs using batch id: 61924497789921878

The batch ID is important; use this to retrieve your results. In a production scenario, you can extract this ID from the Driver log file with grep or some similar approach.

After the job has been submitted, it runs on its own, and the results are queued. You can retrieve the results when you are ready by typing the following command:

pdriver -bcoll 61924497789921878 simplejob2.pds