Welcome to the
HPC Help Desk Video Series “Using Job Arrays on HPCMP
Job Schedulers.”
This presentation will describe the advantages
of creating job arrays to combine many small jobs into one.
and how this benefits the entire HPCMP user community.
It will present script examples and describe how the process
can be executed on any of the job
schedulers currently used in the program.
In the link to supplemental materials accompanying this video.
you will find a Job Aid that includes all the URLs. phone
numbers and email contacts referenced in this presentation.
When you see this graphic and hear this sound.
it means the information is contained
in the accompanying Job Aid.
The High Performance Computing Modernization Program
values all the research
being done on its allocated systems. large and small.
The responsibility of managing these thousands of daily jobs
falls on job schedulers of the various HPC systems.
Unfortunately. users sometimes
fill the batch job queues with hundreds
of small one node scripts at once.
When many users do this over a short period of time.
The queues get clogged with thousands of jobs.
To clear this jam.
the schedulers begin
prioritizing the small jobs and delaying the larger jobs.
To make for a more equitable distribution of resources.
the program encourages
these small job users to incorporate job arrays.
which are simple commands
that can combine
many single focus
scripts into one larger. more efficient script
that delivers the same data outcomes
without monopolizing valuable compute resources.
Job arrays benefit you. the user.
by grouping
many closely related jobs into a single master script.
If these jobs share mostly common tasks and only vary
by small portions. depending on the job.
they are good candidates for a job array.
Current HPCMP systems use PBS. Slurm or LSF job schedulers.
All operate in a similar manner
with most of the differences being in command syntax.
The Job Aid accompanying this presentation
contains a link to a scheduler Rosetta.
which identifies common operations
and how they are implemented in each scheduler.
Let's look at the Step Directive for job arrays.
This example is shown in PBS format.
The PBS scheduler uses the directive
#PBS -J n-m: step
n is the starting index.
m is the ending index and the step is the step size
for each iteration.
This informs PBS this is a job array.
The environment variable
$PBS_ARRAY_INDEX
will contain the current index instance
your script is in.
The command echo $PBS_ARRAY_INDEX
will output the current job
index in which the script is executing.
When you submit a PBS job array. the PBS
job number will have left and right brackets appended to it.
A job that would normally look something like 384294
would now look like 384294[]
in the PBS qstat command.
Here's a PBS job array script that will run five
MPI jobs on an HPC computer within one submission.
It is a typical PBS script until line seven
where we see the Job Array directive that signifies a job array
from 0 to 12 in steps of three.
Line 13 prints
the job array index. which might look like 384294[].
To isolate the job ID. cut it out of PBS_JOBID
using the cut command
to select the portion before the open bracket.
Line 19 creates a directory with the current job ID
line 21 cd’s into that directory.
Line 23 makes a directory for the current job array index.
Line 25 creates a variable containing the full
path for this run.
Line 27 copies the executable.
In our example. a pi calculator
named picalc.exe into the temporary directory
we just created.
Line 29 cd’s into the directory
that will contain all output from this PBS array index.
And finally. line 31 runs
the job and redirects the output.
Running qstat -sw
after submitting the job
will show the job queued in the scheduler.
If you run qstat -sw
during the run. you'll see a display like this.
Notice that PBS places a letter B rather than an R
in the status column.
You should note that the indices might be executed out of order.
In other words. job array 12 could run before number three.
After the job is complete.
manually running ls in the job
directory will show five directories.
Each directory will contain the picalc.exe
executable and the output from that program as follows:
PBS only provides a single parameter index.
but you can partition the single index into as many indices
as you like.
In this example we will use two indices. one for x
and one for y. This script goes from 0 to 1199 into x
and y indices that vary from 0 to 29 for x and 0 to 39 for y.
Again. line seven signifies a job array.
Lines 14 and 15 specify the values
for x and y.
Submitting this script to PBS will produce 1200 files.
The output for the first few files would look like this:
Remember that the jobs are not necessarily executed in numerical
order.
Earlier we mentioned
it is possible to convert scripts
from one job scheduler format to another.
There is a batch scheduler Rosetta on the centers.hpc.mil website.
which will help you translate the user commands into formats
PBS. Slurm and LSF will execute.
There are other Help Desk
videos available covering the basics of specific job schedulers
on the centers.hpc.mil web site.
Thank you for taking the time to view this instructional video.
If you have any questions or experience any problems.
please contact the HPC Help Desk.
The phone numbers. email address. and HPC Help Desk
Service Portal URL information is shown here
and can also be found in the accompanying Job Aid.