Welcome to the HPC Help Desk Video Series. This video will
describe how your jobs on some HPCMP supercomputers
are managed with Portable Batch System, or PBS, scripts and
interactive job management commands.
In the link to supplemental materials accompanying this video
you will find a Job Aid containing all the URLs,
phone numbers and email contacts
referenced in this video.
When you see this graphic and hear this sound,
it means the information is contained in the accompanying Job Aid.
Please note:
This video refers only to systems which use PBS scripts for job
management. Other HPC job schedulers
such as SLURM and LSF are not
covered in this presentation.
Please contact the HPC Help Desk
if you need assistance with these applications.
This video will first provide an overview of PBS
and describe the job queues
and file systems available.
Next, it will describe the structure of a PBS script
needed to submit your jobs
using examples found in the systems’ PBS Guides
available on the center’s web site.
And last, it will introduce the most common interactive
PBS commands used to manage your job.
The PBS workload management system allows you to make more
efficient use of your time when ready to run your programs.
You specify the tasks you need executed
in the form of a batch job script.
PBS then takes over, running these jobs and returning the results
to you when the job is completed.
If a requested resource, such as a particular software license, is not
immediately available, PBS holds your work
and runs it when the license can be obtained.
The HPCMP supercomputers have different types of queues
where batch jobs are are held temporarily,
waiting for the required resources to become available.
Each queue is assigned a priority from highest to lowest.
The order varies from system to system—a priority chart is included
in the user guide for each supercomputer
on the centers.hpc.mil website.
The queues available to you are determined by the projects
to which you are associated.
Shown are four of the common queues available to users.
Standard is for normal, day-to-day jobs. Jobs that run
in Standard queue are charged against your project allocation.
Debug is for debugging jobs. This queue is given a high priority
on some systems,
but is limited to short periods of time.
In the Background queue, though it has the lowest priority,
jobs are not charged against your project allocation.
You might choose the background queue for several reasons:
You don’t care how long it takes for your job to begin running;
You are trying to conserve your allocation;
Or, you have already used up your allocation.
Two other specialized queues,
the Transfer and HPC Interactive Environment,
or HIE queue, will be covered later in this presentation.
Most queues have restrictions such as which users
can utilize them,
the maximum wall clock time the job can run,
and maximum number of cores on the job.
To help you determine which to use, each systems’ specific queues,
priority, and limits are provided in tables from their User Guide
located on the centers.hpc. mil website.
It is helpful to know a few basics about the file systems
on the HPC computers in order to write efficient job scripts.
Each file system is configured for a certain purpose.
The $HOME file system is used to store and edit
the files needed to start jobs such as job scripts and input files.
The $HOME file system is designed to make files
immediately accessible,
so PBS jobs can start without delay.
However, $HOME has limited storage space based on your quota.
The $PROJECTS_HOME file system is similar to $HOME.
It is intended for sharing files such as source code, makefiles,
executables, and data files.
$WORKDIR is also specific to a system.
It provides high bandwidth for running PBS jobs.
However, this is only temporary storage and it is not backed up.
Files left here are typically maintained for up to 21 days
after they were last accessed
by the user, and may be purged after that if space runs low.
The Archive System is where you will move files
you need to retain for longer periods of time
for longer periods of time.
Each DSRC has its own separate archive system.
The Center Wide File System (CWFS) is accessible from all
the HPC system login nodes located at that site.
These files are only maintained temporarily,
typically a maximum of 120 days,
and like $WORKDIR, are not backed up.
The job you run is a PBS script.
This video will review each of the components
of your PBS script and point you toward available
documentation to help you.
Each PBS script first specifies the UNIX shell.
Next, directives specify characteristics of the job,
and provide information on resources your script needs.
There are required PBS directives which are explained in detail
as well as directives that are optional, but useful.
Following your PBS directives is the execution block of your script.
This section will include the launching of the job.
After your programs run, clean up instructions,
such as archiving files you no longer need
are specified.
Shown here is a simple PBS batch job.
You can find sample scripts like this in the User Guide section
of the system you want to run your job on.
A batch job is a shell script containing the set of commands
you want to run.
While you can create a script from scratch,
most users copy one already created
and modify it to fit the particulars of their job.
Each center maintains a Sample Code Repository.
The environment variable $SAMPLES_HOME
points to the repository’s base directory.
Each repository contains an index file
that details exactly what is in the repository.
An expanded description of $SAMPLES_HOME
can be found on the center’s web site
at the URL shown here and found on the video job aid.
Next, we will walk through the components of a typical script.
These components are detailed in the system-tailored PBS Guides
created for each HPCMP system,
accessible from the Systems tab of the Centers website,
as shown here.
Click on a system, then the “more info” link.
A link to the system’s PBS Guide
will be listed under the “Available Documentation” column.
It’s good practice to specify which shell your script is written in.
If you don’t, PBS will use your default login shell to run your script.
To tell PBS which shell to use, start your script with a line
similar to the one shown here:
B A S H stands for the GNU Bourne-again shell “bash”.
Other shell options are the Bourne shell, sh
the C shell csh
The korn shell ksh
The extended C shell tcsh
The Z shell zsh
The next part of your script contains PBS directives.
Directives tell PBS about the resources your job needs,
using a special form of comment that begins with the string:
hashtag PBS. PBS directives must come before the first line
of executable code in your script, otherwise they will be ignored.
Highlighted here are the PBS directives for this sample job script.
They are split into two groupings:
those that are required in every script,
and additional directives that are optional.
Before PBS can schedule your job,
it needs to know how many nodes you want
and how many processes you want to run on each of those nodes.
The format is the dash “l” option, and that is a lowercase “l”,
followed by the “select” directive where
the number following the select statement is
the number of nodes you are requesting.
A colon separates each of the variables;
“ ncpus” is the number of physical cores available on each node;
And the last item in the list,
“mpiprocs” is the number of processes per node.
In the System documentation on the centers website,
a Node Configuration table
for each system provides Cores per Node, which provides the value
for the number of physical cores available on each node.
Next, PBS needs to know how long your job will run.
The wall time directive is used for this. Use the dash l option
followed by the walltime directive, equals,
and your walltime estimate.
You have to make an estimate keeping these factors in mind:
Your estimate is a limit. If your job has not completed
within your estimate, the job will be terminated.
Your estimate will affect how long your job waits in the queue.
In general, shorter jobs will run before longer jobs.
Each queue has a maximum time limit.
You cannot request more time than the queue allows.
On the centers.hpc.mil web site, Systems menu,
the Queue Descriptions and Limits table
for each system provides the maximum wall clock time by queue.
Refer to this table to ensure your walltime specification
does not exceed the maximum.
The next required PBS directive is to state
which queue the job is to run in.
The format of this directive uses the dash q option
followed by the queue name.
Your options are determined by your project.
Most users have access to the debug,
standard, and background queues, as well as the Transfer queue
and the HPC Interactive Environment, or H-I-E.
The Transfer queue is meant for efficiently transferring data
to or from the HPC systems,
within a given center, as part of a PBS job.
The HIE is intended to provide high availability and response
supporting Remote visualization;
Application development for Graphics Processing Units ( or GPUs);
Application development for other non-standard processors
within a particular system.
The following functions are available as well:
Interactive use, including:
• Debugging • Complete pre- and post- processing
• Building applications • Remote visualization tasks.
Batch processing during low-usage times,
and access to specialty nodes such as GPUs and Large Memory.
Please note the HIE is intended to provide
higher response rate in support of interactive workloads
as well as application development and debugging.
There are a limited number of nodes available to the HIE
and they should be reserved for appropriate use cases.
The use of the HIE for regular batch processing
is considered abuse and will be closely monitored.
The HIE should not be used simply as a mechanism
to give your regular batch jobs a higher priority.
Other queues exist, but access is restricted to projects
that have been granted privilege due to urgency or importance.
PBS needs to know which project ID to charge for your job.
You can use the show_usage command to find the projects
that are available to you and their associated project IDs.
In the show_usage output, project IDs appear
in the column labeled “Subproject”.
The Project ID directive uses the dash, capital A option
followed by your project ID.
Returning to the sample script from the PBS Guide,
let us go over the four optional PBS directives included here.
The dash-capital-“N” option is used to specify a name for your job.
A job name may be easier to remember than the numeric ID
assigned by PBS.
In addition, the PBS environment variable,
PBS _ JOBNAME ($PBS_JOBNAME)
inherits this value and can be used instead of the job
ID to create job-specific output directories.
By default, the file names given these files
are in the form shown here,
where job name is specified with the dash N directive and job_i-d,
in this instance, “test job” is the ID of the job.
Your job’s messages are captured
in standard out and standard error files,
notated as shown.
Adding the dash “j” directive "oe", as shown here,
will combine the standard out and standard error values
into one file.
To receive email notification when your job starts and ends,
use the -M directive.
The two PBS directives from the sample script
to manage email are shown here.
Use the capital “M” directive to provide your email address.
The lowercase “m” tells PBS when to send the email.
A “b” option tells PBS to send an email when your job begins.
An “e” option tells PBS to send an email when your job ends.
You can combine these options together
as shown in this sample script.
Each of the optional PBS directives reviewed here are contained
in the PBS guides available for most HPC unclassified systems.
These guides include additional optional directives
not covered here, including:
Job Environment directives and Job Dependency directives.
Once the PBS directives have been supplied,
the execution block begins.
This section of your script contains the actual work to be done.
A well-written execution block will generally contain
the following stages:
Environment Setup • Compilation,
if you don’t already have a pre-compiled executable
• Launching and Clean Up.
The Environment Setup includes defining environment variables,
loading modules,
creating directories, copying files, initializing data, and other tasks.
This line creates a directory to contain the job’s output components
until they are archived in the cleanup section.
As the last step in this stage, you will use the cd command
to move your job to the work directory you want it to execute from.
Otherwise, your script will execute by default
in your home directory,
which is not recommended.
This application is launched using the mpi exec_mpt command for
HPE SGI MPT machines.
Other manufacturers’ systems use the m-p-i run
or a-p run commands.
Check the system guide on the centers.hpc.mil website for details.
A “Clean up” section archives your files using the transfer queue
to clean up your $WORKDIR after your job completes.
This practice is highly recommended.
It avoids data loss and ensures your allocation
is not charged for idle cores
while performing file transfer operations.
The final lines of this script
submit the cleanup routine to PBS for execution.
Once your batch script is complete,
you will need to submit it to PBS,
This is done with a submit command, qsub.
A list of job management commands can be found in the PBS guide
of each system for reference.
Thank you for taking the time to view this instructional video.
If you have any questions, or need assistance creating your scripts
or using PBS,
please contact the HPC Help Desk.
The phone numbers, email address,
and HPC Help Desk Web Portal
URL information are shown here and can also be found
in the accompanying job aid.