Welcome to the HPC Help Desk Video Series.
This presentation will provide
an overview of the Load Sharing Facility,
or LSF as utilized on specific IBM systems in the High Performance
Computing Modernization Program.
Sources for this video
include the SCOUT LSF Guide found on the centers.hpc.mil
website and IBM Spectrum LSF documentation in the
IBM Knowledge Center.
In the link to supplemental materials accompanying this video,
you will find a Job Aid that includes all the URLs,
phone numbers and email contacts referenced in this video.
When you see this graphic and hear this sound,
it means the content is contained in the accompanying Job Aid.
This presentation will review the necessary elements
of an LSF script, job management commands,
optional LSF directives, environment variables
and an example script with analysis.
Currently, SCOUT at the Army Research Laboratory or ARL,
is the only allocated HPC system in the program running LSF.
The IBM Spectrum LSF, an acronym for Load Sharing Facility,
distributes work across SCOUT resources to create a shared,
scalable and fault tolerant infrastructure.
LSF balances load, allocates resources, and provides
access to those resources to the user.
On large scale computers, many users
must share the available resources.
All users’ batch jobs have to get in line and wait their turn.
The jobs on SCOUT are managed and controlled by LSF.
LSF sorts the batch jobs by a number of criteria.
Your job will run when these factors are met:
The system can accommodate your resource requests,
Your job will not overload resources,
All higher priority jobs have run,
Overall throughput can be maximized.
LSF is loaded automatically
into your path at login, allowing you to use the LSF commands.
A batch script is a small text file
that can be created with a text editor
such as VI or Notepad.
You may create your own from scratch
or start with one of the sample batch scripts available
in the sample code repository on the system.
The basic components of a batch
script must appear in the following order:
Specify your shell, required LSF directives,
and the execution block.
Unless you specify otherwise, LSF will use your default login
shell to run your script. To tell LSF which shell to use,
start your script with the following line:
where shell is either bash shell, kshell, cshell or tcshell.
The next block of your script
will use LSF directives to tell LSF
what resources your job needs.
These directives are a special form of comment,
beginning with #BSUB.
The hashtag character tells the shell to ignore the line,
but LSF reads these directives
and uses them to set various values.
Important: all LSF directives must come before the first line
of executable code in your script, otherwise they will be ignored.
Every script must include directives for the following:
total number of cores requested,
how nodes should be allocated,
the maximum amount of time your job should run,
which queue you want your job to run in, and your project ID.
LSF also provides optional directives
which will be discussed later in this presentation.
Before LSF can schedule your job,
it needs to know how many cores are required.
A glance at the SCOUT User Guide on the
centers.hpc.mil website reveals that SCOUT
has four nodes with forty cores per node,
for a total of 160 cores maximum.
The cores you request are specified with the -n option.
Some default behaviors, such as node sharing, can affect the ability
of your scripts to run in certain situations
and may impose restrictions that cause delays in the queue.
-x indicates to LSF
that the job has exclusive access to the requested nodes.
If you want to reserve nodes with graphics
processing units, or GPUs,
which are often used for machine learning acceleration,
then use the -gpu flag
with the “num=n” option to request n GPUs
and the “mode = shared” option to share the GPUs.
Replace “shared” with “exclusive_process”
if you don't want to share GPUs.
By default, your job will run with one process per core,
but you might want more or fewer processes
depending on the programming model you are using.
Or, if you want to make more memory available by using less cores
per node, this option indicates the number of processes
on each node that should be allocated.
The -R option with a span string indicates
the number of processors on each node
that should be allocated to the job.
Example one, the default: one process on one core
gets all the memory per node.
Example two: Two processes
running on two cores; each get half.
Example three: All 40 processes
split the memory on each node.
LSF needs to know how long your job will run.
For this, you will have to make an estimate.
There are three things to keep in mind:
One, your estimate is a limit.
If your job hasn't completed within your estimate,
it will be terminated.
Two: Your estimate will affect how long your job waits in the queue.
In general, shorter jobs will run before longer jobs.
Three.
Each queue has a maximum time limit.
You cannot request more time than the queue allows.
Consult the SCOUT User Guide on the
centers.hpc.mil website for maximum wall clock time.
To specify how long your job will run,
Include the following directive using hours and minutes:
#BSUB -W HH:MM
Next, LSF needs to know which queue to run your job in.
Your options here are determined by your project.
Most users only have access to the debug, standard, interactive,
and background queues.
The standard and debug queue
should be used for normal day to day and debugging jobs.
The background queue may be useful
because although it has the lowest priority, jobs
in this queue are not charged against your project allocation.
Users may choose to run
in the background queue for several reasons:
you don't care how long it takes for your job to begin running.
you are trying to conserve your allocation,
or, you have used up your allocation.
To specify the queue
you want your job to run in, include the following directive:
#BSUB -q queue name
Finally, LSF needs to know which Project ID
to charge for your job.
You can use the show_usage command
to find the projects that are available to you
and their associated project IDs.
In the show_usage output,
Project IDs appear in the column labeled “subproject.”
Note: users with access to multiple projects
should remember that the project they specify may limit
their choice of queues.
To specify the Project ID for your job,
include the following directive:
#BSUB -P Project_ID
Once the LSF directives have been specified,
the execution block defines the actual work to be done.
A well-written execution block
will generally contain the following stages:
Environment Setup.
This may include setting environment variables,
loading modules, creating directories, copying files,
initializing data and such.
As the last step in this stage,
you will generally change directories to the one
that you want your script to execute in.
Otherwise your script will execute by default
in your home directory.
Compilation. You may need to compile your application
if you don't already have a pre-compiled executable available.
Launching.Your application is launched
using the MPI exec command for IBM Spectrum MPI codes.
And cleanup.This usually includes
archiving your results and removing
temporary files and directories.
Once your batch script is complete,
you will need to submit it to LSF for execution
using the bsub command.
For example,
if you have saved your script into a text file named run.lsf,
you would type this: bsub > run.lsf
This batch script contains
all of the required directives and common
script components discussed previously.
In addition to bsub,
there are other common job management commands for LSF.
We've covered the required directives,
but LSF has many optional directives helpful
in creating an efficient, successful script.
The -J directive allows you to designate a name for your job.
In addition to being easier
to remember than a long alphanumeric Job ID,
the LSF environment variable $LSB_JOBNAME inherits
this value and can be used instead of the Job ID
to create job specific output directories.
To use this directive,
add a line in the following form to your batch script:
#BSUB -J jobname
Some tasks
you wish to perform may require special terminal handling.
These must be run using a pseudo terminal
so they can be run over the network.
The -I directive allows you to request
an interactive batch shell
from which you can perform normal Linux commands,
including launching parallel jobs.
To use -I, append it to your BSUB request along with the s directive
which requests the pseudo terminal shell.
The interactive job will not run properly if this is not used.
This directive specifies
whether the allocated GPUs can be used by other jobs.
Use a shell designation to assign your preferred shell.
If no designation is provided, your default shell is used.
The -env directive tells
LSF to export environment variables from your login
environment to your batch environment.
You can use “-env all” to export all environment variables
from your login environment or “-env none” for no export.
The -env directive can also tell LSF to export
specific environment variables
from your login environment into your batch environment.
For example, this line would export the display variable
#BSUB -env DISPLAY
Multiple comma separated variables can be included.
It is also possible to set values for variables
exported in this way, as follows:
#BSUB -env my_variable=my_value,variable2=value and so on.
The -m directive allows you to select the type of node
on which the job will run.
To use this directive, add
a line in the following form to your batch script:
#BSUB -m training
You can also choose inference or visualization node types.
By default, standard out and standard error
error messages are written to files named
x.ojob_id and x.ejob_id respectively,
where x is either the name of the script
or the name specified with the -J
directive and job_id is the ID of the job.
If you want to change this behavior,
the -o and -e directives
allow you to redirect these messages to different-named files.
The -eo directive allows you to combine them into the same file.
Many users want to be notified when their jobs begin and end.
The -B directive makes this possible.
If you use this directive,
you must supply a valid email address or addresses
separated by a comma.
If the -u directive is used,
SCOUT usernames must be provided.
Job dependency directives
allow you to specify dependencies
that your job may have on other jobs.
This allows users to control the order
jobs are run in.
These directives will generally take the following form:
The “dependency_expression” is a comma-delimited list of
one or more of the types from the table of job
dependency directives, and “jobid”
is a colon-delimited list of one or more Job IDs
that your job is dependent upon.
For example, run a job after completion,
success or failure of Job ID 1234.
#BSUB -w “exit(1234)”
Or, run a job after successful completion of Job ID 1234:
#BSUB -w “ended(1234)”
While there are many LSF environment variables,
you only need to know
a few important ones to get started using LSF.
The table below lists the most important LSF Environment Variables
and how you might generally use them.
Refer to the SCOUT LSF guide available on the centers.hpc.mil
website.
All scripts should contain a cleanup section
which automatically archives
your data using the transfer queue and cleans up
$WORKDIR after your job completes.
Using this method helps to avoid data loss
and ensures that your allocation is not charged for idle cores
while performing file transfer operations.
You can find example scripts on the Scout LSF
Guide on the ARL DSRC website or by running the command:
ls $SAMPLES_HOME
Thank you for taking the time to view this instructional video.
If you have any additional questions,
please contact the HPC Help Desk.
The phone numbers, email address and HPC Help Desk
Web Portal URL information is shown here
and can also be found in the accompanying Job Aid.