Welcome to the HPC Help Desk Video Series.
This video will introduce the viewer
to the Lustre High Performance Parallel File System
employed on most of the HPCMP systems.
Sources for this video
include the Lustre Guide found in the user
guide of each DSRC website,
The Lustre Manual from Lustre.org
and the Lustre 101 course developed by the Oak Ridge
Leadership Computing Facility.
When you see this graphic and hear this sound,
It means the information is contained in the accompanying job aid,
which includes all the URLs, phone numbers
and email contacts referenced in this video.
This presentation will consist
of a background of Lustre file handling,
an introduction to Lustre fundamentals
and an explanation of common terms,
a guide to partitioning data into smaller sections, known as striping,
how to modify settings with Lustre commands
and general best practices with Lustre at HPC centers.
Lustre is an open source file system
developed in 1999 and released
to general production in the early 2000's.
It was developed under the direction of the U.S.
Department of Energy
and has been widely deployed in large supercomputing centers.
A common distributed file system uses standard
network file access where the file data and metadata
are managed by a single storage controller.
In bandwidth-intensive workloads,
this single point of access becomes a bottleneck for performance
Lustre is a parallel file system
which divides files into data blocks
and spreads them
across multiple storage servers,
then writes to and reads from these sections in parallel.
Critical information about the files
is stored on a separate metadata server
for more efficient file lookup.
There are a number of acronyms
you should know to understand how Lustre works.
A metadata server, or MDS, tracks metadata such as ownership
and permissions of a file or directory.
Object storage servers or OSSs, provide file input and output
services for object storage targets, or OSTs,
which actually store the data.
An OST is typically a single disk array.
This diagram of a Lustre file system
has one MDS, three OSSs and two OSTs per OSS,
for a total of six OST's.
Lustre achieves its performance by
automatically partitioning data into sections known as stripes,
and writing the stripes in round robin fashion across multiple OSTs.
This process, called striping,
can significantly improve file I/O speeds
by eliminating single disk bottlenecks.
The term "stripe count"
refers to the number of stripes into which a file is divided,
the number of OSTs that are used to store the file.
Each stripe of the file will reside on a different OST.
"Stripe size" refers to the size of the stripe
written as a single block to an OST.
Advantages of striping include:
increased I/O bandwidth due
to multiple areas of the files being read
or written in parallel,
and maintaining usage balance across the pool of OSTs.
Let's look at an example.
200 megabytes are to be written to a file
that was created with a stripe count of ten
and a stripe size of one megabyte.
When the file is initially written,
ten one megabyte blocks will be simultaneously
written to ten different OSTs.
Once those ten blocks have been filled,
Lustre writes another 10 one megabyte blocks to those ten OSTs.
This process is repeated for a total of 20 times
until the entire file has been written.
Upon completion, the file will exist
as 20 one-megabyte blocks of data on each of ten separate OSTs.
Users have the option of configuring the size
and number of stripes used for any file they own.
Determining the best setting sometimes requires experimentation,
but there are general rules of thumb.
Charts with work file specifications and recommended
file-striping guidelines for each
HPC system are available at the host DSRC's website.
From the centers homepage, choose the "For Users" tab,
then documentation.
Choose the center that hosts the computer you want to use,
then scroll down to the Supplemental Guide section.
A link to the Lustre User Guide will be available in the list.
Please note that while similar, the file specs,
recommended stripe count and size will vary from
system to system, so ensure you are using the chart
specific to the computer you wish to use.
These example tables
offer striping guidelines based on file size for an HPC system.
Increasing the stripe count or stripe size
should be done proportionately,
with the number of nodes used for I/O.
As a rule,
an application should try to use as many OSTs as possible.
So, if writing a large single file in parallel,
set the stripe count to the maximum allowed value.
Conversely,
when writing many small files in parallel,
set the stripe count to one. An intermediate number
of simultaneous output files may perform
best with a stripe count greater than one.
Experimentation with the stripe count
is often helpful for best performance.
Stripe parameters can be set for individual files
and set or changed for directories.
Directories can be given a stripe setting
so that all new files created in that directory
and under any subdirectory share that setting.
Lustre provides utilities and application libraries to allow users
to control the striping of individual files at creation time.
However, changing the stripe parameters
on an existing file has no effect.
You must first create an empty file
with the desired striping characteristics
and then write your data to it.
Likewise, changing the stripe parameters on a directory
does not change the striping of files
already existing in that directory.
Only new files created in the modified
directory will inherit the changed striping.
Let's look at Lustre's striping commands
and how they are used in a script.
The LFS getstripe command
reports the stripe characteristics of a file or directory.
Here's an example.
The output shows that files created in the directory
MyDir will be stored using one stripe of 1,048,576 bytes
or one megabyte per block, unless explicitly striped
otherwise before writing,
the stripe underscore offset,
also known as stripe
index of -1,
means that each file will have an OST placement
determined automatically by Lustre.
to set the striping for a file or directory,
use the LFS setstripe command.
Here's the syntax:
The stripe size sets the number of bytes written to one OST
before cycling to the next.
The stripe count sets the number of OST's to be written to.
The stripe index sets the position of the first stripe
among the OSTs, called an index.
Let's take a closer look.
The setstripe command allows you to set the stripe size.
However, the default stripe size
determined by the system-specific
charts on the host DSRC website,
is recommended for most applications.
The setstripe command also has an option
for setting the position of the first stripe
among the OSTs called the index.
Users should not specify an index--instead,
a default setting allows the Lustre file system
to choose an index in order to help maintain
overall file system performance.
Not specifying a parameter will set it to the default.
Let's write a file using the setstripe command.
The following creates an empty file
named LargeFile with a stripe count of eight.
Next, set the stripe count to 16 for a new directory named LargeDir.
Note that any subdirectories or files created under LargeDir
will inherit its new stripe characteristics.
Finally, set the stripe count to 32
and the stripe size to 2 megabytes for a new file named HugeFile.
The LFS setstripe command
can be placed in a PBS batch script
or executed interactively before job submission.
A good practice is to create new special output
directories with appropriate striping
within the PBS batch script for large output files.
This directs files created during program execution
to inherit the characteristics of the directory
into which they are written.
A file can be generated in several ways,
such as by writing data to it from an executing program,
by copying an existing file, by extracting a tar file,
or by using the Linux command "cat."
As far as striping is concerned,
the key criteria is not how the file is generated,
but whether or not a file of that name already exists--
it does not matter if the file is empty or not.
If you generate a file of the same name
as a file that already exists,
it will retain the striping
characteristics of the existing file
because you are only changing the contents of the file.
If the file does not already exist,
then it is new
and will inherit the striping characteristics
of the directory where it is created.
Distributed file systems are ideal for HPC environments,
but can be severely impacted when managing large numbers
of small files and metadata information.
Operations that can be performed quickly on a workstation
local disk may be extremely slow on a Lustre system,
affecting both the user's operations
and possibly other users accessing the system.
Avoid using commands that:
access attributes of files and directories,
access metadata massively, and use wild cards.
Small I/O requests or writing many files should be avoided.
It is better to gather small requests
into a buffer and write the buffer when it is full. On a Cray system.
The iobuf facility is recommended for this kind of I/O aggregation.
For more information
load the iobuf module and view
the manual pages with a "man iobuf" command. On all machines,
the Intel FORTRAN compiler can enable buffering
with the "-assume buffered_io" flag. Application level
I/O libraries that may offer improved performance
include MPI/IO, ADIOS, NetCDF and HDF5.
Some of these libraries
may set striping for you, while others
may require manual setting for output files.
Lustre's ability to inherit file
or directory characteristics can be confusing.
Things to keep in mind:
Overwriting a file without first deleting it
will cause it to inherit the original file striping.
If a file is moved across the same Lustre file system
then it is not a new file and its striping
characteristics are not changed.
A file copied from one directory to another,
such as with cp, cat, scp or tar, inherits
the striping of the new directory
or the existing file if the file name already exists.
Use this to your advantage
by striping a directory for certain types of files.
Consider the stripe characteristics of a target directory
before copying.
For example, a large file into a directory set up for small files.
Moving a file off the Lustre file system
or using archive commands like tar will not preserve
the striping of the file.
When restoring the files from another file system or a tar archive,
the files inherit the striping of the parent directory
or the existing file.
If the file name already exists,
changing the striping parameters for an existing
directory does not change the striping
for files already in that directory.
Only new files written into that directory
will inherit the revised striping.
Thank you for taking the time to view this instructional video.
If you have more questions or have difficulty
using the Lustre file system, please contact the HPC Help Desk.
The phone numbers, email address and HPC Help Desk web portal
URL information are shown here
and can also be found in the accompanying Job Aid.