Build a CUDA source with MPI dependencies when mpi.h can't be found
The Cray MPI environment is hidden from the user in general.
Most platforms provide MPI-specific compiler wrappers such as mpif90 and
mpicxx which adds in the proper include directories for
etc. The Cray wrappers ftn, cc, and CC do this same thing but they apply
to all builds regardless if the source needs MPI headers.
Some CUDA source (.cu) files need MPI library calls and the
header but the CUDA compiler (nvcc) does not have knowledge of the
include path to
mpi.h. The the proper path to
<mpi dir>/include/mpi.h can be tricky to find, especially
with many MPI versions.
The Cray MPI module (cray-mpich) provides the environmental variable
$MPICH_DIR which points to the base of MPI distribution of
whichever module version is loaded. Users can add
to the nvcc build line.
2. OpenMPI Platforms
This is not a Cray-specific problem. Finding the proper path
mpi.h is tricky on many platforms. Cray is a specifically
tricky case since the MPI is explicitly hidden. OpenMPI has some helpful
wrapper options that can be used to find the include and library search paths
and the library names themselves. For example,
$ mpicc --showme:incdirs /home/stonecp/ompi/include $ mpicc --showme:libdirs /home/stonecp/ompi/lib $ mpicc --showme:libs mpi dl m numa rt nsl util m dl
mpicc --showme:incdirs is useful for
this specific problem. Adding
-I`mpicc --showme:incdirs` will
pull in the proper
mpi.h search path.
This all assumes
$C_INCLUDE_PATH) is not
properly set! It would be useful for this to be set properly by the modules
automatically so the users don't have to struggle with this when programming
for a GPU.
2.1. Thunder (AFRL)
The SGI MPT module provides SGI's version of MPI and comes with normal mpicc, mpicxx, and mpif90 wrappers. These work well for the default Intel compiler. But, if you want to pull in the Portland Group (PGI) compiler for OpenACC, you may need to do some extra work.
Users may experience difficulty with the mpi wrappers and PGI. The following is a workaround if this occurs. Note that this is subject to change as Thunder becomes more stable.
Instead of using the mpif90 wrapper, use pgf90 directly and
to your build rules.
$MPI_ROOT should be set by the MPT module,
but some have found this to be unreliable. If
$MPI_ROOT is not
set, you can get the appropriate path with
module display mpt.
$MPI_ROOT to the path shown by the module and then continue
compiling. This workaround is not specific to just Thunder but should work
on any SGI platform with MPT if needed.