The Cray MPI environment is hidden from the user in general. Most platforms provide MPI-specific compiler wrappers such as mpif90 and mpicxx which adds in the proper include directories for mpi.h, etc. The Cray wrappers ftn, cc, and CC do this same thing but they apply to all builds regardless if the source needs MPI headers.

Some CUDA source (.cu) files need MPI library calls and the mpi.h header but the CUDA compiler (nvcc) does not have knowledge of the include path to mpi.h. The the proper path to <mpi dir>/include/mpi.h can be tricky to find, especially with many MPI versions.

1. Cray

The Cray MPI module (cray-mpich) provides the environmental variable $MPICH_DIR which points to the base of MPI distribution of whichever module version is loaded. Users can add -I$(MPICH_DIR)/include to the nvcc build line.

2. OpenMPI Platforms

This is not a Cray-specific problem. Finding the proper path to mpi.h is tricky on many platforms. Cray is a specifically tricky case since the MPI is explicitly hidden. OpenMPI has some helpful wrapper options that can be used to find the include and library search paths and the library names themselves. For example,

$ mpicc --showme:incdirs
/home/stonecp/ompi/include
$ mpicc --showme:libdirs
/home/stonecp/ompi/lib
$ mpicc --showme:libs
mpi dl m numa rt nsl util m dl

The first, mpicc --showme:incdirs is useful for this specific problem. Adding -I`mpicc --showme:incdirs` will pull in the proper mpi.h search path.

This all assumes $CPATH (or $C_INCLUDE_PATH) is not properly set! It would be useful for this to be set properly by the modules automatically so the users don't have to struggle with this when programming for a GPU.

2.1. HPE/SGI

The SGI MPT module provides SGI's version of MPI and comes with normal mpicc, mpicxx, and mpif90 wrappers. These work well for the default Intel compiler. But, if you want to pull in the Portland Group (PGI) compiler for OpenACC, you may need to do some extra work.

Users may experience difficulty with the mpi wrappers and PGI. The following is a workaround if this occurs.

Instead of using the mpif90 wrapper, use pgf90 directly and add -I$(MPI_ROOT)/include and -L$(MPI_ROOT)/lib -lmpi to your build rules. $MPI_ROOT should be set by the MPT module, but some have found this to be unreliable. If $MPI_ROOT is not set, you can get the appropriate path with module display mpt. Set $MPI_ROOT to the path shown by the module and then continue compiling.