Compiling code on NCAR systems¶

Compilers available on Casper¶

Several C/C++ and Fortran compilers are available on all NCAR HPC systems. The information on this page applies to all of those systems except where noted.

ifort / ifx

mpif90

-qopenmp

icc / icx

mpicc

icpc / icpx

mpicxx

nvfortran

mpif90

-mp

nvc

mpicc

nvc++

mpicxx

gfortran

mpif90

-fopenmp

gcc

mpicc

g++

mpicxx

Compiler Commands¶

All supported compilers are available via the module utility. After loading the compiler module you want to use, refer to the table above to identify and run the appropriate compilation wrapper command.

If your script already includes one of the following generic MPI commands, there is no need to change it:

• mpif90, mpif77

• mpicc

• mpiCC

Build any libraries that you need to support an application with the same compiler, compiler version, and compatible flags used to compile the other parts of the application, including the main executable(s). Also, before you run the applications, be sure you have loaded the same module/version environment in which you created the applications. This will help you avoid job failures that can result from missing MPI launchers and library routines.

Compiler man pages¶

To refer to the man page for a compiler, log in to the system where you intend to use it, load the module, then execute man for the compiler. For example:

module load nvhpc
man nvfortran

You can also use -help flags for a description of the command-line options for each compiler. Follow this example:

ifort -help

nvfortran -help [=option]

Changing compilers¶

To change from one compiler to another, use module swap. In this example, you are switching from Intel to NVIDIA:

module swap intel nvhpc

When you load a compiler module or change to a different compiler, the system makes other compatible modules available. This helps you establish a working environment and avoid conflicts.

If you need to link your program with a library, use module load to load the library as in this example:

module load netcdf

Then you can invoke the desired compilation command, including any library linking options such as -lnetcdf. Here's an example:

mpif90 -o foo.exe foo.f90 -lnetcdf

Compiling CPU code¶

Using the default Intel compiler collection¶

The Intel compiler suite is available via the intel module, which is loaded by default. It includes compilers for C, C++, and Fortran codes.

To see which versions are available, use the module avail command.

module avail intel

To load the default Intel compiler, use module load without specifying a version.

module load intel

To load a different version, specify the version number when loading the module.

Similarly, you can swap your current compiler module to Intel by using the module swap command.

module swap gcc intel

man ifort

Optimizing your code with Intel compilers¶

Intel compilers provide several different optimization and vectorization options. By default, they use the -O2 option, which includes some optimizations.

Using -O3 instead will provide more aggressive optimizations that may not improve the performance of some programs, while -O1 enables minimal optimization. A higher level of optimization might increase your compile time significantly.

You can also disable any optimization by using -O0.

Examples¶

To compile and link a single Fortran program and create an executable, follow this example:

ifort filename.f90 -o filename.exe

To enable multi-threaded parallelization (OpenMP), include the -qopenmp flag as shown here:

ifort -qopenmp filename.f90 -o filename.exe

Other compilers¶

These additional compilers are available on Casper.

• NVIDIA’s HPC SDK
• the GNU Compiler Collection (GCC)

Compiling GPU code¶

On Casper, GPU applications should be built with either the NVIDIA HPC SDK compilers and libraries, or with two-stage linking. In the following examples, we demonstrate the use of NVIDIA’s tools. To compile CUDA code to run on the Casper data analysis and visualization nodes, use the appropriate NVIDIA compiler command:

• nvc – NVIDIA C compiler

• nvcc – NVIDIA CUDA compiler (Using nvcc requires a C compiler to be present in the background; nvc, icc, or gcc, for example.)

• nvfortran – CUDA Fortran

Additional compilation flags for GPU code will depend in large part on which GPU-programming paradigm is being used (e.g., OpenACC, OpenMP, CUDA) and which compiler collection you have loaded. The following examples show basic usage, but note that many customizations and optimizations are possible. You are encouraged to read the relevant man page for the compiler you choose.

OpenACC¶

To compile with OpenACC directives, simply add the -acc flag to your invocation of nvc, nvc++, or nvfortan. A Fortran example:

nvfortran -o acc_bin -acc acc_code.f90

You can gather more insight into GPU acceleration decisions made by the compiler by adding -Minfo=accel to your invocation. Using compiler options, you can also specify which GPU architecture to target. This example will request compilation for both V100 and A100 GPUs:

nvfortran -o acc_bin -acc -gpu=cc70,cc80 acc_code.f90

Specifying multiple acceleration targets will increase the size of the binary and the time it takes to compile the code.

OpenMP¶

Using OpenMP to offload code to the GPU is similar to using OpenACC. To compile a code with OpenMP offloading, use the -mp=gpu flag. The aforementioned diagnostic and target flags also apply to OpenMP offloading.

nvfortran -o omp_gpu -mp=gpu omp.f90

CUDA¶

The process for compiling CUDA code depends on whether you are using C++ or Fortran. For C++, the process often involves multiple stages in which you first use nvcc, the NVIDIA CUDA compiler, and then your C++ compiler of choice.

nvcc -c -arch=sm_80 cuda_code.cu
g++ -o cuda_bin -lcuda -lcudart main.cpp cuda_code.o

The compiler handles CUDA code directly, so the compiler you use must support CUDA. This means you should use nvfortran. If your source code file ends with the .cuf extension, nvfortran will enable CUDA automatically. Otherwise, you can specify the -Mcuda flag to the compiler.

nvfortran -Mcuda -o cf_bin cf_code.f90

The sample below demonstrates how to compile CUDA C code on casper.

/* hello_world.cu
 * ---------------------------------------------------
 * A Hello World example in CUDA
 * ---------------------------------------------------
 * This is a short program which uses multiple CUDA
 * threads to calculate a "Hello World" message which
 * is then printed to the screen.  It's intended to
 * demonstrate the execution of a CUDA kernel.
 * ---------------------------------------------------
 */
#define SIZE 12
#include <stdio.h>
#include <stdlib.h>
#include <cuda_runtime.h>

/* CUDA kernel used to calculate hello world message */
__global__ void hello_world(char *a, int N);

int main(int argc, char **argv)
{
   /* data that will live on host */
   char *data;

   /* data that will live in device memory */
   char *d_data;

   /* allocate and initialize data array */
   data = (char*) malloc(SIZE*sizeof(char));
   data[0]  =  72; data[1]  = 100; data[2]  = 106;
   data[3]  = 105; data[4]  = 107; data[5]  =  27;
   data[6]  =  81; data[7]  = 104; data[8]  = 106;
   data[9]  =  99; data[10] =  90; data[11] =  22;

   /* print data before kernel call */
   printf("Contents of data before kernel call: %s\n", data);

   /* allocate memory on device */
   cudaMalloc(&d_data, SIZE*sizeof(char));

   /* copy memory to device array */
   cudaMemcpy(d_data, data, SIZE, cudaMemcpyHostToDevice);

   /* call kernel */
   hello_world<<<4,3>>>(d_data, SIZE);

   /* copy data back to host */
   cudaMemcpy(data, d_data, SIZE, cudaMemcpyDeviceToHost);

   /* print contents of array */
   printf("Contents of data after kernel call:  %s\n",data);

   /* clean up memory on host and device */
   cudaFree(d_data);
   free(data);
   return(0);
}

/* hello_world
 * Each thread increments an element of the input
 * array by its global thread id
 */
__global__ void hello_world(char *a, int N)
{
   int i = blockDim.x * blockIdx.x + threadIdx.x;
   if(i < N) a[i] = a[i] + i;
}

execcasper -l gpu_type=gp100 --ngpus=1

module reset
module load nvhpc cuda
nvcc -o hello hello_world.cu

module reset
module load gnu cuda
nvcc -c hello_world.cu
gcc -o hello hello_world.o

./hello
Contents of data before kernel call: HdjikhjcZ
Contents of data after kernel call:  Hello World!

Native Compiler Commands¶

We recommend using the module wrapper commands described above. However, if you prefer to invoke the compilers directly without the ncarcompilers wrappers, see this note:

module unload ncarcompilers

ifort -o a.out $NCAR_INC_<PACKAGE> program_name.f $NCAR_LDFLAGS_<PACKAGE> -l<package_library>

nvfortran -o a.out $NCAR_INC_<PACKAGE> program_name.f $NCAR_LDFLAGS_<PACKAGE> -l<package_library>

gfortran -o a.out $NCAR_INC_<PACKAGE> program_name.f $NCAR_LDFLAGS_<PACKAGE> -l<package_library>

Multiple Compiler Versions and User Applications¶

In addition to multiple compilers, CISL keeps available multiple versions of libraries to accommodate a wide range of users' needs. Rather than rely on the environment variable LD_LIBRARY_PATH to find the correct libraries dynamically, we encode library paths within the binaries when you build Executable and Linkable Format (ELF) executables. To do this, we use RPATH rather than LD_LIBRARY_PATH to set the necessary paths to shared libraries.

This enables your executable to work regardless of updates to new default versions of the various libraries; it doesn't have to search dynamically at run time to load them. It also means you don't need to worry about setting the variable or loading another module, greatly reducing the likelihood of runtime errors.

Common Compiler Options and Diagnostic Flags¶

Portability and correctness both are important goals when developing code. Non-standard code may not be portable, and its execution may be unpredictable.

Using diagnostic options when you compile your code can help you find potential problems. Since the compiler is going to analyze your code anyway, it pays to take advantage of the diagnostic options to learn as much as you can from the analysis. Please note that some compilers disable the default optimization when you switch on certain debugging flags.

Because of differences in compilers, it also is good practice to compile your code with each compiler that is available on the system, note any diagnostic messages you get, and revise your code accordingly.

The following options can be helpful as you compile code to run in the HPC environment that CISL manages.