How to compile and run WRF on Derecho¶
This documentation provides information on how to download and compile WRF and WPS on NSF NCAR Derecho. Users may also reference the NSF NCAR MMM Laboratory documentation for building WRF and WPS on Derecho. Also, an example of a PBS script is provided to help new users get an idea of how to submit batch jobs efficiently on the system, including setting resource requests, loading modules, and launching the WRF executable.
WRF Long Term Support
Users should be aware that the NSF NCAR MMM Laboratory have detailed the current support framework they continue to provide for WRF family models via the public WRF Support page and this community statement. Although the community is encouraged to consider transitioning to the Model for Prediction Across Scales (MPAS), WRF support will still be provided but in a reduced capacity.
Obtaining WRF and WPS code¶
The WRF and WPS codes can be downloaded from the WRF webpage. Please refer to the download instructions for more information.
Pre-compiled WRF on Derecho
If you do not plan to modify the WRF and WPS code, you can use (latest version recommended) the pre-compiled versions of the code available on Derecho at:
Copy the version you want to run to your working directory:
If you intend to modify the WRF code and contribute changes back to the official repository, it is recommended that you clone the latest development branch directly from the official WRF repository wrf-model GitHub repository. Please consult (WRF support) for more information about the process and requirements.
To clone the latest version (currently v4.7.1), use the following command:
This will create a local copy of the repository that you can edit and manage using Git.
After downloading the WRF model, the next step is to download and extract the WRF Preprocessing System (WPS).
Use the following command to extract the downloaded archive:
You will get a directory called WPS-4.6.0.
Next, the needed libraries are loaded to compile WRF and WPS.
On Derecho, this is done by loading pre-loaded modules. Please refer to Modules for more information.
The final command, module list, will display all currently loaded modules in your environment. A typical setup may look like:
Currently Loaded Modules:
1) ncarenv/24.12 (S) 3) intel/2024.2.1 5) libfabric/1.15.2.0 7) hdf5/1.12.3 9) cmake/3.26.6
2) craype/2.7.31 4) ncarcompilers/1.0.0 6) cray-mpich/8.1.29 8) netcdf/4.9.2
Where:
S: Module is Sticky, requires --force to unload or purge
Configuring the Build¶
Once the environment is set up, you can configure the code for compilation. It is strongly recommended to use the CMake-based configuration option provided with recent versions of WRF, as it simplifies and standardizes the build process.
Configuring WRF with CMake¶
Once your environment is set up and the WRF code is downloaded and extracted, navigate into the WRF source directory and initiate the configuration process using the CMake-based configuration script:
You will be prompted to select a compiler configuration. For this setup, select option 1 to compile WRF using the Intel oneAPI LLVM compilers (ifx/icx):
Using default build directory : _build
Using default install directory : /glade/derecho/scratch/biswas/CSG/WRF4.7.1/install
0 Linux gfortran / gcc / mpif90 / mpicc GNU (gfortran/gcc)
1 Linux ifx / icx / mpif90 / mpicc INTEL (ifx/icx) : oneAPI LLVM
!! - Compiler not found, some configurations will not work and will be hidden
Select configuration [0-1] Default [0] (note !!) : 1
Next, you will be prompted to choose options for various build components, such as WRF core type, nesting capability, case type, and MPI support. For most research applications, the default or commonly selected options are sufficient. The default values are shown. You can press Enter/Return to select the default options.
Select option for WRF_CORE from WRF_CORE_OPTIONS [0-4]
0 : ARW
1 : CONVERT
2 : DA
3 : DA_4D_VAR
4 : PLUS
Default [0] : 0
Select option for WRF_NESTING from WRF_NESTING_OPTIONS [0-3]
0 : NONE
1 : BASIC
2 : MOVES
3 : VORTEX
Default [1] : 1
Select option for WRF_CASE from WRF_CASE_OPTIONS [0-14]
0 : EM_REAL
1 : EM_FIRE
...
...
14 : NONE
Default [0] : 0
[DM] Use MPI? Default [Y] [Y/n] : Y
[SM] Use OpenMP? Default [N] [y/N] : N
Configure additional options? Default [N] [y/N] : N
You will notice several checks done by the script. If successful, you will notice a new directory: _build. Next step is to compile the code using:
Upon successful compilation, the executables wrf, real, ndown, and tc will be generated and located in the _build/main/ directory. Compilation progress and details are recorded in the compile.log file. If the compilation fails, refer to compile.log for diagnostic messages and error information to assist with troubleshooting.
Compiling WRF with Older Intel Compilers (ftn/icc) on Derecho¶
This section outlines how to compile WRF using the classic Intel compiler suite (ftn/icc) on Derecho’s Cray XC system. This method is useful if you prefer using Intel's traditional compilers rather than the newer oneAPI LLVM-based compilers.
Start by purging any previously loaded modules and loading the appropriate environment for Intel 2023 compilers:
The module list command will display all currently loaded modules, which should look similar to the following:
Currently Loaded Modules:
1) ncarenv/23.06 (S) 2) craype/2.7.20 3) intel/2023.0.0 4) ncarcompilers/1.0.0 5) cray-mpich/8.1.25 6) hdf5/1.12.2 7) netcdf/4.9.2
Navigate to the WRF source directory and start the configuration process:
The ./configure script will detect your system architecture and loaded environment settings, including available libraries like NetCDF.
You will then be prompted to choose a compilation option. For INTEL (ftn/icc), select:
Refer to WRF Users Guide for more info. Next, select the desired nesting option. For most applications, "basic nesting" (option 1) is sufficient:
48. (serial) 49. (smpar) 50. (dmpar) 51. (dm+sm) INTEL (ftn/icc): Cray XC
52. (serial) 53. (smpar) 54. (dmpar) 55. (dm+sm) PGI (pgf90/pgcc)
...
76. (serial) 77. (smpar) 78. (dmpar) 79. (dm+sm) INTEL (ifx/icx) : oneAPI LLVM
80. (serial) 81. (smpar) 82. (dmpar) 83. (dm+sm) FUJITSU (frtpx/fccpx): FX10/FX100 SPARC64 IXfx/Xlfx
Enter selection [1-83] : 50
------------------------------------------------------------------------
Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 1
Configuration successful!
Upon successful execution of the ./configure command, a configure.wrf file is generated. This file includes compilation settings and rules tailored to Derecho. Advanced users may edit this file to modify specific compile options if needed.
To compile WRF, use the ./compile command with the desired case option. In this tutorial, we compile the em_real case (Eulerian Mass grid for real-data simulations), which is the most commonly used option:
Note: It is strongly recommended to redirect the output of the compile process to a log file (e.g., compile.log) for easier debugging in case of errors. The redirection syntax may differ based on the shell used; the example above is for bash.
After the compilation is finished, if successful, you will see the following lines at the end of the compile.log file.
==========================================================================
build started: Wed 23 Apr 2025 10:04:19 AM MDT
build completed: Wed 23 Apr 2025 10:39:40 AM MDT
---> Executables successfully built <---
-rwxr-xr-x 1 biswas ncar 51935024 Apr 23 10:39 main/ndown.exe
-rwxr-xr-x 1 biswas ncar 47030584 Apr 23 10:39 main/real.exe
-rwxr-xr-x 1 biswas ncar 45936248 Apr 23 10:39 main/tc.exe
-rwxr-xr-x 1 biswas ncar 59119872 Apr 23 10:38 main/wrf.exe
==========================================================================
If the compilations fail, follow the WRF Users Guide for directions to debug.
Compiling the WRF Preprocessing System (WPS)¶
Once WRF is successfully compiled, the next step is to compile the WRF Preprocessing System (WPS), which is required to process input data for WRF.
Navigate to the WPS source directory. If you are following the standard folder structure, this can be done with:
WPS needs to know where your compiled WRF code resides. Set the WRF_DIR environment variable to point to your WRF installation directory:
WPS requires the JasPer library to handle GRIB2 input files (a common format for meteorological datasets). Set the following environment variables to point to the appropriate library and include paths:
export JASPERINC=/glade/u/home/wrfhelp/UNGRIB_LIBRARIES/include
export JASPERLIB=/glade/u/home/wrfhelp/UNGRIB_LIBRARIES/lib
Next, let's configure and compile WPS.
This will prompt the compiler options available. We opted 21 (Linux x86_64, Intel Classic compilers).
Will use NETCDF in dir: /glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/yzvj
Using WRF I/O library in WRF build identified by $WRF_DIR: /glade/derecho/scratch/biswas/CSG/WRF
Found Jasper environment variables for GRIB2 support...
$JASPERLIB = /glade/u/home/wrfhelp/UNGRIB_LIBRARIES/lib
$JASPERINC = /glade/u/home/wrfhelp/UNGRIB_LIBRARIES/include
------------------------------------------------------------------------
Please select from among the following supported platforms.
1. Linux x86_64, gfortran (serial)
...
20. Linux x86_64, Intel oneAPI compilers (dmpar_NO_GRIB2)
21. Linux x86_64, Intel Classic compilers (serial)
22. Linux x86_64, Intel Classic compilers (serial_NO_GRIB2)
23. Linux x86_64, Intel Classic compilers (dmpar)
24. Linux x86_64, Intel Classic compilers (dmpar_NO_GRIB2)
...
44. Cray XC CLE/Linux x86_64, Intel Classic compilers (dmpar_NO_GRIB2)
Enter selection [1-44] : 21
------------------------------------------------------------------------
Configuration successful. To build the WPS, type: compile
------------------------------------------------------------------------
The next step is to compile the WPS code.
If the build finishes successfully, the three executables (ungrib.exe, geogrid.exe, metgrid.exe) will be present in the WPS-4.6.0 directory.
Now, WPS and WRF codes are compiled, and the necessary executables are obtained. Refer to the WRF documentation for next steps.
Submitting jobs¶
WPS jobs can be run in an interactive mode. However, WRF jobs are more memory-intensive and should be submitted using a PBS batch script. To know more about PBS batch jobs, please refer to the PBS batch jobs scripts.
Load Balancing and Optimizing WRF on Derecho¶
WRF runtime is configured predominantly via a specified namelist.input file that details important variables and other configurations for the model simulation. The file README.namelist details all the variables that may be modified to your needs, with the latest version detailed at the WRF Github for README.namelist.
To note, in order to achieve optimal performance on Derecho, some primary considerations need to be made.
Determining node and MPI process counts with wrf-procs-finder¶
Specific to Derecho, a convenience tool wrf-procs-finder is provided through the ncarenv environment. This tool determines the minimum and maximum number of processors and nodes based on number grid points. The script requires the number of CPU cores per node on the system (default 128 on Derecho) and the domain size via the namelist variables e_we and e-sn or the namelist.input file itself. Using this information, the script will provide a range of MPI process count per node suggestions where given the configuration at which the simulation will likely run optimally.
See below for an example of how to use the tool or reach out to the Research Computing Help Desk for specific support with configuring WRF on Derecho.
# Print usage and help text for the script
wrf-procs-finder -h
# Example usage
wrf-procs-finder --e_we 1000 --e_sn 1000
# OR
wrf-procs-finder --namelist /path/to/namelist
------------------------------------------------------------
DOMAIN 1 of 1
------------------------------------------------------------
e_we = 1000
e_sn = 1000
Total grid points: 1000 x 1000 = 1,000,000
nprocs range:
Minimum....................... 100 procs => 1 node @ 100 procs/node
Maximum....................... 6528 procs => 51 nodes @ 128 procs/node
Decomposition for 6528 procs:
Layout : 68 x 96 tiles
Avg tile : ~ 14 x 10 grid points
NetCDF and Parallel NetCDF¶
Effective usage of the netCDF and parallel netcdf libraries are important for I/O performanceon the scratch system. Serial or parallel NetCDF options can be specified in the run-time WRF namelist, and the parallel netCDF I/O is often significantly faster than the serial I/O option. Therefore, we recommend to build WRF with the netcdf and parallel-netcdf module libraries. This is necessary only when compiling WRF from source. With these two modules, we set the following environmental variables before compiling WRF
module load hdf5
module load netcdf
module load parallel-netcdf
# Recommended environment variables for WRF use cases
export NETCDF_classic=1 # Set classic (CDF1) as default
export WRFIO_NCD_LARGE_FILE_SUPPORT=1 # Enable 64-bit offset format (CDF2) of netcdf files
export USE_NETCDF4_FEATURES=0. # NetCDF4 compression (serial) with the HDF5 module can be slow
Community Open Source Tools for WRF¶
The WRF community has developed some tools that may be of interest to users running WRF simulations detailed below.
- WRF Configuration Studio (GitHub) - A browser-based configuration environment for designing Weather Research and Forecasting Model domains
- WRFtailor - A namelist-based toolkit written in NCL and Bash allowing users to tailor and modify WRF input data (geo files, emissions data, etc.) based on geographic boundaries or spatial datasets.
Other recommended tools may be contributed to this guide.
Example WRF job scripts and workflow¶
Below is a sample job to run WRF using 1 node and utilizing 128 processors.
#!/bin/bash -l
#PBS -N wrf_run
#PBS -A XXXXXXXX
#PBS -q main
#PBS -l walltime=1:00:00
#PBS -m abe
#PBS -j oe
#PBS -l select=1:ncpus=128:mpiprocs=128
# Load modules to match compile-time environment
module --force purge
module load ncarenv/24.12
module reset
# cray-mpich mpiexec option
mpiexec -n 128 -ppn 128 ./wrf.exe
To submit a batch job, use the qsub command followed by the name of your PBS batch script file.
To know more about starting and managing jobs, please refer to the documentation.
Running Long Simulations with Restart Capability¶
For simulations that exceed the wall-clock time limits of Derecho or for operational workflows that require regular checkpoints, WRF supports restart capability, allowing you to break up long simulations into multiple shorter runs without losing progress.
Please refer to the Restart documentation to know how to set it up:
Debugging a Failed WRF Run¶
For detailed instructions and troubleshooting tips, refer to the WRF Forum Post on Debugging. This post includes additional strategies for locating and understanding runtime errors.
If your WRF run fails, it's important to identify the exact point of failure to efficiently resolve the issue.
Enable Debugging with Traceback:
By default, traceback support is disabled in the configure.wrf file. To enable it:
-
Open your configure.wrf file in a text editor.
-
Locate the line that defines FCDEBUG. It may be commented out.
-
Uncomment and modify it to the following:
Save the file and recompile WRF using the appropriate compile command (e.g., ./compile em_real).
Analyze the Log Files:
- After a failed run, check the rsl.out.* and rsl.error.* files in your run directory. These files contain the output from each processor and often indicate where and why the model stopped.
By enabling traceback and inspecting the relevant logs, you can often pinpoint the issue—whether it's a segmentation fault, array bounds error, or missing input—and proceed with targeted fixes.