Portable Batch System (PBS) Job Scripts¶

Job scripts form the basis of batch jobs. A job script is simply a text file with instructions of the work to execute. Job scripts are usually written in bash or tcsh and thus mimic commands a user would execute interactively through a shell; but instead are executed on specific resources allocated by the scheduler when available. Scripts can also be written in other languages - commonly Python.

Anatomy of a Job Script¶

Sample basic PBS scripts are listed below:

#!/bin/bash
#PBS -N hello_pbs
#PBS -A <project_code>
#PBS -j oe
#PBS -k eod
#PBS -q main
#PBS -l walltime=00:05:00
#PBS -l select=2:ncpus=128:mpiprocs=128

### Set temp to scratch
export TMPDIR=${SCRATCH}/${USER}/temp && mkdir -p $TMPDIR

### specify desired module environment
module purge
module load ncarenv/23.09 gcc/12.2.0 cray-mpich/8.1.25
module list

### Compile & Run MPI Program
mpicxx -o hello_world_derecho /glade/u/home/benkirk/hello_world_mpi.C -fopenmp
mpiexec -n 256 -ppn 128 ./hello_world_derecho

#!/bin/bash

#!/bin/tcsh
#PBS -N hello_pbs
#PBS -A <project_code>
#PBS -j oe
#PBS -k eod
#PBS -q main
#PBS -l walltime=00:05:00
#PBS -l select=2:ncpus=128:mpiprocs=128

source /etc/csh.cshrc

### Set temp to scratch
setenv TMPDIR ${SCRATCH}/${USER}/temp && mkdir -p ${TMPDIR}

### specify desired module environment
module purge
module load ncarenv/23.09 gcc/12.2.0 cray-mpich/8.1.25
module list

### Compile & Run MPI Program
mpicxx -o hello_world_derecho /glade/u/home/benkirk/hello_world_mpi.C -fopenmp
mpiexec -n 256 -ppn 128 ./hello_world_derecho

#!/bin/tcsh

#!/glade/u/apps/opt/conda/envs/npl/bin/python
#PBS -N hello_pbs
#PBS -A <project_code>
#PBS -j oe
#PBS -k eod
#PBS -q main
#PBS -l walltime=00:05:00
#PBS -l select=1:ncpus=128:mpiprocs=128

import sys
print("Hello, world!!\n\n")

print("Python version:")
print(sys.version)
print("Version info:")
print(sys.version_info)

#!/glade/u/apps/opt/conda/envs/npl/bin/python

Focusing on the bash example for discussion, the remainder of the script contains two main sections:

1. The lines beginning with #PBS are directives that will be interpreted by PBS when this script is submitted with qsub. Each of these lines contains an instruction that will be used by qsub to control job resources, execution, etc...

2. The remaining script contents are simply bash commands that will be run inside the batch environment on the selected resources and define the work to be done in this job.

PBS directives¶

The example above contains several directives which are interpreted by the qsub submission program:

• -N hello_pbs provides a job name. This name will be displayed by the scheduler for diagnostic and file output. If omitted, and a script is used to submit the job, the job's name is the name of the script.
• -A <project_code> indicates which NCAR Project Accounting code resource allocation will be applicable to this job. (You will want to replace <project_code> with your project's specific code.)
• -j oe requests we combine any standard text output (o) and error (e) into one output file. (By default, PBS will write program output and error to different log files. This behavior is contrary to what many users expect from terminal interaction, where output and error are generally interspersed. This optional flag changes that behavior.)
• -q main specifies the desired PBS queue for this job.
• -l walltime=00:05:00 requests 5 minutes as the maximum job execution (walltime) time. Specified in HH:MM:SS format.
• -l select=2:ncpus=128:mpiprocs=128 is a computational resource chunk request, detailing the quantity and configuration of compute nodes required for this job. This example requests a selection of 2 nodes, where each node must have 128 CPU cores, each of which we will use as an MPI rank in our application.

Script contents¶

The remaining script contains shell commands that define the job execution workflow. The commands here are arbitrary, however we strongly recommend the general structure presented above. This includes:

1. Explicitly setting the TMPDIR variable.

   As described here, many programs write temporary data to TMPDIR, which is usually small and shared among st users. Specifying your own directory for temporary files can help you avoid the risk of your own programs and other users' programs failing when no more space is available.

2. Loading and reporting the specific module environment required for this job.

   While strictly not necessary (in general, the system default modules will be loaded anyway), we recommend this as best practice as it facilitates debugging and reproducing later. (While the system default modules will change over time, manually specifying module versions allows you to recreate the same execution environment in the future.)

3. (Optional) Defining any environment variables specific to the chosen module environment.

   Occasionally users will want to define particular run time environment variables e.g. for a specific MPI or library chosen via the module load commands.

4. Remaining job-specific steps.

   In the example above, we first compile and then execute hello_world_mpi.C, a simple MPI program.

Common #PBS directives¶

Resource requests¶

Resources (compute node configuration, job duration) are requested through a combination of resource selection flags, each preceded with -l.

For example:

#PBS -l walltime=00:05:00
#PBS -l select=1:ncpus=64:mpiprocs=4:ngpus=4:mem=400GB
#PBS -l gpu_type=a100
#PBS -l job_priority=economy

select statements¶

Resources are specified through a select statement. The general form of a homogeneous selection statement is

select=<# NODES>:ncpus=<# CPU Cores/node>:mpiprocs=<# MPI Ranks/node>:ompthreads=<# OpenMP Threads/rank>:mem=<RAM/node>:ngpus=<# GPUs/node>

• <# NODES> is the total number of compute nodes requested, followed by a colon-separated list of

• <# CPU Cores/node> is the total number of CPUs requested on each node, which can be a mix of MPI Ranks and/or OpenMP threads,

• <# MPI Ranks/node is the number of MPI Ranks on each node,

• <# OpenMP Threads/node> is the number of OpenMP ranks per MPI Rank on each node. (Optional, defaults to 1),

• <RAM/node> is how much main memory (RAM) the job will be able to access on each node. (Optional, default is system dependent), and

• <# GPUs/node> is the number of GPUs per node. (Optional, defaults to 0).

Taken together, this specifies a resource chunk. Homogeneous resource chunks are the most common case, however, heterogeneous selection statements can be constructed by multiple chunks separated by a + (examples below).

Examples¶

• 4 128-core nodes, each running 128 MPI ranks (4 x 128 = 512 MPI ranks total).

  select=4:ncpus=128:mpiprocs=128
  

• 4 128-core nodes, each running 32 MPI ranks with 4 OpenMP threads per ranl (4 x 32 = 128 MPI ranks total, each with 4 threads = 512 total CPU cores).

  select=4:ncpus=128:mpiprocs=32:ompthreads=4
  

• 2 64-core nodes, each running 4 MPI ranks, 4 GPUS, and 384 GB memory (8 GPUs total, with 8 MPI ranks).

  select=2:ncpus=64:mpiprocs=4:ngpus=4:mem=384GB
  

• 4 36-core nodes, each running 4 MPI ranks, 4 GPUS configured with NVIDIA's Multi-Process Service (MPS), and 768 GB memory (16 GPUs total, with 16 MPI ranks).

  select=4:ncpus=36:mpiprocs=4:ngpus=4:mem=768GB:mps=1
  

  MPS is simply enabled via mps=1, and is disabled by default (mps=0)

• A heterogeneous selection, 96 128-core nodes each with 128 MPI ranks, and 32 128-core nodes each with 16 MPI ranks and 8 OpenMP threads

  select=96:ncpus=128:mpiprocs=128+32:ncpus=16:ompthreads=8
  

The particular values for ncpus, mem, ngpus are node-type dependent, and most NCAR systems have more than one available node type. (See system specific documentation for recommended values.)

select=4:ncpus=128:mpiprocs=64:ompthreads=1:mem=235GB

walltime¶

The -l walltime=HH:MM:SS resource directive specifies maximum job duration. Jobs still running when this wall time is exceeded will be terminated automatically by the scheduler.

walltime=HH:MM:SS

job_priority¶

Users may request a specific job priority with the -l job_priority=... resource directive. Valid options are:

job_priority=<regular|premium|economy>

gpu_type¶

For highly heterogeneous systems such as Casper, a resource chunk statement including GPUS may match more than one particular GPU type. The resource specification -l gpu_type=... requests a particular GPU type, removing such ambiguity. Valid options are:

gpu_type=<gp100|v100|a100>

Listing of frequently used #PBS directives¶

qsub script.pbs

 qsub -A <OTHER_ACCOUNT> \
      -N testing \
      -l job_priority=premium \
      script.pbs

Execution environment variables¶

Within the script contents of the job script, it is common for the specifics of the job to depend slightly on the PBS and specific module execution environment. Both running under PBS and loading certain module files create some environment variables that might be useful when writing portable scripts; for example scripts that might be shared among users or executed within several different configurations.

PBS execution environment variables¶

PBS creates a number of environment variables that are accessible within a job's execution environment. Some of the more useful ones are:

NCAR module-specific execution environment variables¶

PBS Job Arrays¶

Occasionally users may want to execute a large number of similar jobs. Such workflows may arise when post-processing a large number of similar files, for example, often with a serial post-processing tool. One approach is simply to create a unique job script for each. While simple, this approach has some drawbacks, namely scheduler overhead and job management complexity.

PBS provides a convenient job array mechanism for such cases. When using job arrays, the queue script contents can be thought of as a template that is applied repeatedly, for different instances of the PBS_ARRAY_INDEX. Consider the following example:

#PBS -N job_array
#PBS -A <project_code>
### Each array sub-job will be assigned a single CPU with 4 GB of memory
#PBS -l select=1:ncpus=1:mem=4GB
#PBS -l walltime=00:10:00
#PBS -q casper
### Request 11 sub-jobs with array indices spanning 2010-2020 (input year)
#PBS -J 2010-2020
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program on a specific input file:
./executable_name data.year-${PBS_ARRAY_INDEX}

PBS job arrays provide a convenient mechanism to launch a large number of small, similar jobs. The approach outlined above is particularly suitable for Casper, where nodes are typically shared and individual CPU cores are scheduled. This allows a job array sub-job to be as small as a single core.

When entire compute nodes are assigned to jobs (and therefore also array sub-jobs) we need a slightly different approach, as employed in the following use case.

Using job arrays to launch a "command file"¶

Multiple Program, Multiple Data (MPMD) jobs run multiple independent, typically serial executables simultaneously. Such jobs can easily be dispatched with PBS job arrays, even on machines like Derecho where compute nodes are exclusively and entirely assigned to a users' job. This process is outlined in the example below.

# this is a comment line for demonstration
./cmd1.exe < input1 # comments are allowed for steps too, but not required
./cmd2.exe < input2
./cmd3.exe < input3
...
./cmdN.exe < inputN

# launches the commands listed in ./cmdfile:
launch_cf -A PBS_ACCOUNT -l walltime=1:00:00

# launches the OpenMP-threaded commands listed in ./omp_cmdfile:
#  (requires ppn=128 = (32 steps/node) * (4 threads/step)
launch_cf -A PBS_ACCOUNT -l walltime=1:00:00 --nthreads 4 --steps-per-node 32 ./omp_cmdfile

launch_cf <-h|--help>
     <--queue PBS_QUEUE>
     <--ppn|--processors-per-node #CPUS>
     <--steps-per-node #Steps/node>
     <--nthreads|--threads-per-step #Threads/step>
     <--mem|--memory RAM/node>
     -A PBS_ACCOUNT -l walltime=01:00:00
     ... other PBS args ...
     <command file>

------------------------------------------------------------------
All options in "<>" brackets are optional.
Any unrecognized arguments are passed through directly to qsub.
The PBS options -A and -l walltime are required at minimum.

Sample PBS job scripts¶

Casper¶

#!/bin/bash -l
### Job Name
#PBS -N htc_job
### Charging account
#PBS -A <project_code>
### Request one chunk of resources with 1 CPU and 10 GB of memory
#PBS -l select=1:ncpus=1:mem=10GB
### Allow job to run up to 30 minutes
#PBS -l walltime=30:00
### Route the job to the casper queue
#PBS -q casper
### Join output and error streams into single file
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Load Conda/Python module and activate NPL environment
module load conda
conda activate npl

### Run analysis script
python myscript.py datafile.dat

#!/bin/tcsh
### Job Name
#PBS -N htc_job
### Charging account
#PBS -A <project_code>
### Request one chunk of resources with 1 CPU and 10 GB of memory
#PBS -l select=1:ncpus=1:mem=10GB
### Allow job to run up to 30 minutes
#PBS -l walltime=30:00
### Route the job to the casper queue
#PBS -q casper
### Join output and error streams into single file
#PBS -j oe

setenv TMPDIR ${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Load Conda/Python module and activate NPL environment
module load conda
conda activate npl

### Run analysis script
python myscript.py datafile.dat

#!/bin/bash -l
#PBS -N mpi_job
#PBS -A <project_code>
#PBS -l select=2:ncpus=4:mpiprocs=4:ngpus=4:mem=40GB
#PBS -l gpu_type=v100
#PBS -l walltime=01:00:00
#PBS -q casper
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Provide CUDA runtime libraries
module load cuda

### Run program
mpirun ./executable_name

#!/bin/tcsh
### Job Name
#PBS -N mpi_gpu_job
### Charging account
#PBS -A <project_code>
### Request two resource chunks, each with 4 CPUs, GPUs, MPI ranks, and 40 GB of memory
#PBS -l select=2:ncpus=4:mpiprocs=4:ngpus=4:mem=40GB
### Specify that the GPUs will be V100s
#PBS -l gpu_type=v100
### Allow job to run up to 1 hour
#PBS -l walltime=01:00:00
### Route the job to the casper queue
#PBS -q casper
### Join output and error streams into single file
#PBS -j oe

setenv TMPDIR ${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Provide CUDA runtime libraries
module load cuda

### Run program
mpirun ./executable_name

#!/bin/bash -l
#PBS -N OpenMP_job
#PBS -A <project_code>
#PBS -l select=1:ncpus=8:ompthreads=8
#PBS -l walltime=00:10:00
#PBS -q casper
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
./executable_name

#!/bin/tcsh
#PBS -N OpenMP_job
#PBS -A <project_code>
#PBS -l select=1:ncpus=8:ompthreads=8
#PBS -l walltime=00:10:00
#PBS -q casper
#PBS -j oe

setenv TMPDIR ${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
./executable_name

#!/bin/bash -l
#PBS -N hybrid_job
#PBS -A <project_code>
#PBS -l select=2:ncpus=8:mpiprocs=2:ompthreads=4
#PBS -l walltime=00:10:00
#PBS -q casper
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
mpirun ./executable_name

#!/bin/tcsh
#PBS -N hybrid_job
#PBS -A <project_code>
#PBS -l select=2:ncpus=8:mpiprocs=2:ompthreads=4
#PBS -l walltime=00:10:00
#PBS -q casper
#PBS -j oe

setenv TMPDIR ${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
mpirun ./executable_name

#!/bin/bash -l
#PBS -N job_array
#PBS -A <project_code>
### Each array subjob will be assigned a single CPU with 4 GB of memory
#PBS -l select=1:ncpus=1:mem=4GB
#PBS -l walltime=00:10:00
#PBS -q casper
### Request 10 subjobs with array indices spanning 2010-2020 (input year)
#PBS -J 2010-2020
#PBS -j oe

export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
./executable_name data.year-$PBS_ARRAY_INDEX

#!/bin/tcsh
#PBS -N job_array
#PBS -A <project_code>
### Each array subjob will be assigned a single CPU with 4 GB of memory
#PBS -l select=1:ncpus=1:mem=4GB
#PBS -l walltime=01:00:00
#PBS -q casper
### Request 10 subjobs with array indices spanning 2010-2020 (input year)
#PBS -J 2010-2020
#PBS -j oe

setenv TMPDIR ${SCRATCH}/temp
mkdir -p ${TMPDIR}

### Run program
./executable_name data.year-$PBS_ARRAY_INDEX

qsub -W "depend=afterok:317485[]" postprocess.pbs

#PBS -l select=1:ncpus=8:mpiprocs=8:mem=60GB:ngpus=1:mps=1

#!/bin/bash
#PBS -A <project_code>
#PBS -N gpu_mps_job
#PBS -q casper@casper-pbs
#PBS -l walltime=01:00:00
#PBS -l select=2:ncpus=36:mpiprocs=36:ngpus=4:mem=300GB:mps=1
#PBS -l gpu_type=v100

# Use scratch for temporary files to avoid space limits in /tmp
export TMPDIR=${SCRATCH}/temp
mkdir -p ${TMPDIR}

# Load modules to match compile-time environment
module purge
module load ncarenv nvhpc/22.5 cuda/11.4 openmpi/4.1.4

# Run application using Open MPI
mpirun ./executable_name

Derecho¶

#PBS -A <project_code>
#PBS -N hybrid_job
#PBS -q main
#PBS -l walltime=01:00:00
#PBS -l select=2:ncpus=128:mpiprocs=32:ompthreads=4

# Load modules to match compile-time environment
module purge
module load ncarenv/23.09 intel-oneapi/2023.2.1 craype/2.7.23 cray-mpich/8.1.27

# Run application with MPI binding helper script
mpibind ./executable_name

# Or run application using cray-mpich with explicit binding
# mpiexec --cpu-bind depth -n 64 -ppn 32 -d 4 ./executable_name

#!/bin/bash
#PBS -A <project_code>
#PBS -N gpu_job
#PBS -q main
#PBS -l walltime=01:00:00
#PBS -l select=2:ncpus=64:mpiprocs=4:ngpus=4

# Load modules to match compile-time environment
module purge
module load ncarenv/23.09 nvhpc/24.1 cuda/12.2.1 cray-mpich/8.1.27

# (Optional: Enable GPU managed memory if required.)
#   From ‘man mpi’: This setting will allow MPI to properly
#   handle unify memory addresses. This setting has performance
#   penalties as MPICH will perform buffer query on each buffer
#   that is handled by MPI)
# If you see runtime errors like
# (GTL DEBUG: 0) cuIpcGetMemHandle: invalid argument,
#  CUDA_ERROR_INVALID_VALUE
# make sure this variable is set
export MPICH_GPU_MANAGED_MEMORY_SUPPORT_ENABLED=1

# Run application using the cray-mpich MPI
#   The ‘set_gpu_rank’ command is a script that sets several GPU-
#   related environment variables to allow MPI-enabled GPU
#   applications to run. The set_gpu_rank script is detailed
#   in the binding section below, and is also made available
#   via the ncarenv module.
mpiexec -n 8 -ppn 4 set_gpu_rank ./executable_name

#!/bin/bash

export MPICH_GPU_SUPPORT_ENABLED=1
export LOCAL_RANK=${PMI_LOCAL_RANK}
export GLOBAL_RANK=${PMI_RANK}
export CUDA_VISIBLE_DEVICES=$(expr ${LOCAL_RANK} % 4)

echo "Global Rank ${GLOBAL_RANK} / Local Rank ${LOCAL_RANK} / CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES} / $(hostname)"

exec $*

#PBS -l select=2:ncpus=64:mpiprocs=4:ngpus=4
...
# Run application using the cray-mpich MPI, binding the local
# mpi rank [0-3] to corresponding GPU index [0-3]:
mpiexec -n 8 -ppn 4 ./set_gpu_rank ./executable_name

mpiexec -n 8 -ppn 4 --cpu-bind verbose,list:0:16:32:48 ./set_gpu_rank ./executable_name

#!/bin/bash
#PBS -q main
#PBS -j oe
#PBS -o fasteddy_job.log
#PBS -l walltime=02:00:00
#PBS -l select=6:ncpus=64:mpiprocs=4:ngpus=4

module load ncarenv/23.09
module load apptainer gcc cuda || exit 1
module list

nnodes=$(cat ${PBS_NODEFILE} | sort | uniq | wc -l)
nranks=$(cat ${PBS_NODEFILE} | sort | wc -l)
nranks_per_node=$((${nranks} / ${nnodes}))

container_image="./rocky8-openhpc-fasteddy.sif"

singularity \
    --quiet \
    exec \
    ${container_image} \
    ldd /opt/local/FastEddy-model/SRC/FEMAIN/FastEddy

singularity \
    --quiet \
    exec \
    --bind ${SCRATCH} \
    --bind ${WORK} \
    --pwd $(pwd) \
    --bind /run \
    --bind /opt/cray \
    --bind /usr/lib64:/host/lib64 \
    --env LD_LIBRARY_PATH=${CRAY_MPICH_DIR}/lib-abi-mpich:/opt/cray/pe/lib64:${LD_LIBRARY_PATH}:/host/lib64 \
    --env LD_PRELOAD=/opt/cray/pe/mpich/${CRAY_MPICH_VERSION}/gtl/lib/libmpi_gtl_cuda.so.0 \
    ${container_image} \
    ldd /opt/local/FastEddy-model/SRC/FEMAIN/FastEddy



echo "# --> BEGIN execution"; tstart=$(date +%s)

mpiexec \
    --np ${nranks} --ppn ${nranks_per_node} --no-transfer \
    set_gpu_rank \
    singularity \
    --quiet \
    exec \
    --bind ${SCRATCH} \
    --bind ${WORK} \
    --pwd $(pwd) \
    --bind /run \
    --bind /opt/cray \
    --bind /usr/lib64:/host/lib64 \
    --env LD_LIBRARY_PATH=${CRAY_MPICH_DIR}/lib-abi-mpich:/opt/cray/pe/lib64:${LD_LIBRARY_PATH}:/host/lib64 \
    --env LD_PRELOAD=/opt/cray/pe/mpich/${CRAY_MPICH_VERSION}/gtl/lib/libmpi_gtl_cuda.so.0 \
    --env MPICH_GPU_SUPPORT_ENABLED=1 \
    --env MPICH_GPU_MANAGED_MEMORY_SUPPORT_ENABLED=1 \
    --env MPICH_SMP_SINGLE_COPY_MODE=NONE \
    ${container_image} \
    /opt/local/FastEddy-model/SRC/FEMAIN/FastEddy \
    ./Example02_CBL.in

echo "# --> END execution"
echo $(($(date +%s)-${tstart})) " elapsed seconds; $(date)"

#!/bin/bash
#PBS -N multi-mpi
#PBS -l select=2:ncpus=128:mpiprocs=128
#PBS -l walltime=04:00:00
#PBS -q main
#PBS -A <project_code>

# *** Job Configurables ***
num_runs=8   # Number of concurrent MPI applications running
ppr=32       # Processes per run
ppn=128      # Processes per node

# Load explicit module versions to preserve reproducibility
module purge
module load ncarenv/23.09 intel/2024.0.2 cray-mpich/8.1.27

# Define driver function to set up and start model runs
function run_model {
    ni=$((ppr * ($1 - 1) / ppn))    # Index of node to use
    sc=$((ppr * ($1 - 1) % ppn))    # Starting core of range to bind to
    ec=$((sc + ppr - 1))            # Ending core of range to bind to

    mkdir run-$1; cd run-$1         # Create unique directory for each run
    ln -s ../model .                # Reuse same executable via symbolic linking
    mpiexec -host ${nodes[$ni]} -n $ppr --cpu-bind list:$sc-$ec ./model &> outerr.log
    cd ../
}

# Store node list in a bash array
nodes=( $(uniq $PBS_NODEFILE) )

# Start our independent runs as background processes
for run in $(seq $num_runs); do
    run_model $run &
done

# Block job exit until all processes are finished
wait