dwarf-p-cloudsc

dwarf-p-cloudsc is intended to test the CLOUDSC cloud microphysics scheme of the IFS.

The dwarf comes with many implementations:

• For CPU benchmarking, the relevant binary is dwarf-cloudsc-fortran.
• For GPU performance, the variants are all incarnations of the same 4 optimizations:
  1. SCC: The baseline GPU adaptation – loop order flipped, as many temporaries demoted to scalars as possible, Pragma annotations for offload
  2. SCC-hoist: Same as above, and additionally allocation of non-demoted temporaries hoisted to the driver
  3. SCC-stack: Same as hoist, but using a single memory pool and Cray pointers for all temporaries instead of hoisting them as individual allocations; This is the variant closest to what we expect to use in the full IFS model in the foreseeable future.
  4. SCC-k-caching: Aggressively optimised variant with loop fusion to demote additional temporaries to scalars; remaining temporaries with loop-carried dependencies are demoted to 2- or 3-entry stencils

Variants 1,2, 4 are available as OpenACC implementations in the repository (dwarf-cloudsc-gpu-scc[-hoist/k-caching]). Variant 2 is available as OpenMP-target implementation in the repository (dwarf-p-cloudsc-gpu-omp-scc-hoist). Variants 1-3 are also available as build-time transformed variants using the Loki source-to-source translation toolchain (dwarf-cloudsc-loki-scc[-hoist, -stack]). This resembles the build mode used for GPU-offloading physics in IFS.

In HPCW, we include test cases for the fastest variant per GPU programming model:

• OpenACC: dwarf-cloudsc-gpu-scc-k-caching
• OpenMP: dwarf-cloudsc-gpu-omp-scc-hoist
• CUDA: dwarf-cloudsc-c-cuda-k-caching
• HIP: dwarf-cloudsc-hip-k-caching

For information on the names of the associated test cases, see here.

Building CLOUDSC

If you want to build the GPU variants of CLOUDSC, set the CMake variable ENABLE_cloudsc_gpu=ON. This is required for the GPU versions and enables building/testing for every version. If you do not want to build each GPU variant, you may specify which GPU variants you want to exclude by setting the corresponding CMake variable to OFF, e.g. ENABLE_cloudsc_hip=OFF. For an example, refer to this toolchain. It also contains information on how to set the compiler flags for the different programming models. If you want to execute multi-GPU runs with CLOUDSC, MPI support needs to be enabled as well, as each GPU needs to be assigned to one MPI rank.

CLOUDSC can be built in either single precision or double precision mode (default). The single precision mode is enabled via the CMake variable ENABLE_cloudsc_single_precision.

Remarks on the OpenMP variant

During our tests, we noticed that compiling the OpenMP variant of CLOUDSC with NVHPC also required OpenACC to be active, otherwise the compiler would throw errors. If you encounter a similar problem, make sure to either activate the OpenACC version as well, or pass the CMake variable -DENABLE_ACC=ON to cloudsc_cmake_args when building the OpenMP version.

Regarding externally-built versions

As with the other applications, you can also use externally-built CLOUDSC GPU versions and run the HPCW test cases with them. However, if you want to run multiple test cases simultaneously, you might run into crashes due to HDF5 file locks. This can be prevented by setting the environment variable HDF5_USE_FILE_LOCKING=FALSE.

Both the CMake build and the spack package provided by HPCW contain patches that make this environment variable obsolete by adjusting the HDF5 file accesses, so you only need to worry about this if you use a CLOUDSC version built via other means.

Testing CLOUDSC

CLOUDSC requires three parameters. All of them are set by HPCW, but you can influence some of them. The first parameter is the number of threads. The test cases rely on the environment variable OMP_NUM_THREADS to determine this value at the time of test execution. The second parameter is the problem size. This value is fixed per test case. The last parameter is the block size, which mostly affects cache utilization. HPCW sets sensible defaults for each test case. For the test cases cloudsc-fortran-{small, medium, big} you can choose your own block size and specify it via the CMake variable cloudsc_block_size.