- Generated by
1.11.0
|
HPCW 3.0
|
HPCW provides many different models and test cases. It is up to the user whether they want to build the models with the help of HPCW or if they want to use an externally built model with the provided test cases. This guide will go over the basics for both scenarios.
If you want to build the models in the context of HPCW, you need to provide some system-specific files to inform HPCW of available compilers, of how to run applications on your system etc. The following toolchain files need to be provided:
You can find examples for these in the toolchains directory. For some systems, the necessary files are already part of HPCW. In this case, you can immediately build and run whichever test case you want, using the recommended build wrapper.
To get started, create the environment file and the CMake toolchain file. In HPCW, the environment files usually end in .env.sh, while the CMake toolchain file ends with .cmake.
With these files created, now is the time to decide on the compiler that you want to use for the builds. On many systems, this requires loading the compilers, either via module load or spack load. This command should be added to the environment file. Afterwards, it is recommended to export the corresponding compilers in the variables CC, CXX and FC. In the CMake toolchain file, you can also use the set command to set the corresponding CMake variables explicitly, for example:
Note that CMake generally picks up on the environment variables for the different compilers (CC etc), so the explicit use of the set commands demonstrated above is not necessary most of the time, if you already specified the appropriate environment variables.
After choosing the compilers, the next step is to choose an MPI implementation. Similarly to the compilers, if you need to load a module, do so in the environment file. The presence of mpicc on your $PATH should be enough for CMake to subsequently be able to detect the MPI installation.
If you want to use the recommended build wrapper for building and testing, the next step is to define a variable called cmakeFlags in your environment file. The build wrapper contains a variable with the same name that is initially populated with the definition in the environment file. cmakeFlags is used to define arguments that you want to pass to the CMake builds afterwards. For the start, we only need to inform CMake of the CMake toolchain file that we want to use. In order to do this, add the following lines to your environment file:
After creating these files, the minimal setup for building models with HPCW is complete. In order to build a model, you can now use the previously created environment and CMake toolchain file with the build wrapper as such:
$HPCW corresponds to the root directory of HPCW. Invoking the build wrapper like this would build NICAM-DC with the previously selected compilers and MPI implementation.
When starting out with HPCW, we recommend building the "simpler" models first. "Simplicity" in this case is measured by the number of dependencies outside of the compilers and MPI. Following this terminology, the simplest model is NICAM-DC, as it boasts no other dependencies and only runs on CPUs. For detailed information on building NICAM-DC, refer to this part of the documentation.
For completeness, ICON would be the least "simple" model according to this definition, with NEMO and the IFS dwarfs being inbetween NICAM-DC and ICON in no particular order.
Once you are able to build NICAM-DC, you will likely want to run test cases with it. At this point, you need to create the third file that was mentioned previously, the job launcher. This launcher is used by each test case of HPCW, providing the necessary information to run each test case on your system. This file depends heavily on your system and what scheduler you are using. You can find many Slurm-based examples in the toolchains directory. The names of the different test cases are all documented here. Some test cases, for example the ICON ones, have multiple stages with differing resource requirements. For details, refer to the documentation for the different models.
Once you have created the job launcher, you can incorporate it into your CMake toolchain by adding the following line:
Alternatively, you can also append this to the cmakeFlags in your environment file, though we recommend the approach using the CMake toolchain file for the most part.
With this done, your environment and CMake toolchain file might look akin to this:
Note that we did not add any compile flags to the toolchain or the environment. While setting your own flags is not strictly necessary, the default flags may not be suitable for your system. You can look at the existing toolchains as examples of how you can set different flags for the models. As a quick reference, most toolchains export the flags in the environment as such:
Both Autotools and CMake automatically pick up on these environment variables, with one exception: CMake does not recognize the FCFLAGS environment variable. If you want to use the same Fortran flags for the Autotools-based models and the CMake-based ones, you should set the following variable in your CMake toolchain file:
With the model built, you can execute any given test case by invoking the build wrapper as such:
This example invocation of the build wrapper would launch the test case nicamdc-small (--ctest-flags are used in the ctest call inside the build wrapper) with the resources defined in your job launcher. If you wanted to run all NICAM-DC test cases, you can remove the --ctest-flags arguments from above.
If the tests fail, the exit code of the build wrapper will tell you, plus you will get extensive output from CTest, telling you which tests failed. If the tests succeed, the exit code of the build wrapper will be 0, and CTest also logs all executed tests as successful. Alongside this, the build wrapper produces some small analysis files per default, a CSV file and an OpenMetrics file. They are stored in the build directory (currently they are called analyse.csv and metrics.txt). These analysis files contain details on the exit codes and the execution times of all test cases, so you can use these files for comparisons later on.
This minimal setup is enough to build all models that are part of HPCW. However, in this case every single dependency would need to be built alongside the corresponding model. While HPCW can and will do this for you, building the likes of NetCDF and HDF5 can be very time consuming. Many systems already have a lot of the dependencies pre-installed and sometimes even tuned to the specific architecture. In this case, you can use these dependencies instead of the ones that are built by HPCW. This requires one additional line in the CMake toolchain file for every system-provided dependency. For example, if there is a cdo installation available on your system, you can add this line to the CMake toolchain file:
Now HPCW will look for the cdo executable at build time. If you need to load a module for the executable to be available in your $PATH, you can do so in the environment file. There are many examples regarding this functionality in the existing toolchains. You can also refer to this section of the documentation for more information regarding system-provided packages. This section of the documentation contains some more information regarding the toolchain creation.
If you want to run the HPCW test cases with a model version built outside of HPCW, you can instruct HPCW to search your system for the pre-installed model as well. This enables many different workflows, for example you can install the models ahead of time via spack or test source code changes for the models and assess their impact on performance. Regardless of your use case, the procedure to set up HPCW accordingly is the same.
First, make sure that the executable for the specific model is available in your $PATH. Afterwards, you can instruct HPCW to use the system-provided model by setting the corresponding USE_SYSTEM_<model> CMake variable to ON, similar to the usage of system-provided dependencies. For reference, the HPCW-internal model names are:
For example, if you want to use an externally built version of ICON, you can instruct HPCW to do so by adding the following line to your CMake toolchain file:
Alternatively, you can also set the variable when invoking the build wrapper, like this:
This invocation of the build wrapper would set up all ICON-specific tests and also run them with your externally built version of ICON.
Note that you are still required to provide the toolchain files. The job launcher in particular is still necessary to define the resource utilization per test case. Refer to the previous section for some tips regarding these files.
While the models in HPCW are all climate and weather codes, their specific usage can vary greatly. For example, ecTrans is controlled exclusively via command line arguments and requires little to no additional changes from users, while ICON relies solely on Fortran Namelists, which can be (and in some cases even need to be) heavily customized for certain tests. Refer to the model-specific sections of the HPCW documentation for hints regarding the respective models. When in doubt, you can also always read the respective models' own documentation, or you can ask for support with HPCW at suppo.nosp@m.rt@d.nosp@m.krz.d.nosp@m.e.
1.11.0