Danger
This page is tailored to experienced users and collaborators developing ACCESS models.
This step is not required if you only want to run a model. If you are looking for information on how to run a model, refer to the Run a Model section.
Modify and build an ACCESS model's source code
About
The following instructions below outline how to build an ACCESS model and its dependencies, using the build-from-source package manager Spack.
This build workflow is specifically designed to run on the National Computating Infrastructure (NCI) supercomputer Gadi.
The following instructions outline how to trigger a prerelease build of [ACCESS-OM2][om2 config] after modifying its [MOM5 component]. All other components of the official [ACCESS-OM2 release][om2 repo] will remain unchanged.
As an example, in the following instructions we will show how to modify MOM5 component for ACCESS-ESM1.5 and re-compile the relevant ACCESS-ESM1.5 dependencies. All other components and packages (i.e., dependencies) of the official ACCESS-ESM1.5 release will remain unchanged.
Tip
The following instructions are valid (with simple tweaks) for all [ACCESS models].
Prerequisites
-
NCI account
Before building a model, you need to Set Up your NCI Account. -
Spack
To set up Spack on Gadi, refer to Set up Spack for building ACCESS models.
Navigate into your Spack directory
Navigate into the directory where you cloned the Spack repositories during the Spack setup.
The suggested directory is /g/data/$PROJECT/$USER/spack/0.22:
cd /g/data/$PROJECT/$USER/spack/0.22
Enable spack
Warning
This step needs to be carried out for any new login or new shell environment.
To add the spack command to your shell, as well as other settings, run:
module purge
. spack-config/spack-enable.bash
Warning
There is a space between the . and the path to the file, as we are sourcing the file.
Create a Spack development environment
Spack has environments that are, in some ways, similar to Conda environments or Python virtual environments: they create an isolated operating environment where Spack can only see and access specific packages. This allows the build and deployment of new packages in a coherent fashion.
Spack managed environments are located inside /g/data/$PROJECT/$USER/spack/0.22/environments, each in a separate directory. Each environment configuration is defined within its directory using the spack.yaml file.
Warning
Spack managed environments' location can be changed within Spack configuration files and the directory specified above represents the default location for a Spack instance that has been set up following the Spack setup instructions.
Create the environment
To ensure we keep all dependecies the same as the official release, we will create our Spack development environment as a copy of the released ACCESS-ESM1.5 Spack environment.
To create a Spack development environment called mom5_dev, run:
git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5.git
spack env create mom5_dev ACCESS-ESM1.5/spack.yaml
The newly-created mom5_dev Spack environment folder is /g/data/$PROJECT/$USER/spack/0.22/environments/mom5_dev.
Activate the environment
To activate the mom5_dev Spack environment, run:
spack env activate -p mom5_dev
Compile Spack environment packages (optional)
It is recommended to first compile all the packages in the newly created Spack environment as is, without making any changes.
Compiling all the packages present in a Spack environment is referred to as installing the environment.
Concretize the Spack environment
Concretizing the Spack environment is necessary anytime the environment's spack.yaml gets changed, to force Spack to update its knowledge of the environment configuration.
To concretize the mom5_dev environment, run:
spack concretize -f --reuse-deps
Warning
This command might take a few minutes to complete
If the command above fails, try running the following command instead:
spack concretize -f --fresh
Install the Spack environment
To compile the packages in the mom5_dev environment, run:
spack install
Tip
It takes 30-40 minutes to compile all the packages for the first time.
Subsequent installations, however, will compile quicker as the built dependencies are reused.
Info
The full output has been truncated for brevity.
Create development package
When you develop a package within a Spack environment, Spack needs to know that the desired package is marked as "in development", and be able to access its source code.
This is done through the spack develop command.
To mark a package as a development package, the general command to run is:
spack develop <package_specifier>
Tip
This command should not display any output
When specifying a Spack development package, there are 3 elements that can be set within the package_specifier:
- package name (required)
- package source code (required)
- package Spack version (optional)
The spack develop command adds the following lines at the end of the spack.yaml file inside the environment's folder:
develop:
<package_name>:
spec: <package_specifier>
Info
In the case of a development package with a local source code, the following line is also added:
path: <source_path>
Warning
After setting a development package, it is important to also fix any inconsistencies within the spack.yaml file.
Specify the package name element
The package name identifies the package to be set for development.
For example, in the case of mom5, the package name should be exactly mom5.
Specify the package source code element
In general, a package source code can be:
Specify the source code cloned from a git repository
For remote packages, the source code can be specified as a git reference in the form:
<package_name>@git.<git_reference>
Tip
The git_reference can be either a branch, tag or commit hash.
For example, for a mom5 package with source code residing in the development branch, the package specifier would be mom5@git.development.
I this case, the development branch of the mom5 repository would be automatically cloned and used as source code for the mom5 development package, that can be later modified.
Info
The source code is automatically cloned in the environment's folder.
The repository URL is set within the package definition file package.py.
In the case of mom5, the package definition file is in spack-packages/packages/mom5/package.py.
For more information about Spack packages definition, please refer to Creating new spack packages.
Specify the source code from a local folder
To set a local path as the source code of a Spack development package, the path needs to be specified directly within the spack develop command, by adding the --path <path_to_source_code> option.
In this case, no git reference should be provided.
For example, if the mom5 development package's source code resides in /path/to/mom5/new/source/code, the command to run would be:
spack develop --path /path/to/mom5/new/source/code mom5
Warning
Care needs to be taken when multiple Spack development environments point to the same source code location. If these environments require different independent changes of the source code, the user needs to make sure to sync the source code version (e.g., using different git branches for the different versions of the source code) with the desired one when switching between development environments.
This would still prevent building both environments simultaneously.
Specify the package Spack version element
A Spack version can be assigned to a development package by setting a version specifier.
The syntax for the version specifier varies depending whether the package source code is remote or local.
Specify the package Spack version for a remote package source
If the development package's source code is to be cloned from git, the package Spack version can be set by appending =<package_version> to the package specifier.
For example, to develop mom5 code from the the development branch and build it as Spack version access-om2, run:
spack develop mom5@git.development=access-om2
Specify the package Spack version for a local package source
When the development package's source code is local, no git reference is provided.
In this case a package Spack version can be added by appending @<package_version> to the package specifier.
For example, to develop mom5 code from the /path/to/mom5/new/source/code folder and build it as Spack version access-om2, run:
spack develop --path /path/to/mom5/new/source/code mom5@access-om2
Tip
When in doubt about which Spack version to assign to a specific package, a useful command to retrieve the existing versions of a package is:
spack versions <package_name>
Warning
It is strongly recommended to specify a Spack version, as Spack always requires a version to be associated with a development package.
If no Spack version is specified by the user:
- If the package has a git reference, the Spack version will be taken from the closest valid git tag among ancestors of the git reference.
- If the package source code is local, an error will be thrown.
Fix inconsistencies within the environment file
At times, setting development packages might cause inconsistencies within the spack.yaml environment file.
This occurs whenever an environment contains a required package with the same name as the development package.
For example, the mom5_dev environment spack.yaml file contains the following lines:
spack:
specs:
- access-esm1p5@git.2024.12.0
packages:
# other package ...
# other package ...
mom5:
require:
- '@git.access-esm1.5_2024.08.23=access-esm1.5'
# other package ...
# other package ...
# other specifications ...
# other specifications ...
modules:
default:
tcl:
include:
# other package ...
# other package ...
mom5
# other package ...
# other package ...
projections:
# other package ...
# other package ...
mom5: '{name}/access-esm1.5_2024.08.23-{hash:7}'
# other package ...
# other package ...
# other specifications ...
# other specifications ...
mom5_dev environment depends on mom5 version access-esm1.5, coming from the git reference access-esm1.5_2024.08.23, and the module produced after compiling the mom5 package can be loaded with module load mom5/access-esm1.5_2024.08.23-<hash_of_the_compiled_package>.
If mom5 is set as a development package, the spack.yaml file needs to be edited to:
- comment out any
spack.packagesentry with the same name as the development package - change the
modules.default.tcl.projectionfor the development package.
A simple way to manually edit the spack.yaml file is to run:
spack config edit
For example, after setting mom5 as a development package, the final spack.yaml file should be modified to look similar to the following:
spack:
specs:
- access-esm1p5@git.2024.12.0
packages:
# other package ...
# other package ...
# mom5:
# require:
# - '@git.access-esm1.5_2024.08.23=access-esm1.5'
# other package ...
# other package ...
# other specifications ...
# other specifications ...
modules:
default:
tcl:
include:
# other package ...
# other package ...
mom5
# other package ...
# other package ...
projections:
# other package ...
# other package ...
mom5: '{name}/<custom_name_for_development_package_module>-{hash:7}'
# other package ...
# other package ...
# other specifications ...
# other specifications ...
develop:
mom5:
spec: <package_specifier>
path: /path/to/mom5/source/code # Only if a local source code was specified
Compile modified Spack environment packages
After setting a development package the Spack environment needs to be re-concretized (because the spack.yaml file changed). The Spack environment can be concretized following the same steps listed in Concretize the Spack environment. Then, the new package can be built following the steps listed in Compile Spack environment packages:
spack concretize -f --reuse-deps
spack install
Warning
Although this time the spack install command will only build the development package, it might still take a long time to complete, depending on the specific package.
If the concretization step fails, try running the following command instead:
spack concretize -f --fresh
Info
The full output has been truncated for brevity.
Tip
From now on, the source code can be modified and the Spack environment installed without repeating the concretization step.
The Spack environment will need to be re-concretized only if further changes occur in the spack.yaml file.
Output directory for compiled packages
Tip
For the Spack instance obtained through the Spack setup instructions, $spack(referred to as $(prefix) in Spack configuration scopes documentation) corresponds to the /g/data/$PROJECT/$USER/spack/0.22/spack directory.
For the Spack instance obtained through the Spack setup instructions, all compiled packages will be placed in directories having the following format: <install_tree.root>/<architecture>/<compiler.name>-<compiler.version>/<name>-<version>-<hash>.
<install_tree.root> depends on the install_tree.root configuration field. Spack reads this configuration field from files in several directories, following Spack's configuration scopes.
Warning
For instances of Spack on Gadi you should ignore the system scope.
For the example above, mom5_dev Spack environment's configuration file (spack.yaml) contains the following lines that fall in the environment scope:
config:
install_tree:
root: $spack/../restricted/ukmo/release
This means the packages built in this example can be found in /g/data/$PROJECT/$USER/spack/0.22/spack/../restricted/ukmo/release/<architecture>/<compiler.name>-<compiler.version>/<name>-<version>-<hash>.
Troubleshooting build errors
Sometimes you might encounter errors while compiling the packages.
Spack prints out the error message and generates a full build log that can be viewed by the user. The location of the build log is shown at the end of the error message.
For example, if we try to install the mom5_dev environment with an error in the new mom5 source code (in this example a use statement in the <new-mom5-source-code-folder>/src/accessom_coupler/ocean_solo.F90 file has been purposely commented out to force an error in compilation), we might get an output error similar to the following:
...
>> 415 /g/data/$PROJECT/$USER/spack/0.22/environments/mom5_dev/mom5/src/access_coupler/ocean_solo.F90:224: undefined reference to `constant s_init_'
>> 416 make: *** [Makefile:931: fms_ACCESS-CM.x] Error 1
...
See build log for details:
/scratch/$PROJECT/$USER/tmp/path/to/the/spack-stage-mom5-git.access-esm1.5_2024.08.23_access-esm1.5-l34w7is54xzer7s4ztvb5ymgjbtduknh/spack-build-out.txt
...
==> Error: access-esm1p5-git.2024.05.1=2024.05.1-aysea5r7rbwy22lluvl64baperlokktv: Package was not installed
==> Error: Installation request failed. Refer to reported errors for failing package(s).If the error is not obvious from the error message, see the build log for more information.