Bug
There is a known issue with the ACCESS-ESM configurations currently set up in the GitHub repository below. ACCESS-ESM configurations are currently being updated and an ACCESS-NRI release will come in the following weeks. If you encounter problems with the current guide in this page, consider asking for help on the ACCESS-Hive Forum.
Run ACCESS-ESM
Prerequisites
General prerequisites
Before running ACCESS-ESM, you need to fulfil general prerequisites outlined in the First Steps section.
If you are unsure whether ACCESS-ESM is the right choice for your experiment, take a look at the overview of ACCESS Models.
Model-specific prerequisites
ACCESS-ESM is installed on NCI's supercomputer Gadi and uses payu, a tool for running and managing model experiments. Following these prerequisites ensures you have access to this infrastructure.
- Join the access and hh5 projects at NCI
To join these projects, request membership on the respective access and hh5 NCI project pages.
For more information on joining specific NCI projects, refer to How to connect to a project. - Payu
Payu on Gadi is available through theconda/analysis3environment in the hh5 project.
After obtaining hh5 project membership, load theconda/analysis3environment to automatically retrieve payu as follows:To check that payu is available, run:module use /g/data/hh5/public/modules module load conda/analysis3payu --versionpayu --version 1.0.19
Get ACCESS-ESM configuration
ACCESS-ESM configurations are available in the coecms GitHub repository.
To get it on Gadi, clone the ACCESS-ESM GitHub repository by running:
git clone https://github.com/coecms/access-esm.git
access-esm folder.
Warning
Some modules may interfere with git commands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again:
module purge
conda/analysis3 module to retrieve payu, as specified in the Model-specific prerequisites section.
Each ACCESS-ESM configuration is stored as a specially-named branch in the ACCESS-ESM GitHub repository.
To check all the available branches on the repo, run the following command inside the newly-created access-esm folder:
git branch -a
The green-coloured branch (preceded by a star sign *) indicates the local branch you are currently in.
The red-coloured branches are the available remote branches, formatted as remotes/origin/<branch-name>.
To switch to a specific branch you can run the following command:
git checkout <branch-name>
pre-industrial branch. To use the pre-industrial configuration, run:
git checkout pre-industrial
Edit ACCESS-ESM configuration
It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration.
To create a new branch called example_run, as a copy of the pre-industrial branch, from within the access-esm directory execute:
git checkout -b example_run --no-track origin/pre-industrial
This command will also switch to the new example_run branch.
Payu
Payu is a workflow management tool for running numerical models in supercomputing environments.
The general layout of a payu-supported model run consists of two main directories:
- The laboratory directory, where all the model components reside. For ACCESS-ESM, it is typically
/scratch/$PROJECT/$USER/access-esm. - The control directory, where the model configuration resides and from where the model is run (in this example, the cloned directory
~/access-esm).
This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the $HOME directory (as it is the only filesystem actively backed-up on Gadi), without overloading it with too much data.
Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously.
To set up the laboratory directory, run the following command from within the control directory:
payu init
work→ a temporary directory where the model is run. It gets cleaned after each run.archive→ the directory where output is stored after each run.
Edit the Master Configuration file
The config.yaml file located in the control directory, is the Master Configuration file.
This file, which controls the general model configuration, contains several parts:
-
PBS resources
These lines can be edited to change the PBS directives for the PBS job.jobname: pre-industrial queue: normal walltime: 3:10:00
For example, to run ACCESS-ESM under thetm70project (ACCESS-NRI), add the following line:project: tm70Warning
The
projectentry should always refer to a project with allocated Service Units (SU), that you are a member of. If not set explicitly, ACCESS-ESM will run using your default project (this default project still needs to have allocated SU). For more information, check how to join relevant NCI projects. -
Link to the laboratory directory
These lines set the laboratory directory path, which is relative to# note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER laboratory: access-esm/scratch/$PROJECT/$USER. Absolute paths can also be specified. -
Model
This line tells payu which driver to use for the main model (model: accessaccessrefers to ACCESS-ESM). -
Submodels
Expand to show the full
submodelssectionsubmodels: - name: atmosphere model: um ncpus: 192 exe: /g/data/access/payu/access-esm/bin/coe/um7.3x input: - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - /g/data/access/payu/access-esm/input/pre-industrial/start_dump<br> - name: ocean model: mom ncpus: 180 exe: /g/data/access/payu/access-esm/bin/coe/mom5xx input: - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial<br> - name: ice model: cice ncpus: 12 exe: /g/data/access/payu/access-esm/bin/coe/cicexx input: - /g/data/access/payu/access-esm/input/pre-industrial/ice<br> - name: coupler model: oasis ncpus: 0 input: - /g/data/access/payu/access-esm/input/pre-industrial/couplerACCESS-ESM is a coupled model deploying multiple submodels (i.e. model components). This section specifies the submodels and configuration options required to execute the model correctly.
Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel'sname(e.g., configuration options for theatmospheresubmodel are in the~/access-esm/esm-pre-industrial/atmospheredirectory). -
Collate
Thecollate: exe: /g/data/access/payu/access-esm/bin/mppnccombine restart: true mem: 4GBcollateprocess combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if therestartoption is set totrue. -
Restart
This is the location of the files used for a warm restart.restart: /g/data/access/payu/access-esm/restart/pre-industrial -
Start date and run length
This section specifies the start date and run length.calendar: start: year: 101 month: 1 days: 1 runtime: years: 1 months: 0 days: 0Warning
The run length (controlled by
runtime) can be different from the total experiment length. Also, whileruntimecan be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between run length and total experiment length, or how to run the model for more than 1 year, refer to the section Run configuration for multiple years. -
Number of runs per PBS submission
ACCESS-ESM configurations are often run in multiple steps (or cycles), with payu running a maximum ofrunspersub: 1runspersubruns for every PBS job submission.Warning
If you increase
runspersub, you may need to increase thewalltimein the PBS resources.
To find out more about other configuration settings for the config.yaml file, check out how to configure your experiment with payu.
Edit a single ACCESS-ESM component configuration
Each of ACCESS-ESM components contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files.
These configuration options are specified in files located inside a subfolder of the control directory, named according to the submodel's name specified in the config.yaml submodels section (e.g., configuration options for the atmosphere submodel are in the ~/access-esm/esm-pre-industrial/atmosphere directory).
To modify these options please refer to the User Guide of the respective model component.
Run ACCESS-ESM configuration
After editing the configuration, you are ready to run ACCESS-ESM.
ACCESS-ESM suites run on Gadi through a PBS job submission managed by payu.
Payu setup (optional)
As a first step, from within the control directory, it is good practice to run:
payu setup
This will prepare the model run, based on the experiment configuration.
Tip
This step can be skipped as it is also included in the run command. However, running it explicitly helps to check for errors and make sure executable and restart directories are accessible.
Run configuration
To run ACCESS-ESM configuration for one run length (controlled by runtime in the config.yaml file), from within the control directory execute:
payu run -f
This will submit a single job to the queue with a total experiment length of runtime. If there is no previous run, it will start from the start date indicated in the config.yaml file. Otherwise, it will perform a warm restart from a previously saved restart file.
Tip
The -f option ensures that payu will run even if there is an existing non-empty work directory created from a previous failed run.
Run configuration for multiple years
If you want to run ACCESS-ESM configuration for multiple run lengths (controlled by runtime in the config.yaml file), use the option -n:
payu run -f -n <number-of-runs>
This will run the configuration number-of-runs times with a total experiment length of runtime * number-of-runs. The number of consecutive PBS jobs submitted to the queue depends on the runspersub value specified in the config.yaml file.
Understand runtime, runspersub, and -n parameters
With the correct use of runtime, runspersub and -n parameters, you can have full control of your run.
runtimedefines the run length.runspersubdefines the maximum number of runs for every PBS job submission.-nsets the number of runs to be performed.
Now some practical examples:
-
Run 20 years of simulation with resubmission every 5 years
To have a total experiment length of 20 years with a 5-year resubmission cycle, leaveruntimeas the default value of1 yearand setrunspersubto5. Then, run the configuration with-nset to20:This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs.payu run-f -n 20 -
Run 7 years of simulation with resubmission every 3 years
To have a total experiment length of 7 years with a 3-year resubmission cycle, leaveruntimeas the default value of1 yearand setrunspersubto3. Then, run the configuration with-nset to7:This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs.payu run -f -n 7 -
Run 3 months and 10 days of simulation in a single submission
To have a total experiment length of 3 months and 10 days in a single submission, set theruntimeas follows:Setyears: 0 months: 3 days: 10runspersubto1(or any value > 1) and run the configuration without-noption (or with-nset to1):payu run -f -
Run 1 year and 4 months of simulation with resubmission every 4 months
To have a total experiment length of 1 year and 4 months (16 months), you need to split it into multiple runs. For example, 4 runs of 4 months each. In this case, set theruntimeas follows:Since the run length is set to 4 months, setyears: 0 months: 4 days: 0runspersubto1to resubmit your jobs every 4 months (i.e. every run). Then, run the configuration with-nset to4:payu run -f -n 4
Monitor ACCESS-ESM runs
You can execute the following command to show the status of all your submitted PBS jobs:
qstat -u $USER
If you changed the jobname in the PBS resources of the Master Configuration file, that will appear as your job's Name instead of pre-industrial.
S indicates the status of your run, where:
- Q → Job waiting in the queue to start
- R → Job running
- E → Job ending
- H → Job on hold
If there are no jobs listed with your jobname (or if no job is listed), your run either successfully completed or was terminated due to an error.
For more information, check NCI documentation.
Stop a run
If you want to manually terminate a run, you can do so by executing:
qdel <job-ID>
Error and output log files
While the model is running, payu saves the standard output and standard error in the respective access.out and access.err files in the control directory. You can examine the contents of these files to check on the status of a run as it progresses.
When the model completes its run, or if it crashes, the output and error log files are by default renamed as jobname.o<job-ID> and jobname.e<job-ID>, respectively.
Model Live Diagnostics
ACCESS-NRI developed the Model Live Diagnostics framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi.
For a complete documentation on how to use this framework, check the Model Diagnostics documentation.
ACCESS-ESM outputs
At the end of the model run, output files (and restart files) are moved from the work directory into the archive directory under /scratch/$PROJECT/$USER/access-esm/archive/access-esm, where they are further subdivided for each run. They are also symlinked in the control directory to ~/access-esm/archive.
The naming format for a typical output folder is outputXXX and for a restart folder restartXXX, where XXX is the run number starting from 000.
Tip
A run with a different ACCESS-ESM configuration (different git branch) counts as a new run.
Thus, if output folders already exist, the internal number of the new output folder will be set to the first available XXX number.
Outputs and restarts are separated in the respective folders for each model component.
For the atmospheric output data, the files are usually UM fieldsfile, formatted as <UM-suite-identifier>a.p<output-stream-identifier><time-identifier>.