Before running ACCESS-ESM, you need to fulfil general prerequisites outlined in the First Steps section.
Join the access and hh5 projects at NCI
To join these projects, request membership on the respective access and hh5 NCI project pages.
For more information on joining specific NCI projects, refer to How to connect to a project.
Payu on Gadi is available through the
conda/analysis3environment in the hh5 project.
After obtaining hh5 project membership, load the
conda/analysis3environment to automatically retrieve payu as follows:
To check that payu is available, run:
module use /g/data/hh5/public/modules module load conda/analysis3
payu --version 1.0.19
Get ACCESS-ESM configuration
A pre-industrial configuration of ACCESS-ESM is available on the coecms GitHub.
To get it on Gadi, create a directory to store the model configuration.Navigate to this directory and clone the GitHub repo in it by running:
git clone https://github.com/coecms/esm-pre-industrial.git
gitcommands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again:
Edit ACCESS-ESM configuration
It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration.
To create a local branch called "example_run", from within the cloned repo execute:
git checkout -b example_run
Payu is a workflow management tool for running numerical models in supercomputing environments.
The general layout of a payu-supported model run consists of two main directories:
The laboratory directory, where all the model components reside. For ACCESS-ESM, it is typically
The control directory, where the model configuration resides and from where the model is run (in this example, the cloned directory
This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the
$HOME directory (as it is the only filesystem actively backed-up on Gadi), without overloading it with too much data.
Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously.
To setup the laboratory directory, run the following command from the control directory:
This creates the laboratory directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are:
work→ a temporary directory where the model is run. It gets cleaned after each run.
archive→ the directory where output is stored after each run.
Edit the Master Configuration file
config.yaml file located in the control directory, is the Master Configuration file.
This file, which controls the general model configuration, contains several parts:
These lines can be edited to change the PBS directives for the PBS job.
jobname: pre-industrial queue: normal walltime: 20:00:00
For example, to run ACCESS-ESM under the
tm70project (ACCESS-NRI), add the following line:
project: tm70To run ACCESS-ESM, you need to be a member of a project with allocated Service Units (SU). For more information, check how to join relevant NCI projects.
Link to the laboratory directory
These lines set the laboratory directory path, which is relative to
# note: if laboratory is relative path, it is relative to /scratch/$PROJECT/$USER laboratory: access-esm
/scratch/$PROJECT/$USER. Absolute paths can also be specified.
This line tells payu which driver to use for the main model (
accessrefers to ACCESS-ESM).
ACCESS-ESM is a coupled model deploying multiple submodels (i.e. model components). This section specifies the submodels and configuration options required to execute the model correctly.
submodels: - name: atmosphere model: um ncpus: 192 exe: /g/data/access/payu/access-esm/bin/coe/um7.3x input: - /g/data/access/payu/access-esm/input/pre-industrial/atmosphere - /g/data/access/payu/access-esm/input/pre-industrial/start_dump
- name: ocean model: mom ncpus: 180 exe: /g/data/access/payu/access-esm/bin/coe/mom5xx input: - /g/data/access/payu/access-esm/input/pre-industrial/ocean/common - /g/data/access/payu/access-esm/input/pre-industrial/ocean/pre-industrial
- name: ice model: cice ncpus: 12 exe: /g/data/access/payu/access-esm/bin/coe/cicexx input: - /g/data/access/payu/access-esm/input/pre-industrial/ice
- name: coupler model: oasis ncpus: 0 input: - /g/data/access/payu/access-esm/input/pre-industrial/coupler
Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel's name (e.g., configuration options for the
atmospheresubmodel are in the
collate: exe: /g/data/access/payu/access-esm/bin/mppnccombine restart: true mem: 4GB
collateprocess combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the
restartoption is set to
This is the location of the files used for a warm restart.
Start date and internal run length
This section specifies the start date and internal run length.
calendar: start: year: 101 month: 1 days: 1
runtime: years: 1 months: 0 days: 0The internal run length (controlled by
runtime) can be different from the total run length. Also, while
runtimecan be reduced, it should not be increased to more than 1 year to avoid errors. For more information about the difference between internal run and total run lengths, or how to run the model for more than 1 year, refer to the section Run configuration for multiple years.
Number of runs per PBS submission
ACCESS-ESM configurations are often run in multiple steps (or cycles), with payu running a maximum of
runspersubinternal runs for every PBS job submission.If you increase
runspersub, you may need to increase the walltime in the PBS resources.
To find out more about other configuration settings for the
config.yaml file, check out how to configure your experiment with payu.
Run ACCESS-ESM configuration
Payu setup (optional)
As a first step, from within the control directory, it is good practice to run:
This will prepare the model run, based on the experiment configuration.
To run ACCESS-ESM configuration for one internal run length (controlled by
runtime in the
config.yaml file), execute:
payu run -f
This will submit a single job to the queue with a total run length of
runtime. If there is no previous run, it will start from the
start date indicated in the
config.yaml file. Otherwise, it will perform a warm restart from a previously saved restart file.
-foption ensures that payu will run even if there is an existing non-empty work directory created from a previous failed run.
Run configuration for multiple years
If you want to run ACCESS-ESM configuration for multiple internal run lengths (controlled by
runtime in the
config.yaml file), use the option
payu run -f -n <number-of-runs>
This will run the configuration
number-of-runs times with a total run length of
runtime * number-of-runs. The number of consecutive PBS jobs submitted to the queue depends on the
runspersub value specified in the
With the correct use of
-n parameters, you can have full control of your run.
runtimedefines the internal run length.
runspersubdefines the maximum number of internal runs for every PBS job submission.
-nsets the number of internal runs to be performed.
Now some practical examples:
Run 20 years of simulation with resubmission every 5 years
To have a total run length of 20 years with a 5-year resubmission cycle, leave
runtimeas the default value of
1 yearand set
5. Then, run the configuration with
This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs.
payu run-f -n 20
Run 7 years of simulation with resubmission every 3 years
To have a total run length of 7 years with a 3-year resubmission cycle, leave
runtimeas the default value of
1 yearand set
3. Then, run the configuration with
This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs.
payu run -f -n 7
Run 3 months and 10 days of simulation in a single submission
To have a total run length of 3 months and 10 days in a single submission, set the
years: 0 months: 3 days: 10
1(or any value > 1) and run the configuration without option
payu run -f
Run 1 year and 4 months of simulation with resubmission every 4 months
To have a total run length of 1 year and 4 months (16 months), you need to split it into multiple internal runs. For example, 4 internal runs of 4 months each. In this case, set the
Since the internal run length is set to 4 months, set
years: 0 months: 4 days: 0
1to resubmit your jobs every 4 months (i.e. every internal run). Then, run the configuration with
payu run -f -n 4
Monitor ACCESS-ESM runs
Currently, there is no specific tool to monitor ACCESS-ESM runs.
You can execute the following command to show the status of all your submitted PBS jobs:
qstat -u $USER
jobname in the PBS resources of the Master Configuration file, that will appear as your job's Name instead of
S indicates the status of your run, where:
- Q → Job waiting in the queue to start
- R → Job running
- E → Job ending
If there are no jobs listed with your
jobname (or if no job is listed), your run either successfully completed or was terminated due to an error.
Stop a run
If you want to manually terminate a run, you can do so by executing:
Error and output log files
While the model is running, payu saves the standard output and standard error in the respective
access.err files in the control directory. You can examine the contents of these files to check on the status of a run as it progresses.
When the model completes its run, or if it crashes, the output and error log files are by default renamed as
Model Live Diagnostics
ACCESS-NRI developed the Model Live Diagnostics framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi.
For a complete documentation on how to use this framework, check the Model Diagnostics documentation.
While the configuration is running, output files (and restart files) are moved from the
work directory to the
/scratch/$PROJECT/$USER/access-esm/archive. They are also symlinked in the control directory to
Both outputs and restarts are stored in subfolders for each different configuration (in this case,
esm-pre-industrial). Inside the configuration folder, they are further subdivided for each internal run.
The naming format for a typical output folder is
outputXXX and for a restart folder
restartXXX, where XXX is the internal run number starting from
Outputs and restarts are separated in the respective folders for each model component.
For the atmospheric output data, the files are usually UM fieldsfile, formatted as