Before running ACCESS-OM, you need to fulfil general prerequisites outlined in the First Steps section.
Join the hh5, qv56, ua8 and ik11 projects at NCI
To join these projects, request membership on the respective hh5, qv56, ua8 and ik11 NCI project pages.
For more information on how to join specific NCI projects, please refer to How to connect to a project.
Payu on Gadi is available through the
conda/analysis3environment in the hh5 project.
After obtaining hh5 project membership, load the
conda/analysis3environment to automatically retrieve payu as follows:
To check that payu is available, run:
module use /g/data/hh5/public/modules module load conda/analysis3
payu --version 1.0.19
Get ACCESS-OM configuration
A standard ACCESS-OM configuration is available on the COSIMA GitHub.
This is a 1° horizontal resolution configuration with interannual forcing from 1 Jan 1958 to 31 Dec 2018.
To get it on Gadi, create a directory to store the model configuration.Navigate to this directory and clone the GitHub repo in it by running:
git clone https://github.com/COSIMA/1deg_jra55_iaf.git
gitcommands (e.g., matlab/R2018a). If you have trouble cloning the repository, run the following command before trying again:
Edit ACCESS-OM configuration
It is good practice to create a new git branch to store all your modifications for a particular run, so as not to modify the reference configuration.
To create a local branch called "example_run", from within the cloned repo execute:
git checkout -b example_run
Payu is a workflow management tool for running numerical models in supercomputing environments.
The general layout of a payu-supported model run consists of two main directories:
The laboratory directory, where all the model components reside. For ACCESS-OM, it is typically
The control directory, where the model configuration resides and from where the model is run (in this example, the cloned directory
This distinction of directories separates the small-size configuration files from the larger binary outputs and inputs. In this way, the configuration files can be placed in the
$HOME directory (as it is the only filesystem actively backed-up on Gadi), without overloading it with too much data.
Furthermore, this separation allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously.
To setup the laboratory directory, run the following command from the control directory:
This creates the laboratory directory, together with relevant subdirectories, depending on the configuration. The main subdirectories of interest are:
work→ a temporary directory where the model is run. It gets cleaned after each run.
archive→ the directory where output is stored after each run.
Edit the Master Configuration file
config.yaml file located in the control directory, is the Master Configuration file.
This file, which controls the general model configuration, contains several parts:
These lines can be edited to change the PBS directives for the PBS job.
queue: normal walltime: 3:00:00 jobname: 1deg_jra55_iaf mem: 1000GB
For example, to run ACCESS-OM under the
tm70project (ACCESS-NRI), add the following line:
project: tm70To run ACCESS-OM, you need to be a member of a project with allocated Service Units (SU). For more information, check how to join relevant NCI projects.
These lines let payu know which driver to use for the main model configuration (access-om).
name: common model: access-om2 input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55
namefield here is not actually used for the configuration run so you can safely ignore it.
ACCESS-OM is a coupled model deploying multiple submodels (i.e. model components). This section specifies the submodels and configuration options required to execute the model correctly.
submodels: - name: atmosphere model: yatm exe: /g/data/access/payu/access-om2/bin/coe/um7.3x input: - /g/data/ik11/inputs/access-om2/input_20201102/yatm_1deg - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hr/rsds/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hr/rlds/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hr/prra/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hr/prsn/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hrPt/psl/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/land/day/friver/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hrPt/tas/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hrPt/huss/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hrPt/uas/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/atmos/3hrPt/vas/gr/v20190429 - /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0/landIce/day/licalvf/gr/v20190429 ncpus: 1
- name: ocean model: mom exe: /g/data/ik11/inputs/access-om2/bin/fms_ACCESS-OM_730f0bf_libaccessom2_d750b4b.x input: /g/data/ik11/inputs/access-om2/input_20201102/mom_1deg ncpus: 216
- name: ice model: cice exe: /g/data/ik11/inputs/access-om2/bin/cice_auscom_360x300_24p_edcfa6f_libaccessom2_d750b4b.exe input: /g/data/ik11/inputs/access-om2/input_20201102/cice_1deg ncpus: 24
Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the control directory, whose name matches the submodel's name (e.g., configuration options for the
atmospheresubmodel are in the
collate: restart: true walltime: 1:00:00 mem: 30GB ncpus: 4 queue: normal exe: /g/data/ik11/inputs/access-om2/bin/mppnccombine
collateprocess combines a number of smaller files, which contain different parts of the model grid, into target output files. Restart files are typically tiled in the same way and will also be combined together if the
restartoption is set to
When running a new configuration, payu automatically commits changes in git if
runlogis set to
Should not be changed to avoid losing track of the current experiment.
stacksizeis the maximum size (in KiB) of the per-thread resources allocated for each process. This is often set to
unlimitedas explicit stacksize values may not be correctly communicated across Gadi nodes.
The restart frequency specifies the rate of saved restart files.
For example, to save restart files every fifth run (i.e. restart004, restart009, restart014, etc.), you need to set
Intermediate restarts are still temporarily saved and deleted only after a permanently archived restart has been produced.
Line to append mpirun arguments to the
mpirun: --mca io ompio --mca io_ompio_num_aggregators 1
mpiruncall of the model.
This line is the configuration marker for any additional qsub flags.
qsub_flags: -W umask=027
Line to add the specified variables to the run environment.
env: UCX_LOG_LEVEL: 'error'
Lines to control the platform-specific default parameters.
platform: nodesize: 48
nodesize: 48sets the default number of cpus per node to 48, to fully utilise nodes regardless of the requested number of cpus.
A namelist to include separate user scripts or subcommands at various stages of a payu submission.
userscripts: error: resub.sh run: rm -f resubmit.count
errorgets called if the model does not run correctly and returns an error code;
rungets called after the model execution, but prior to model output archive.
To find out more about other configuration settings for the
config.yaml file, check out how to configure your experiment with payu.
Change run length
To change the internal run length, edit the
restart_period field in the
&date_manager_nml section of the
&date_manager_nml forcing_start_date = '1958-01-01T00:00:00' forcing_end_date = '2019-01-01T00:00:00'
! Runtime for a single segment/job/submit, format is years, months, seconds, ! two of which must be zero. restart_period = 5, 0, 0 &end
restart_period) can be different from the total run length. Also, while
restart_periodcan be reduced, it should not be increased to more than 5 years to avoid errors. For more information about the difference between internal run and total run lengths, or how to run the model for more than 5 years, refer to the section Run configuration for multiple years.
Run ACCESS-OM configuration
Payu setup (optional)
As a first step, from within the control directory, it is good practice to run:
This will prepare the model run, based on the experiment configuration.
To run ACCESS-OM configuration for one internal run length (controlled by
restart_period in the
~/access-om/1deg_jra55_iaf/accessom2.nml file), execute:
payu run -f
This will submit a single job to the queue with a total run length of
-foption ensures that payu will run even if there is an existing non-empty work directory created from a previous failed run.
Run configuration for multiple years
If you want to run ACCESS-OM configuration for multiple internal run lengths (controlled by
restart_period in the
~/access-om/1deg_jra55_iaf/accessom2.nml file), use the option
payu run -f -n <number-of-runs>
This will run the configuration
number-of-runs times with a total run length of
restart_period * number-of-runs.
For example, to run the configuration for a total of 50 years with
restart_period = 5, 0, 0 (5 years), the
number-of-runs should be set to
payu run -f -n 10
Monitor ACCESS-OM runs
Currently, there is no specific tool to monitor ACCESS-OM runs.
You can execute the following command to show the status of all your submitted PBS jobs:
qstat -u $USER
jobname in the PBS resources of the Master Configuration file, that will appear as your job's Name instead of
S indicates the status of your run, where:
- Q → Job waiting in the queue to start
- R → Job running
- E → Job ending
If there are no jobs listed with your
jobname (or if no job is listed), your run either successfully completed or was terminated due to an error.
Stop a run
If you want to manually terminate a run, you can do so by executing:
Error and output log files
While the model is running, payu saves the standard output and standard error in the respective
access-om2.err files in the control directory. You can examine the contents of these files to check on the status of a run as it progresses.
When the model completes its run, or if it crashes, the output and error log files are by default renamed as
Model Live Diagnostics
ACCESS-NRI developed the Model Live Diagnostics framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi.
For a complete documentation on how to use this framework, check the Model Diagnostics documentation.
While the configuration is running, output files (and restart files) are moved from the
work directory to the
/scratch/$PROJECT/$USER/access-om2/archive. They are also symlinked in the control directory to
Both outputs and restarts are stored in subfolders for each different configuration (in this case,
1deg_jra55_iaf). Inside the configuration folder, they are further subdivided for each internal run.
The naming format for a typical output folder is
outputXXX and for a restart folder
restartXXX, where XXX is the internal run number starting from
Outputs and restarts are separated in the respective folders for each model component.
- Kiss et al. (2020)