Release
This is a Beta Release.
It will be fit to use and test. Known issues are listed in the release notes, but there may be some issues that have not yet been discovered and fixed. The suite and these instructions may change during the beta testing period as improvements are made. If that happens, updates will be included in the release notes.
Run ACCESS-rAM3
About
ACCESS-rAM3 is an ACCESS-NRI-supported configuration of the UK Met Office (UKMO) Regional Nesting Suite for high-resolution regional atmosphere modelling.
A description of the model and its components is available in the ACCESS-rAM3 overview.
ACCESS-rAM3 comprises two suites: a Regional Ancillary Suite (RAS), which generates ancillary files (i.e., input files), and a Regional Nesting Suite (RNS) which runs the regional forecast.
The instructions below outline how to run ACCESS-rAM3 using ACCESS-NRI's supported configuration, specifically designed to run on the National Computating Infrastructure (NCI) supercomputer Gadi.
The example experiment within this page focuses on a flood event in Lismore, NSW, using ERA5-Land
land-surface initial conditions. Its configuration is specified in Nesting configuration.
If you are unsure whether ACCESS-rAM3 is the right choice for your experiment, take a look at the overview of ACCESS Models.
All ACCESS-rAM3 configurations are and available on MOSRS via links at the top of this page.
ACCESS-rAM3 release notes are available on the ACCESS-Hive Forum and are updated when new releases are made available.
Prerequisites
-
NCI Account
Before running ACCESS-rAM3, you need to Set Up your NCI Account. -
MOSRS account
The Met Office Science Repository Service (MOSRS) is a server run by the UK Met Office (UKMO) to support collaborative development with other partners organisations. MOSRS contains the source code and configurations for some model components in ACCESS-rAM3 (e.g., the UM).
To apply for a MOSRS account, please contact your local institutional sponsor. -
Join NCI projects
Join the following projects by requesting membership on their respective NCI project pages:Tip
To request membership for the ki32_mosrs subproject, you need to:
- already be member of the ki32 project
- have a MOSRS account
For more information on joining specific NCI projects, refer to How to connect to a project.
-
Connection to an ARE VDI Desktop (optional)
To run ACCESS-rAM3, start an Australian Research Environment (ARE) VDI Desktop session.
If you are not familiar with ARE, check out the Getting Started on ARE section.
Warning
The waiting time to complete some of the above prerequisites may be 2-3 weeks.
Quick Start guide
The following Quick Start guide is aimed at experienced users wanting to run ACCESS-rAM3. For more detailed instructions, please refer to the Detailed guide.
Required setup for running ACCESS-rAM3
-
Start a new persistent session
In a Gadi login node or from an ARE terminal instance run:This will use your default project.persistent-sessions start <name>
For further instructions on starting a persistent session, refer to the Detailed guide.
-
Assign the persistent session to Rose/Cylc workflows
Run the following command:substitutingecho "<name>.${USER}.<project>.ps.gadi.nci.org.au" > ~/.persistent-sessions/cylc-session
<name>
with the name given to your persistent session, and<project>
with the project assigned to it.Tip
This step should only be done once
For further instructions on assigning the target persistent session, refer to the Detailed guide.
-
Rose/Cylc setup
To get the required Rose/Cylc setup, run:module use /g/data/hr22/modulefiles module load cylc7
For further instructions on getting the Rose/Cylc setup, refer to the Detailed guide.
-
MOSRS authentication
Authenticate using your MOSRS credentials:mosrs-auth
For further instructions on MOSRS authentication, refer to the Detailed guide.
Regional Ancillary Suite (RAS)
-
Copy the RAS from UKMO
rosie checkout u-dg767
For further instructions on getting the RAS configuration, refer to the Detailed guide.
-
Run the RAS
cd ~/roses/u-dg767 rose suite-run
For further instructions on running the RAS configuration, refer to the Detailed guide.
Regional Nesting Suite (RNS)
-
Copy the RNS from UKMO
rosie checkout u-dg768
For further instructions on getting the RNS configuration, refer to the Detailed guide.
-
Run the RNS
From within the RNS directory:rose suite-run
For further instructions on getting the RNS configuration, refer to the Detailed guide.
Detailed guide
Set up an ARE VDI Desktop (optional)
Info
If you want to skip this step and run ACCESS-rAM3 from Gadi login node instead, refer directly to the instructions on how to Set up persistent session.
Launch ARE VDI Session
Go to the ARE VDI page and launch a session with the following entries:
-
Walltime (hours) →
5
This is the amount of time the ARE VDI session will stay active for.
ACCESS-rAM3 does not run directly on ARE.
This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. -
Queue →
normalbw
-
Compute Size →
tiny
(1 CPU)
As mentioned above, the ARE VDI session is only needed for setup and startup tasks, which can be easily accomplished with 1 CPU. -
Project → a project of which you are a member.
The project must have allocated Service Units (SU) to run your simulation. Usually, but not always, this corresponds to your$PROJECT
.
For more information, refer to Join relevant NCI projects. -
Storage →
gdata/access+gdata/hr22+gdata/ki32+gdata/rt52+gdata/zz93+gdata/vk83
(minimum)
This is a list of all project data storage, joined by plus (+
) signs, needed for the ACCESS-rAM3 simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance.
Every ACCESS-rAM3 simulation can be unique and input data can originate from various sources. Hence, if your simulation requires data stored in project folders other than the ones listed in the minimum storage above, you need to add those projects to the storage path.
For example, if your ACCESS-rAM3 simulation requires data stored in/g/data/gx60
and/scratch/gx60
, the following should be added to the minimum storage above:+gdata/gx60+scratch/gx60
Launch the ARE session and, once it starts, click on Launch VDI Desktop.
Open the terminal in the VDI Desktop
Once the new tab opens, you will see a Desktop with a few folders on the left.
To open the terminal, click on the black terminal icon at the top of the window. You should now be connected to a Gadi computing node.
Set up persistent session
To support the use of long-running processes, such as ACCESS model runs, NCI provides a service on Gadi called persistent sessions.
To run ACCESS-rAM3, you need to start a persistent session and set it as the target session for the model run.
Start a new persistent session
To start a new persistent session, using either a Gadi login node or an ARE terminal instance, run the following command:
persistent-sessions start <name>
This will start a persistent session with the given name
that runs under your default project.
If you want to assign a different project to the persistent session, use the option -p
:
persistent-sessions start -p <project> <name>
Tip
While the project assigned to a persistent session does not have to be the same as the project used to run the ACCESS-rAM3 configuration, it does need to have allocated Service Units (SU).
For more information, check how to Join relevant NCI projects.
To list all active persistent sessions run:
persistent-sessions list
The label of a newly-created persistent session has the following format:
<name>.<$USER>.<project>.ps.gadi.nci.org.au
.
Specify target persistent session
After starting the persistent session, it is essential to assign it to the ACCESS-rAM3 run.
The easiest way to create a file ~/.persistent-sessions/cylc-session
that contains the target of the persistent session.
You can do it manually, or by running the following command (by substituting <name>
with the name given to the persistent session, and <project>
with the project assigned to it):
echo "<name>.${USER}.<project>.ps.gadi.nci.org.au" > ~/.persistent-sessions/cylc-session
For example, if the user abc123
started a persistent session named cylc
under the project xy00
, the command will be:
For more information on how to specify the target session, refer to Specify Target Session with Cylc7 Suites.
Tip
You can simultaneously submit multiple ACCESS-rAM3 runs using the same persistent session without needing to start a new one. Hence, the process of specifying the target persistent session for ACCESS-rAM3 should only be done once. Then, to run ACCESS-rAM3, you just need to ensure that you have an active persistent session named like the target one you specified above. If the persistent session is not active, simply start one.
Terminate a persistent session
Tip
Logging out of a Gadi login node or an ARE terminal instance will not affect your persistent session.
To stop a persistent session, run:
persistent-sessions kill <persistent-session-uuid>
Warning
When you terminate a persistent session, any model running on that session will stop. Therefore, you should check whether you have any active model runs before terminating a persistent session.
Rose/Cylc/MOSRS setup
To run ACCESS-rAM3, access to multiple software and MOSRS authentication is needed.
Cylc setup
Cylc (pronounced ‘silk’) is a workflow manager that automatically executes tasks according to the model's main cycle script suite.rc
. Cylc controls how the job will be run and manages the time steps of each model component. It also monitors all tasks, reporting any errors that may occur.
To get the Cylc setup required to run ACCESS-rAM3, execute the following commands:
module use /g/data/hr22/modulefiles
module load cylc7
Warning
Cylc version >= cylc7/24.03
required.
Also, before loading the Cylc module, make sure to have started a persistent session and have assigned it to the ACCESS-rAM3 workflow. For more information about these steps, refer to Set up persistent session.
Rose setup
Rose is a toolkit that can be used to view, edit, or run an ACCESS modelling suite.
By completing the Cylc setup, also Rose will be automatically available. Hence, no additional step is required.
MOSRS authentication
To authenticate using your MOSRS credentials, run:
mosrs-auth
Warning
This step needs to be done once for each new session (e.g., Gadi login, ARE terminal window)
ACCESS-rAM3 configuration
ACCESS-rAM3 comprises 2 different suites: a Regional Ancillary Suite (RAS) and a Regional Nesting Suite (RNS).
Both suites within ACCESS-rAM3 have a suite-ID
in the format u-<suite-name>
, where <suite-name>
is a unique identifier.
Typically, an existing suite is copied and then edited as needed for a particular experiment.
For more information on ACCESS-rAM3, refer to the ACCESS-rAM3 configuration page.
Info
Many of the following steps appear in both the RAS and RNS. For this reason, these steps will be detailed only within the RAS section below, and subsequenltly linked to within the RNS section.
Regional Ancillary Suite (RAS)
For the domain of interest, the RAS generates a set of ancillary files, such as initial conditions. These ancillary files are then used by the RNS.
The suite-ID
of the RAS is u-dg767
.
Get the RAS configuration
Rosie is an SVN repository wrapper with a set of options specific for ACCESS modelling suites. It is automatically available within the Rose setup.
The RAS configuration can be copied from the MOSRS repository in 2 ways:
Suites are, by default, created in the user's Gadi home directory under ~/roses/<suite-ID>
.
This path will be referred to as the suite directory.
The suite directory contains multiple subdirectories and files, including:
app
→ directory containing the configuration files for various tasks within the suite.meta
→ directory containing the GUI metadata.rose-suite.conf
→ main suite configuration file.rose-suite.info
→ suite information file.suite.rc
→ Cylc control script file (Jinja2 language).
Local-only copy
To create a local copy of the RAS from MOSRS repository, run:
rosie checkout u-dg767
Remote and local copy
To create a new copy of the RAS both locally and remotely in the MOSRS repository, run:
rosie copy u-dg767
<suite-ID>
folder is generated within the MOSRS repository and populated with descriptive information about the suite and its initial configuration.
For additional rosie
options, run:
rosie help
Run the RAS
ACCESS-rAM3 suites run on Gadi through a PBS job submission.
When a suite runs, its configuration files are copied in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>
. A symbolic link to this directory is also created in the $USER
's home directory under ~/cylc-run/<suite-ID>
.
ACCESS-rAM3 suites comprise several tasks, such as checking out code repositories, compiling and building the different model components, running the model, etc. The workflow of these tasks is controlled by Cylc.
To run the RAS, execute the following command from within your RAS suite directory:
rose suite-run
After the initial tasks are executed, the Cylc GUI will open, where it is possible to view and control the different tasks in the suite as they are run.
Tip
The Cylc GUI can be safely closed without impacting the experiment run.
To open it again, run the following command from within the suite directory:
rose suite-gcontrol
All steps are completed!!
You will be able to check the suite output files after the run successfully completes.
If you get errors or you can't find the outputs, check the suite logs for debugging.
Check suite logs
It is not unusual that new users experience errors and job failures.
When a suite task fails, a red icon appears next to the respective task name in the Cylc GUI.
To investigate the cause of a failure, or to monitor the progress of a suite, it is helpful to look at the suite's log files.
These files can be found in the directory ~/cylc-run/<suite-ID>
within a folder named log.<TIMESTAMP>
, which is also symlinked as log
(referred to as logs folder below). Logs from previous runs of the same suite are archived as compressed files with the naming pattern log.<TIMESTAMP>.tar.gz
.
Inside the logs folder, various files and subfolders can be found. The most relevant logs are typically:
Suite execution log
The primary suite execution log resides in ~/cylc-run/<suite-ID>/log/suite/log
.
This file contains a chronological record of the suite's run history. Each line is a distinct log entry, generally formatted as <TIMESTAMP> <LOG-TYPE> - [<task-name>.<cylc-cycle-point>] <status>
.
Example of a suite execution log file (click to see content)
2025-03-14T04:11:56Z INFO - Suite server: url=https://cylc.$USER$.$PROJECT.ps.gadi.nci.org.au:PORT/ pid=PID
2025-03-14T04:11:56Z INFO - Run: (re)start=0 log=1
2025-03-14T04:11:56Z INFO - Cylc version: 7.9.9
2025-03-14T04:11:56Z INFO - Run mode: live
2025-03-14T04:11:56Z INFO - Initial point: 01000101T0000Z
2025-03-14T04:11:56Z INFO - Final point: 01000328T2359Z
2025-03-14T04:11:56Z INFO - Cold Start 01000101T0000Z
2025-03-14T04:11:56Z INFO - [make_drivers.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [install_cold.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [make_cice.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [make_mom.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [install_ancil.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [fcm_make_um.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:58Z INFO - [fcm_make_um.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [fcm_make_um.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [install_ancil.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [install_ancil.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [install_cold.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [install_cold.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_cice.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_cice.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_drivers.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_drivers.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_mom.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_mom.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:04Z INFO - [make_drivers.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_drivers.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [install_cold.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [install_cold.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [make_cice.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_cice.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [make_mom.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_mom.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [install_ancil.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [install_ancil.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [client-command] get_latest_state dm5220@gadi-login-08.gadi.nci.org.au:cylc-gui c842e8fb-017a-47ec-8706-7fef0af6d5f5
2025-03-14T04:12:06Z INFO - [make_drivers.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:05Z for job(01)
2025-03-14T04:12:09Z INFO - [make_cice.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:07Z for job(01)
2025-03-14T04:12:09Z INFO - [install_ancil.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:07Z for job(01)
2025-03-14T04:12:10Z INFO - [make2_cice.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:12:11Z INFO - [make2_cice.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:12:10Z for job(01)
2025-03-14T04:12:11Z INFO - [make2_cice.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:14Z INFO - [make_mom.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:12Z for job(01)
2025-03-14T04:12:15Z INFO - [make2_mom.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:12:18Z INFO - [make2_mom.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:12:15Z for job(01)
2025-03-14T04:12:18Z INFO - [make2_mom.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:37Z INFO - [install_cold.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:35Z for job(01)
2025-03-14T04:12:48Z INFO - [make2_mom.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:45Z for job(01)
2025-03-14T04:12:48Z INFO - [make2_mom.01000101T0000Z] -health check settings: execution timeout=PT40M, polling intervals=PT31M,PT2M,PT7M,...
2025-03-14T04:12:59Z INFO - [fcm_make_um.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:57Z for job(01)
2025-03-14T04:12:59Z INFO - [fcm_make_um.01000101T0000Z] -health check settings: execution timeout=PT1H10M, polling intervals=PT1H1M,PT2M,PT7M,...
2025-03-14T04:13:00Z INFO - [make2_cice.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:58Z for job(01)
2025-03-14T04:13:00Z INFO - [make2_cice.01000101T0000Z] -health check settings: execution timeout=PT30M, polling intervals=PT21M,PT2M,PT7M,...
2025-03-14T04:15:58Z INFO - [make2_cice.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:15:57Z for job(01)
2025-03-14T04:18:19Z INFO - [make2_mom.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:18:17Z for job(01)
2025-03-14T04:20:10Z INFO - [fcm_make_um.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:20:08Z for job(01)
2025-03-14T04:20:11Z INFO - [fcm_make2_um.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:20:13Z INFO - [fcm_make2_um.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:20:12Z for job(01)
2025-03-14T04:20:13Z INFO - [fcm_make2_um.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:20:37Z INFO - [fcm_make2_um.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:20:35Z for job(01)
2025-03-14T04:20:37Z INFO - [fcm_make2_um.01000101T0000Z] -health check settings: execution timeout=PT1H10M, polling intervals=PT1H1M,PT2M,PT7M,...
2025-03-14T04:37:11Z INFO - [fcm_make2_um.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:37:09Z for job(01)
2025-03-14T04:37:12Z INFO - [recon.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:37:19Z INFO - [recon.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:37:14Z for job(01)
2025-03-14T04:37:19Z INFO - [recon.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:37:53Z INFO - [recon.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:37:52Z for job(01)
2025-03-14T04:37:53Z INFO - [recon.01000101T0000Z] -health check settings: execution timeout=PT30M, polling intervals=PT21M,PT2M,PT7M,...
2025-03-14T04:38:28Z INFO - [recon.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:38:28Z for job(01)
2025-03-14T04:38:29Z INFO - [coupled.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:38:30Z INFO - [coupled.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:38:30Z for job(01)
2025-03-14T04:38:30Z INFO - [coupled.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:42:00Z INFO - [coupled.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:41:59Z for job(01)
2025-03-14T04:42:00Z INFO - [coupled.01000101T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T04:45:28Z INFO - [coupled.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:45:27Z for job(01)
2025-03-14T04:45:29Z INFO - [filemove.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:45:30Z INFO - [filemove.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:45:30Z for job(01)
2025-03-14T04:45:30Z INFO - [filemove.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:12Z INFO - [filemove.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:46:11Z for job(01)
2025-03-14T04:46:12Z INFO - [filemove.01000101T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T04:46:26Z INFO - [filemove.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:46:25Z for job(01)
2025-03-14T04:46:27Z INFO - [history_postprocess.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:46:27Z INFO - [coupled.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:46:28Z INFO - [coupled.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:46:28Z for job(01)
2025-03-14T04:46:28Z INFO - [coupled.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:28Z INFO - [history_postprocess.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:46:28Z for job(01)
2025-03-14T04:46:28Z INFO - [history_postprocess.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:58Z INFO - [history_postprocess.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:46:57Z for job(01)
2025-03-14T04:46:58Z INFO - [history_postprocess.01000101T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T04:47:09Z INFO - [coupled.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:47:09Z for job(01)
2025-03-14T04:47:09Z INFO - [coupled.01000201T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T04:47:11Z INFO - [history_postprocess.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:47:10Z for job(01)
2025-03-14T04:47:12Z INFO - [housekeep.01000101T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T04:47:13Z INFO - [housekeep.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:47:13Z for job(01)
2025-03-14T04:47:13Z INFO - [housekeep.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:47:18Z INFO - [housekeep.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:47:17Z for job(01)
2025-03-14T04:47:18Z INFO - [housekeep.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:47:21Z INFO - [housekeep.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:47:20Z for job(01)
2025-03-14T04:50:13Z INFO - [coupled.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:50:11Z for job(01)
2025-03-14T04:50:14Z INFO - [filemove.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:50:15Z INFO - [filemove.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:50:14Z for job(01)
2025-03-14T04:50:15Z INFO - [filemove.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:50:54Z INFO - [filemove.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:50:52Z for job(01)
2025-03-14T04:50:54Z INFO - [filemove.01000201T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T04:51:08Z INFO - [filemove.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:06Z for job(01)
2025-03-14T04:51:09Z INFO - [history_postprocess.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:51:09Z INFO - [coupled.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:51:10Z INFO - [coupled.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:09Z for job(01)
2025-03-14T04:51:10Z INFO - [coupled.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:10Z INFO - [history_postprocess.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:09Z for job(01)
2025-03-14T04:51:10Z INFO - [history_postprocess.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:23Z INFO - [history_postprocess.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:51:22Z for job(01)
2025-03-14T04:51:23Z INFO - [history_postprocess.01000201T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T04:51:35Z INFO - [history_postprocess.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:34Z for job(01)
2025-03-14T04:51:36Z INFO - [housekeep.01000201T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T04:51:37Z INFO - [housekeep.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:37Z for job(01)
2025-03-14T04:51:37Z INFO - [housekeep.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:41Z INFO - [housekeep.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:51:40Z for job(01)
2025-03-14T04:51:41Z INFO - [housekeep.01000201T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:51:45Z INFO - [housekeep.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:43Z for job(01)
2025-03-14T05:01:00Z INFO - [coupled.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:00:59Z for job(01)
2025-03-14T05:01:00Z INFO - [coupled.01000301T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T05:04:40Z INFO - [coupled.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:04:39Z for job(01)
2025-03-14T05:04:41Z INFO - [filemove.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T05:04:43Z INFO - [filemove.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:04:43Z for job(01)
2025-03-14T05:04:43Z INFO - [filemove.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:05:25Z INFO - [filemove.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:05:24Z for job(01)
2025-03-14T05:05:25Z INFO - [filemove.01000301T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T05:05:40Z INFO - [filemove.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:05:39Z for job(01)
2025-03-14T05:05:41Z INFO - [history_postprocess.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T05:05:42Z INFO - [history_postprocess.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:05:42Z for job(01)
2025-03-14T05:05:42Z INFO - [history_postprocess.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:05:57Z INFO - [history_postprocess.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:05:56Z for job(01)
2025-03-14T05:05:57Z INFO - [history_postprocess.01000301T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T05:06:08Z INFO - [history_postprocess.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:06:07Z for job(01)
2025-03-14T05:06:09Z INFO - [housekeep.01000301T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T05:06:10Z INFO - [housekeep.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:06:10Z for job(01)
2025-03-14T05:06:10Z INFO - [housekeep.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:06:18Z INFO - [housekeep.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:06:16Z for job(01)
2025-03-14T05:06:18Z INFO - [housekeep.01000301T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T05:06:20Z INFO - [housekeep.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:06:19Z for job(01)
2025-03-14T05:06:20Z INFO - Suite shutting down - AUTOMATIC
2025-03-14T05:06:28Z INFO - DONE
This file helps identify specific tasks that failed during the suite run.
Tip
When a task fails, the LOG-TYPE
will typically be ERROR
or CRITICAL
, instead of the more common INFO
.
Once a specific task and Cylc cycle point are identified, the task-specific logs can be inspected.
Task-specific logs
Logs for individual tasks are located in subfolders within the logs folder, following this path structure:
~/cylc-run/<suite-ID>/log/job/<cylc-cycle-point>/<task-name>/<retry-number>
<retry-number>
indicates the number of retries for the same task, with the latest retry symlinked to NN
.
For example, logs for most recent retry of a task named Lismore_d1000_GAL9_um_recon
at Cylc cycle point 20220226T0000Z
can be found in the folder ~/cylc-run/<suite-ID>/log/job/20220226T0000Z/Lismore_d1000_GAL9_um_recon/NN
.
Within this directory, the job.out
and job.err
files (representing STDOUT
and STDERR
, respectively) can be found, along with other related log files.
Tip
Within the Cylc GUI, logs for a specific task can be viewed by right-clicking on the task and selecting the desired log from the View Job Logs (Viewer) menu.
Stop, restart, reload and clean suites
In some cases, you may want to control the running state of a suite.
If your Cylc GUI has been closed and you are unsure whether your suite is still running, you can scan for active suites and reopen the GUI.
To scan for active suites, run:
cylc scan
rose suite-gcontrol
STOP a suite
To shutdown a suite in a safe manner, run the following command from within the suite directory:
rose suite-stop -y
-
Check the status of all your PBS jobs:
qstat -u $USER
-
Delete any job related to your run:
qdel <job-ID>
RESTART a suite
There are two main ways to restart a suite:
-
SOFT restart
To reinstall the suite and reopen Cylc in the same state it was prior to being stopped, run the following command from within the suite directory:rose suite-run --restart
Warning
You may need to manually trigger failed tasks from the Cylc GUI.
-
HARD restart
To overwrite any previous runs of the suite and start afresh, run the following command from within the suite directory:rose suite-run --new
Warning
This will overwrite all existing model output and logs for the same suite.
RELOAD a suite
In some cases, the suite needs to be updated without necessarily having to stop it (e.g., after fixing a typo in a file). Updating an active suite is called a reload, where the suite is re-installed and Cylc is updated with the changes. This is similar to a SOFT restart, except new changes are installed, so you may need to manually trigger failed tasks from the Cylc GUI.
To reload a suite, run the following command from within the suite directory:
rose suite-run --reload
CLEAN a suite
To remove all files and folders created by the suite within the /scratch/$PROJECT/$USER/cylc-run/<suite-ID>
directory, run the following command from within the suite directory:
rose suite-clean
Alternatively, you can achieve the same behaviour within a new submission of an experiment, by appending the --new
option to the rose suite-run
command:
rose suite-run --new
Warning
Cleaning a suite folder will remove any non-archived data (i.e., output files, logs, executables, etc.) associated with the suite.
RAS output files
The RAS output ancillary files can be found in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils
.
Ancillaries are divided into folders according to each nested region name, and then further separated according to each nest (i.e., Resolution) name. The path of ancillaries for a specific nest (i.e., Resolution) is /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils/<nested_region_name>/<nest_name>
.
The example above has one nested_region_name
called Lismore
, 1 nest named era5
(outer domain corresponding to Resolution 1), and 2 inner nests (Resolution 2 and Resolution 3) named d1000
and d0198
, respectively.
Thus, the ancillary files directory /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils/
contains the following subdirectories:
Lismore/d1000
Lismore/d0198
Lismore/era5
Ancillary data files are typically output in the UM fieldsfile format.
Regional Nesting Suite (RNS)
The RNS uses the ancillary files produced by the RAS to run the regional forecast for the domain of interest.
The suite-ID
of the RNS is u-dg768
.
Get and run RNS configuration
Steps to obtain and run the RNS, as well as monitor logs, are similar to those listed above for the RAS.
The main difference is the suite-ID
, which for the RNS is u-dg768
.
To get the RNS configuration, follow the steps listed in Get the RAS configuration, making sure you use the correct RNS suite-ID
u-dg768
when copying the suite.
To run the RNS configuration, follow the steps listed in Run the suite.
To check the RNS suite logs, follow the steps listed in Check suite logs.
RNS output files
All the RNS output files are available in the directory /scratch/$PROJECT/$USER/cylc-run/<suite-ID>
. They are also symlinked in ~/cylc-run/<suite-ID>
.
The RNS output data can be found in the directory /scratch/$PROJECT/$USER/cycl-run/<suite-ID>/share/cycle
, grouped for each cycle.
Within the cycle
directory, outputs are divided into multiple nested subdirectories in the format <nested_region_name>/<science_configuration>/<nest_name>
, with <nested_region_name>
and <nest_name>
referring to the respective configurable options. The <science_configuration>
is usually GAL9
or RAL3.2
, depending on the nest resolution.
Each <nest_name>
directory has the following subdirectories:
ics
→ initial conditionslbcs
→ lateral boundary conditionsum
→ model output data
For example, the model output data for the first cycle (20220226T0000Z
) of the Lismore experiment (Lismore
nested_region_name
, using a RAL3P2
science_configuration
and d0198
as a nest_name
) can be found in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/cycle/20220226T0000Z/Lismore/d0198/RAL3P2/um
.
The RNS output data files are typically in the UM fieldsfile format.
Tip
To change which output variables are produced, refer to Change the output variables
Edit ACCESS-rAM3 configuration
Tip
Due to the presence of two distinct suites (RAS and RNS), specific modifications might be required in either one or both.
For each of the following possible modifications, the relevant suite that needs editing is listed as a bullet point.
This section describes how to modify the ACCESS-rAM3 configuration.
The modifications discussed in this section can change the way the RAS and RNS are run, or how specific model components are configured and coupled together.
In general, ACCESS modelling suites can be edited either by directly modifying the configuration files within the suite directory, or by using the Rose GUI.
Warning
Unless you are an experienced user, directly modifying configuration files is usually discouraged to avoid encountering errors.
Rose GUI
To open the Rose GUI, run the following command from within the suite directory:
rose edit &
Tip
The &
is optional. It allows the terminal prompt to remain active while running the Rose
GUI as a separate process in the background.
Change run length
-
RNS
The RNS runs in multiple PBS jobs submissions, each one constituting a cycle. The job scheduler automatically resubmits the suite every chosen cycling frequency until the total run length is reached.Warning
The cycling frequency is currently set to
24
hours (1 day) and should be left unchanged to avoid errors.
This also means the model will run for a minimum of 1 day.The run length is calculated using the
INITIAL_CYCLE_POINT
andFINAL_CYCLE_POINT
fields.
Both these fields use ISO 8601 date format, withFINAL_CYCLE_POINT
also accepting relative ISO 8601 Durations.Warning
INITIAL_CYCLE_POINT
andFINAL_CYCLE_POINT
define all the Cylc cycle points that are set within the experiment run.
The model will always run for a full cycling frequency (1 day) for each Cylc cycle point.
This means, for example, that withINITIAL_CYCLE_POINT
set to20220226T0000Z
, andFINAL_CYCLE_POINT
set to+P1D
(plus 1 day), 2 Cylc cycle points will be set (20220226T0000Z
and20220227T0000Z
). Therefore, the model will run for a total of 2 days!
To avoid running the model for longer that desired, we suggest adding-PT1S
(minus 1 second) to the relative duration specified in theFINAL_CYCLE_POINT
(refer to the example below).To modify these parameters within the Rose GUI, navigate to suite conf → Nesting Suite → Cycling options. Edit the related field and click the Save button
.
For example, to run the experiment for 2 days starting on the 5th April 2000, setINITIAL_CYCLE_POINT
to20000405T0000Z
andFINAL_CYCLE_POINT
to+P2D-PT1S
(due to the run length mismatch).
Change the land-surface initial conditions source
-
RNS
To change the land-surface initial conditions source within the Rose GUI, navigate to suite conf → Nesting Suite → Driving model setup. Edit theNCI_HRES_ECCB
field and click the Save button.
For example, to get the land-surface initial conditions from the
BARRA-R2
dataset, set theNCI_HRES_ECCB
field toBARRA2-R
.
Warning
When changing the land-surface initial conditions source, it is important to ensure that the configuration of the nested region aligns with the nest configuration requirements.
Change the simulation region
In ACCESS-rAM3, users can perform simulations for a particular region of the Earth by configuring specific parameters for each domain of interest (referred to as nested region).
In ACCESS-rAM3, the following parameters are supported to configure the nested regions:
Warning
Domain-specific changes need to be consistent between RAS and RNS. Therefore, for each of the configuration parameters listed above, consistent changes to both RAS and RNS will be required.
Change the nested region name
-
RAS
To change a nested region name within the Rose GUI, navigate to suite conf → Regional Ancillary Suite → Nested region 1 setup. Edit therg01_name
field and click the Save button.
For example, to set the name of the nested region to
Darwin
, set therg01_name
field toDarwin
. -
RNS
Changing the RAS nested region name changes the RAS output path. As a consequence, the following changes are required within the RNS:-
Ancillary directory
To change the first nest ancillary directory within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup → Resolution 1 setup. Change therg01_rs01_ancil_dir
field by replacingLismore
with the chosen RAS nested region name, and click the Save button.
The same step needs to be repeated for:- suite conf → Nesting Suite → Nested region 1 setup → Resolution 2 setup
rg01_rs02_ancil_dir
field - suite conf → Nesting Suite → Driving model setup
dm_ec_lam_ancil_dir
field
For example, if the RAS nested region name was set to
Darwin
, replaceLismore
in therg01_rs01_ancil_dir
,rg01_rs02_ancil_dir
anddm_ec_lam_ancil_dir
fields withDarwin
. - suite conf → Nesting Suite → Nested region 1 setup → Resolution 2 setup
-
RNS nested region name
To change the nested region name within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup. Edit therg01_name
field and click the Save button.
For example, to set the name of the nested region to
Darwin
, set therg01_name
field toDarwin
.Tip
Changing the RNS nested region name is not strictly necessary, but it affects the RNS outputs path. Therefore, for consistency, it is strongly recommended for RAS and RNS to have the same nested region names.
-
Change the nested region position
The nested region position is usually defined by the latitude and longitude coordinates of the nested region centre.
-
RAS
To change the nested region centre within the Rose GUI, navigate to suite conf → Regional Ancillary Suite → Nested region 1 setup. Edit therg01_centre
field and click the Save button.
For example, to set the centre of the nested region to
-12.4
/130.8
, set therg01_centre
field to-12.4
/130.8
.
Warning
When changing the land-surface initial conditions source, it is important to ensure that the configuration of the nested region aligns with the nest configuration requirements.
Change the nested region's nest configuration
Each nested region can contain multiple nests (referred to as Resolutions within the RAS and RNS), each of them being a separate domain where the simulation experiment is carried out.
Typically, nests within the same nested region are arranged concentrically, with increasingly smaller dimensions and higher resolutions towards the innermost nests.
Warning
Currently, ACCESS-rAM3 only supports specific nest configurations that meet the following criteria:
The grid points of the RAS first inner nest (i.e., Resolution 2, because Resolution 1 always corresponds to the outer ERA5 domain) must align with those of the land-surface initial conditions dataset. Thus, the configuration of the RAS first inner nest (Resolution 2), including its position, dimension and resolution, need to be modified accordingly. Note that the position of a nest is also influenced by the nested region position.
Change the output variables
UM outputs are usually provided as a list of STASH variables.
Manually specifying each STASH variable can be complex. To simplify the selection process for commonly used climate analysis variables, predefined groups of STASH variables are set up, known as stashpacks.
- RNS
To toggle a stashpack within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup → Resolution 1 setup → Config 1 setup. Toggle a specific stashpack within therg01_rs01_m01_stashpack
field and click the Save button.
Similar steps can be repeated for the suite conf → Nesting Suite → Nested region 1 setup → Resolution 2 setup → Config 2 setuprg01_rs02_m02_stashpack
field.
For example, to enablestashpack 6
(that includes variables such as wind gust, mean sea level pressure and rainfall amount, for every model timestep) in all nests, set the6th
button of bothrg01_rs01_m01_stashpack
andrg01_rs02_m02_stashpack
fields totrue
.
Get Help
If you have questions or need help regarding ACCESS-rAM3, consider creating a topic in the Regional Nesting Suite category of the ACCESS-Hive Forum.
For assistance on how to request help from ACCESS-NRI, follow the guidelines on how to get help.