Release

This is a Beta Release.
It will be fit to use and test. Known issues are listed in the release notes, but there may be some issues that have not yet been discovered and fixed. The suite and these instructions may change during the beta testing period as improvements are made. If that happens, updates will be included in the release notes.

RAS configuration RNS configuration Release notes

Terminal animations are active. Click to disable them.

In this documentation, the same code is sometimes shown in a code block and also as a terminal animation.
The code blocks show the commands to be run in a terminal. They can be easily copied by clicking on the icon over the right side of the code block.
The terminal animations are produced using animated-terminal.js and provide examples of the output to expect when the commands are run. Sometimes they might slightly differ from the actual outputs.

Run ACCESS-rAM3

About

ACCESS-rAM3 is an ACCESS-NRI-supported configuration of the UK Met Office (UKMO) Regional Nesting Suite for high-resolution regional atmosphere modelling.
A description of the model and its components is available in the ACCESS-rAM3 overview.

ACCESS-rAM3 comprises two suites: a Regional Ancillary Suite (RAS), which generates ancillary files (i.e., input files), and a Regional Nesting Suite (RNS) which runs the regional forecast.

The instructions below outline how to run ACCESS-rAM3 using ACCESS-NRI's supported configuration, specifically designed to run on the National Computating Infrastructure (NCI) supercomputer Gadi.
The example experiment within this page focuses on a flood event in Lismore, NSW, using ERA5-Land land-surface initial conditions. Its configuration is specified in Nesting configuration.

If you are unsure whether ACCESS-rAM3 is the right choice for your experiment, take a look at the overview of ACCESS Models.

All ACCESS-rAM3 configurations are and available on MOSRS via links at the top of this page.

ACCESS-rAM3 release notes are available on the ACCESS-Hive Forum and are updated when new releases are made available.

Prerequisites

• NCI Account
  Before running ACCESS-rAM3, you need to Set Up your NCI Account.

• MOSRS account
  The Met Office Science Repository Service (MOSRS) is a server run by the UK Met Office (UKMO) to support collaborative development with other partners organisations. MOSRS contains the source code and configurations for some model components in ACCESS-rAM3 (e.g., the UM).
  To apply for a MOSRS account, please contact your local institutional sponsor.

• Join NCI projects
  Join the following projects by requesting membership on their respective NCI project pages:

  • access
  • hr22
  • ki32
  • ki32_mosrs
  • rt52
  • zz93
  • vk83

  Tip

  To request membership for the ki32_mosrs subproject, you need to:

  • already be member of the ki32 project
  • have a MOSRS account

  For more information on joining specific NCI projects, refer to How to connect to a project.

• Connection to an ARE VDI Desktop (optional)
  To run ACCESS-rAM3, start an Australian Research Environment (ARE) VDI Desktop session.
  If you are not familiar with ARE, check out the Getting Started on ARE section.

Warning

The waiting time to complete some of the above prerequisites may be 2-3 weeks.

Quick Start guide

The following Quick Start guide is aimed at experienced users wanting to run ACCESS-rAM3. For more detailed instructions, please refer to the Detailed guide.

Required setup for running ACCESS-rAM3

• Start a new persistent session
  In a Gadi login node or from an ARE terminal instance run:

  persistent-sessions start <name>
  

  This will use your default project.

  For further instructions on starting a persistent session, refer to the Detailed guide.

• Assign the persistent session to Rose/Cylc workflows
  Run the following command:

  echo "<name>.${USER}.<project>.ps.gadi.nci.org.au" > ~/.persistent-sessions/cylc-session
  

  substituting <name> with the name given to your persistent session, and <project> with the project assigned to it.

  Tip

  This step should only be done once

  For further instructions on assigning the target persistent session, refer to the Detailed guide.

• Rose/Cylc setup
  To get the required Rose/Cylc setup, run:

  module use /g/data/hr22/modulefiles
  module load cylc7
  

  For further instructions on getting the Rose/Cylc setup, refer to the Detailed guide.

• MOSRS authentication
  Authenticate using your MOSRS credentials:

  mosrs-auth
  

  For further instructions on MOSRS authentication, refer to the Detailed guide.

Regional Ancillary Suite (RAS)

1. Copy the RAS from UKMO
   

   rosie checkout u-dg767
   

   For further instructions on getting the RAS configuration, refer to the Detailed guide.

2. Run the RAS
   

   cd ~/roses/u-dg767
   rose suite-run
   

   For further instructions on running the RAS configuration, refer to the Detailed guide.

Regional Nesting Suite (RNS)

1. Copy the RNS from UKMO
   

   rosie checkout u-dg768
   

   For further instructions on getting the RNS configuration, refer to the Detailed guide.

2. Run the RNS
   From within the RNS directory:

   rose suite-run
   

   For further instructions on getting the RNS configuration, refer to the Detailed guide.

---

Detailed guide

Set up an ARE VDI Desktop (optional)

Info

If you want to skip this step and run ACCESS-rAM3 from Gadi login node instead, refer directly to the instructions on how to Set up persistent session.

Launch ARE VDI Session

Go to the ARE VDI page and launch a session with the following entries:

• Walltime (hours) → 5
  This is the amount of time the ARE VDI session will stay active for.
  ACCESS-rAM3 does not run directly on ARE.
  This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself.

• Queue → normalbw

• Compute Size → tiny (1 CPU)
  As mentioned above, the ARE VDI session is only needed for setup and startup tasks, which can be easily accomplished with 1 CPU.

• Project → a project of which you are a member.
  The project must have allocated Service Units (SU) to run your simulation. Usually, but not always, this corresponds to your $PROJECT.
  For more information, refer to Join relevant NCI projects.

• Storage → gdata/access+gdata/hr22+gdata/ki32+gdata/rt52+gdata/zz93+gdata/vk83 (minimum)
  This is a list of all project data storage, joined by plus (+) signs, needed for the ACCESS-rAM3 simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance.
  Every ACCESS-rAM3 simulation can be unique and input data can originate from various sources. Hence, if your simulation requires data stored in project folders other than the ones listed in the minimum storage above, you need to add those projects to the storage path.
  For example, if your ACCESS-rAM3 simulation requires data stored in /g/data/gx60 and /scratch/gx60, the following should be added to the minimum storage above: +gdata/gx60+scratch/gx60

Launch the ARE session and, once it starts, click on Launch VDI Desktop.

Open the terminal in the VDI Desktop

Once the new tab opens, you will see a Desktop with a few folders on the left.
To open the terminal, click on the black terminal icon at the top of the window. You should now be connected to a Gadi computing node.

Set up persistent session

To support the use of long-running processes, such as ACCESS model runs, NCI provides a service on Gadi called persistent sessions.

To run ACCESS-rAM3, you need to start a persistent session and set it as the target session for the model run.

Start a new persistent session

To start a new persistent session, using either a Gadi login node or an ARE terminal instance, run the following command:

persistent-sessions start <name>

This will start a persistent session with the given name that runs under your default project.
If you want to assign a different project to the persistent session, use the option -p:

persistent-sessions start -p <project> <name>

Tip

While the project assigned to a persistent session does not have to be the same as the project used to run the ACCESS-rAM3 configuration, it does need to have allocated Service Units (SU).
For more information, check how to Join relevant NCI projects.

persistent-sessions start <name>session <persistent-session-uuid> running - connect using ssh <name>.<$USER>.<project>.ps.gadi.nci.org.au

To list all active persistent sessions run:

persistent-sessions list

persistent-sessions list             UUID  PROJECT   ADDRESS    CPUTIME MEMORY<persistent-session-uuid> <project> 10.9.0.62 00:00:05.213 30.5M

The label of a newly-created persistent session has the following format:
<name>.<$USER>.<project>.ps.gadi.nci.org.au.

Specify target persistent session

After starting the persistent session, it is essential to assign it to the ACCESS-rAM3 run.
The easiest way to create a file ~/.persistent-sessions/cylc-session that contains the target of the persistent session.
You can do it manually, or by running the following command (by substituting <name> with the name given to the persistent session, and <project> with the project assigned to it):

echo "<name>.${USER}.<project>.ps.gadi.nci.org.au" > ~/.persistent-sessions/cylc-session

For example, if the user abc123 started a persistent session named cylc under the project xy00, the command will be:

echo "cylc.abc123.xy00.ps.gadi.nci.org.au" > ~/.persistent-sessions/cylc-session cat ~/.persistent-sessions/cylc-sessioncylc.abc123.xy00.ps.gadi.nci.org.au

For more information on how to specify the target session, refer to Specify Target Session with Cylc7 Suites.

Tip

You can simultaneously submit multiple ACCESS-rAM3 runs using the same persistent session without needing to start a new one. Hence, the process of specifying the target persistent session for ACCESS-rAM3 should only be done once. Then, to run ACCESS-rAM3, you just need to ensure that you have an active persistent session named like the target one you specified above. If the persistent session is not active, simply start one.

Terminate a persistent session

Tip

Logging out of a Gadi login node or an ARE terminal instance will not affect your persistent session.

To stop a persistent session, run:

persistent-sessions kill <persistent-session-uuid>

Warning

When you terminate a persistent session, any model running on that session will stop. Therefore, you should check whether you have any active model runs before terminating a persistent session.

Rose/Cylc/MOSRS setup

To run ACCESS-rAM3, access to multiple software and MOSRS authentication is needed.

Cylc setup

Cylc (pronounced ‘silk’) is a workflow manager that automatically executes tasks according to the model's main cycle script suite.rc. Cylc controls how the job will be run and manages the time steps of each model component. It also monitors all tasks, reporting any errors that may occur.

To get the Cylc setup required to run ACCESS-rAM3, execute the following commands:

module use /g/data/hr22/modulefiles
module load cylc7

module use /g/data/hr22/modulefilesmodule load cylc7Using the cylc session <name>.<$USER>.<project>.ps.gadi.nci.org.auLoading cylc7/24.03 Loading requirement: mosrs-setup/2.0.1

Warning

Cylc version >= cylc7/24.03 required.

Also, before loading the Cylc module, make sure to have started a persistent session and have assigned it to the ACCESS-rAM3 workflow. For more information about these steps, refer to Set up persistent session.

Rose setup

Rose is a toolkit that can be used to view, edit, or run an ACCESS modelling suite.

By completing the Cylc setup, also Rose will be automatically available. Hence, no additional step is required.

MOSRS authentication

To authenticate using your MOSRS credentials, run:

mosrs-auth

mosrs-authINFO: You need to enter your MOSRS credentials here so that GPG can cache your password.Please enter the MOSRS password for <MOSRS-username>:Checking your credentials using Subversion. Please wait.INFO: Successfully accessed Subversion with your credentials.INFO: Checking your credentials using rosie. Please wait.INFO: Successfully accessed rosie with your credentials.

Warning

This step needs to be done once for each new session (e.g., Gadi login, ARE terminal window)

ACCESS-rAM3 configuration

ACCESS-rAM3 comprises 2 different suites: a Regional Ancillary Suite (RAS) and a Regional Nesting Suite (RNS).

Both suites within ACCESS-rAM3 have a suite-ID in the format u-<suite-name>, where <suite-name> is a unique identifier.
Typically, an existing suite is copied and then edited as needed for a particular experiment.

For more information on ACCESS-rAM3, refer to the ACCESS-rAM3 configuration page.

Info

Many of the following steps appear in both the RAS and RNS. For this reason, these steps will be detailed only within the RAS section below, and subsequenltly linked to within the RNS section.

Regional Ancillary Suite (RAS)

For the domain of interest, the RAS generates a set of ancillary files, such as initial conditions. These ancillary files are then used by the RNS.

The suite-ID of the RAS is u-dg767.

Get the RAS configuration

Rosie is an SVN repository wrapper with a set of options specific for ACCESS modelling suites. It is automatically available within the Rose setup.

The RAS configuration can be copied from the MOSRS repository in 2 ways:

• Local-only copy
• Remote and local copy

Suites are, by default, created in the user's Gadi home directory under ~/roses/<suite-ID>. This path will be referred to as the suite directory.

The suite directory contains multiple subdirectories and files, including:

• app → directory containing the configuration files for various tasks within the suite.
• meta → directory containing the GUI metadata.
• rose-suite.conf → main suite configuration file.
• rose-suite.info → suite information file.
• suite.rc → Cylc control script file (Jinja2 language).

Local-only copy

To create a local copy of the RAS from MOSRS repository, run:

rosie checkout u-dg767

rosie checkout u-dg767[INFO] create: /home/565/<$USER>/roses[INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/u-dg767 This option is mostly used for testing and examining suites.

Remote and local copy

To create a new copy of the RAS both locally and remotely in the MOSRS repository, run:

rosie copy u-dg767

rosie copy u-dg767Copy "u-dg767/trunk@<trunk-ID>" to "u-?????"? [y or n (default)]y[INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/>[INFO] <new-suite-ID>: copied items from u-dg767/trunk@<trunk-ID>[INFO] u-dg767: local copy created at /home/565/<$USER>/roses/<new-suite-ID> When a new suite is created in this way, a unique <suite-ID> folder is generated within the MOSRS repository and populated with descriptive information about the suite and its initial configuration.

For additional rosie options, run:

rosie help

Run the RAS

ACCESS-rAM3 suites run on Gadi through a PBS job submission.
When a suite runs, its configuration files are copied in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>. A symbolic link to this directory is also created in the $USER's home directory under ~/cylc-run/<suite-ID>.
ACCESS-rAM3 suites comprise several tasks, such as checking out code repositories, compiling and building the different model components, running the model, etc. The workflow of these tasks is controlled by Cylc.

To run the RAS, execute the following command from within your RAS suite directory:

rose suite-run

After the initial tasks are executed, the Cylc GUI will open, where it is possible to view and control the different tasks in the suite as they are run.

Tip

The Cylc GUI can be safely closed without impacting the experiment run.
To open it again, run the following command from within the suite directory:

rose suite-gcontrol

All steps are completed!!
You will be able to check the suite output files after the run successfully completes.
If you get errors or you can't find the outputs, check the suite logs for debugging.

Check suite logs

It is not unusual that new users experience errors and job failures.
When a suite task fails, a red icon appears next to the respective task name in the Cylc GUI.

To investigate the cause of a failure, or to monitor the progress of a suite, it is helpful to look at the suite's log files.

These files can be found in the directory ~/cylc-run/<suite-ID> within a folder named log.<TIMESTAMP>, which is also symlinked as log (referred to as logs folder below). Logs from previous runs of the same suite are archived as compressed files with the naming pattern log.<TIMESTAMP>.tar.gz.

Inside the logs folder, various files and subfolders can be found. The most relevant logs are typically:

• Suite execution log
• Task-specific logs

Suite execution log

The primary suite execution log resides in ~/cylc-run/<suite-ID>/log/suite/log.

This file contains a chronological record of the suite's run history. Each line is a distinct log entry, generally formatted as <TIMESTAMP> <LOG-TYPE> - [<task-name>.<cylc-cycle-point>] <status>.

Example of a suite execution log file (click to see content)

2025-03-14T04:11:56Z INFO - Suite server: url=https://cylc.$USER$.$PROJECT.ps.gadi.nci.org.au:PORT/ pid=PID
2025-03-14T04:11:56Z INFO - Run: (re)start=0 log=1
2025-03-14T04:11:56Z INFO - Cylc version: 7.9.9
2025-03-14T04:11:56Z INFO - Run mode: live
2025-03-14T04:11:56Z INFO - Initial point: 01000101T0000Z
2025-03-14T04:11:56Z INFO - Final point: 01000328T2359Z
2025-03-14T04:11:56Z INFO - Cold Start 01000101T0000Z
2025-03-14T04:11:56Z INFO - [make_drivers.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [install_cold.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [make_cice.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [make_mom.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [install_ancil.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:56Z INFO - [fcm_make_um.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:11:58Z INFO - [fcm_make_um.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [fcm_make_um.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [install_ancil.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [install_ancil.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [install_cold.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [install_cold.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_cice.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_cice.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_drivers.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_drivers.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:11:58Z INFO - [make_mom.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:11:57Z for job(01)
2025-03-14T04:11:58Z INFO - [make_mom.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:04Z INFO - [make_drivers.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_drivers.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [install_cold.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [install_cold.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [make_cice.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_cice.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [make_mom.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [make_mom.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [install_ancil.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:02Z for job(01)
2025-03-14T04:12:04Z INFO - [install_ancil.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:12:04Z INFO - [client-command] get_latest_state dm5220@gadi-login-08.gadi.nci.org.au:cylc-gui c842e8fb-017a-47ec-8706-7fef0af6d5f5
2025-03-14T04:12:06Z INFO - [make_drivers.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:05Z for job(01)
2025-03-14T04:12:09Z INFO - [make_cice.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:07Z for job(01)
2025-03-14T04:12:09Z INFO - [install_ancil.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:07Z for job(01)
2025-03-14T04:12:10Z INFO - [make2_cice.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:12:11Z INFO - [make2_cice.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:12:10Z for job(01)
2025-03-14T04:12:11Z INFO - [make2_cice.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:14Z INFO - [make_mom.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:12Z for job(01)
2025-03-14T04:12:15Z INFO - [make2_mom.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:12:18Z INFO - [make2_mom.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:12:15Z for job(01)
2025-03-14T04:12:18Z INFO - [make2_mom.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:12:37Z INFO - [install_cold.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:12:35Z for job(01)
2025-03-14T04:12:48Z INFO - [make2_mom.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:45Z for job(01)
2025-03-14T04:12:48Z INFO - [make2_mom.01000101T0000Z] -health check settings: execution timeout=PT40M, polling intervals=PT31M,PT2M,PT7M,...
2025-03-14T04:12:59Z INFO - [fcm_make_um.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:57Z for job(01)
2025-03-14T04:12:59Z INFO - [fcm_make_um.01000101T0000Z] -health check settings: execution timeout=PT1H10M, polling intervals=PT1H1M,PT2M,PT7M,...
2025-03-14T04:13:00Z INFO - [make2_cice.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:12:58Z for job(01)
2025-03-14T04:13:00Z INFO - [make2_cice.01000101T0000Z] -health check settings: execution timeout=PT30M, polling intervals=PT21M,PT2M,PT7M,...
2025-03-14T04:15:58Z INFO - [make2_cice.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:15:57Z for job(01)
2025-03-14T04:18:19Z INFO - [make2_mom.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:18:17Z for job(01)
2025-03-14T04:20:10Z INFO - [fcm_make_um.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:20:08Z for job(01)
2025-03-14T04:20:11Z INFO - [fcm_make2_um.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:20:13Z INFO - [fcm_make2_um.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:20:12Z for job(01)
2025-03-14T04:20:13Z INFO - [fcm_make2_um.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:20:37Z INFO - [fcm_make2_um.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:20:35Z for job(01)
2025-03-14T04:20:37Z INFO - [fcm_make2_um.01000101T0000Z] -health check settings: execution timeout=PT1H10M, polling intervals=PT1H1M,PT2M,PT7M,...
2025-03-14T04:37:11Z INFO - [fcm_make2_um.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:37:09Z for job(01)
2025-03-14T04:37:12Z INFO - [recon.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:37:19Z INFO - [recon.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:37:14Z for job(01)
2025-03-14T04:37:19Z INFO - [recon.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:37:53Z INFO - [recon.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:37:52Z for job(01)
2025-03-14T04:37:53Z INFO - [recon.01000101T0000Z] -health check settings: execution timeout=PT30M, polling intervals=PT21M,PT2M,PT7M,...
2025-03-14T04:38:28Z INFO - [recon.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:38:28Z for job(01)
2025-03-14T04:38:29Z INFO - [coupled.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:38:30Z INFO - [coupled.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:38:30Z for job(01)
2025-03-14T04:38:30Z INFO - [coupled.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:42:00Z INFO - [coupled.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:41:59Z for job(01)
2025-03-14T04:42:00Z INFO - [coupled.01000101T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T04:45:28Z INFO - [coupled.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:45:27Z for job(01)
2025-03-14T04:45:29Z INFO - [filemove.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:45:30Z INFO - [filemove.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:45:30Z for job(01)
2025-03-14T04:45:30Z INFO - [filemove.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:12Z INFO - [filemove.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:46:11Z for job(01)
2025-03-14T04:46:12Z INFO - [filemove.01000101T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T04:46:26Z INFO - [filemove.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:46:25Z for job(01)
2025-03-14T04:46:27Z INFO - [history_postprocess.01000101T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:46:27Z INFO - [coupled.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:46:28Z INFO - [coupled.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:46:28Z for job(01)
2025-03-14T04:46:28Z INFO - [coupled.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:28Z INFO - [history_postprocess.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:46:28Z for job(01)
2025-03-14T04:46:28Z INFO - [history_postprocess.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:46:58Z INFO - [history_postprocess.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:46:57Z for job(01)
2025-03-14T04:46:58Z INFO - [history_postprocess.01000101T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T04:47:09Z INFO - [coupled.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:47:09Z for job(01)
2025-03-14T04:47:09Z INFO - [coupled.01000201T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T04:47:11Z INFO - [history_postprocess.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:47:10Z for job(01)
2025-03-14T04:47:12Z INFO - [housekeep.01000101T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T04:47:13Z INFO - [housekeep.01000101T0000Z] status=ready: (internal)submitted at 2025-03-14T04:47:13Z for job(01)
2025-03-14T04:47:13Z INFO - [housekeep.01000101T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:47:18Z INFO - [housekeep.01000101T0000Z] status=submitted: (received)started at 2025-03-14T04:47:17Z for job(01)
2025-03-14T04:47:18Z INFO - [housekeep.01000101T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:47:21Z INFO - [housekeep.01000101T0000Z] status=running: (received)succeeded at 2025-03-14T04:47:20Z for job(01)
2025-03-14T04:50:13Z INFO - [coupled.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:50:11Z for job(01)
2025-03-14T04:50:14Z INFO - [filemove.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:50:15Z INFO - [filemove.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:50:14Z for job(01)
2025-03-14T04:50:15Z INFO - [filemove.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:50:54Z INFO - [filemove.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:50:52Z for job(01)
2025-03-14T04:50:54Z INFO - [filemove.01000201T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T04:51:08Z INFO - [filemove.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:06Z for job(01)
2025-03-14T04:51:09Z INFO - [history_postprocess.01000201T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:51:09Z INFO - [coupled.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T04:51:10Z INFO - [coupled.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:09Z for job(01)
2025-03-14T04:51:10Z INFO - [coupled.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:10Z INFO - [history_postprocess.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:09Z for job(01)
2025-03-14T04:51:10Z INFO - [history_postprocess.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:23Z INFO - [history_postprocess.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:51:22Z for job(01)
2025-03-14T04:51:23Z INFO - [history_postprocess.01000201T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T04:51:35Z INFO - [history_postprocess.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:34Z for job(01)
2025-03-14T04:51:36Z INFO - [housekeep.01000201T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T04:51:37Z INFO - [housekeep.01000201T0000Z] status=ready: (internal)submitted at 2025-03-14T04:51:37Z for job(01)
2025-03-14T04:51:37Z INFO - [housekeep.01000201T0000Z] -health check settings: submission timeout=P3D
2025-03-14T04:51:41Z INFO - [housekeep.01000201T0000Z] status=submitted: (received)started at 2025-03-14T04:51:40Z for job(01)
2025-03-14T04:51:41Z INFO - [housekeep.01000201T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T04:51:45Z INFO - [housekeep.01000201T0000Z] status=running: (received)succeeded at 2025-03-14T04:51:43Z for job(01)
2025-03-14T05:01:00Z INFO - [coupled.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:00:59Z for job(01)
2025-03-14T05:01:00Z INFO - [coupled.01000301T0000Z] -health check settings: execution timeout=PT2H10M, polling intervals=PT2H1M,PT2M,PT7M,...
2025-03-14T05:04:40Z INFO - [coupled.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:04:39Z for job(01)
2025-03-14T05:04:41Z INFO - [filemove.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T05:04:43Z INFO - [filemove.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:04:43Z for job(01)
2025-03-14T05:04:43Z INFO - [filemove.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:05:25Z INFO - [filemove.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:05:24Z for job(01)
2025-03-14T05:05:25Z INFO - [filemove.01000301T0000Z] -health check settings: execution timeout=PT15M, polling intervals=PT6M,PT2M,PT7M,...
2025-03-14T05:05:40Z INFO - [filemove.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:05:39Z for job(01)
2025-03-14T05:05:41Z INFO - [history_postprocess.01000301T0000Z] -submit-num=01, owner@host=localhost
2025-03-14T05:05:42Z INFO - [history_postprocess.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:05:42Z for job(01)
2025-03-14T05:05:42Z INFO - [history_postprocess.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:05:57Z INFO - [history_postprocess.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:05:56Z for job(01)
2025-03-14T05:05:57Z INFO - [history_postprocess.01000301T0000Z] -health check settings: execution timeout=PT1H40M, polling intervals=PT1H31M,PT2M,PT7M,...
2025-03-14T05:06:08Z INFO - [history_postprocess.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:06:07Z for job(01)
2025-03-14T05:06:09Z INFO - [housekeep.01000301T0000Z] -submit-num=01, owner@host=cylc.dm5220.tm70.ps.gadi.nci.org.au
2025-03-14T05:06:10Z INFO - [housekeep.01000301T0000Z] status=ready: (internal)submitted at 2025-03-14T05:06:10Z for job(01)
2025-03-14T05:06:10Z INFO - [housekeep.01000301T0000Z] -health check settings: submission timeout=P3D
2025-03-14T05:06:18Z INFO - [housekeep.01000301T0000Z] status=submitted: (received)started at 2025-03-14T05:06:16Z for job(01)
2025-03-14T05:06:18Z INFO - [housekeep.01000301T0000Z] -health check settings: execution timeout=PT12H
2025-03-14T05:06:20Z INFO - [housekeep.01000301T0000Z] status=running: (received)succeeded at 2025-03-14T05:06:19Z for job(01)
2025-03-14T05:06:20Z INFO - Suite shutting down - AUTOMATIC
2025-03-14T05:06:28Z INFO - DONE

This file helps identify specific tasks that failed during the suite run.

Tip

When a task fails, the LOG-TYPE will typically be ERROR or CRITICAL, instead of the more common INFO.

Once a specific task and Cylc cycle point are identified, the task-specific logs can be inspected.

Task-specific logs

Logs for individual tasks are located in subfolders within the logs folder, following this path structure:

~/cylc-run/<suite-ID>/log/job/<cylc-cycle-point>/<task-name>/<retry-number>

The <retry-number> indicates the number of retries for the same task, with the latest retry symlinked to NN.

For example, logs for most recent retry of a task named Lismore_d1000_GAL9_um_recon at Cylc cycle point 20220226T0000Z can be found in the folder ~/cylc-run/<suite-ID>/log/job/20220226T0000Z/Lismore_d1000_GAL9_um_recon/NN.

Within this directory, the job.out and job.err files (representing STDOUT and STDERR, respectively) can be found, along with other related log files.

Tip

Within the Cylc GUI, logs for a specific task can be viewed by right-clicking on the task and selecting the desired log from the View Job Logs (Viewer) menu.

Stop, restart, reload and clean suites

In some cases, you may want to control the running state of a suite.
If your Cylc GUI has been closed and you are unsure whether your suite is still running, you can scan for active suites and reopen the GUI.
To scan for active suites, run:

cylc scan

To reopen the Cylc GUI, run the following command from within the suite directory:

rose suite-gcontrol

STOP a suite

To shutdown a suite in a safe manner, run the following command from within the suite directory:

rose suite-stop -y

Alternatively, you can directly kill the PBS jobs connected to your run. To do so:

1. Check the status of all your PBS jobs:

   qstat -u $USER
   

2. Delete any job related to your run:

   qdel <job-ID>
   

RESTART a suite

There are two main ways to restart a suite:

• SOFT restart
  To reinstall the suite and reopen Cylc in the same state it was prior to being stopped, run the following command from within the suite directory:

  rose suite-run --restart
  

  Warning

  You may need to manually trigger failed tasks from the Cylc GUI.

• HARD restart
  To overwrite any previous runs of the suite and start afresh, run the following command from within the suite directory:

  rose suite-run --new
  

  Warning

  This will overwrite all existing model output and logs for the same suite.

RELOAD a suite

In some cases, the suite needs to be updated without necessarily having to stop it (e.g., after fixing a typo in a file). Updating an active suite is called a reload, where the suite is re-installed and Cylc is updated with the changes. This is similar to a SOFT restart, except new changes are installed, so you may need to manually trigger failed tasks from the Cylc GUI.

To reload a suite, run the following command from within the suite directory:

rose suite-run --reload

CLEAN a suite

To remove all files and folders created by the suite within the /scratch/$PROJECT/$USER/cylc-run/<suite-ID> directory, run the following command from within the suite directory:

rose suite-clean

Alternatively, you can achieve the same behaviour within a new submission of an experiment, by appending the --new option to the rose suite-run command:

rose suite-run --new

Warning

Cleaning a suite folder will remove any non-archived data (i.e., output files, logs, executables, etc.) associated with the suite.

RAS output files

The RAS output ancillary files can be found in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils.
Ancillaries are divided into folders according to each nested region name, and then further separated according to each nest (i.e., Resolution) name. The path of ancillaries for a specific nest (i.e., Resolution) is /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils/<nested_region_name>/<nest_name>.

The example above has one nested_region_name called Lismore, 1 nest named era5 (outer domain corresponding to Resolution 1), and 2 inner nests (Resolution 2 and Resolution 3) named d1000 and d0198, respectively.
Thus, the ancillary files directory /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/data/ancils/ contains the following subdirectories:

• Lismore/d1000
• Lismore/d0198
• Lismore/era5

Ancillary data files are typically output in the UM fieldsfile format.

Regional Nesting Suite (RNS)

The RNS uses the ancillary files produced by the RAS to run the regional forecast for the domain of interest.

The suite-ID of the RNS is u-dg768.

Get and run RNS configuration

Steps to obtain and run the RNS, as well as monitor logs, are similar to those listed above for the RAS.
The main difference is the suite-ID, which for the RNS is u-dg768.

To get the RNS configuration, follow the steps listed in Get the RAS configuration, making sure you use the correct RNS suite-ID u-dg768 when copying the suite.

To run the RNS configuration, follow the steps listed in Run the suite.

To check the RNS suite logs, follow the steps listed in Check suite logs.

RNS output files

All the RNS output files are available in the directory /scratch/$PROJECT/$USER/cylc-run/<suite-ID>. They are also symlinked in ~/cylc-run/<suite-ID>.

The RNS output data can be found in the directory /scratch/$PROJECT/$USER/cycl-run/<suite-ID>/share/cycle, grouped for each cycle.
Within the cycle directory, outputs are divided into multiple nested subdirectories in the format <nested_region_name>/<science_configuration>/<nest_name>, with <nested_region_name> and <nest_name> referring to the respective configurable options. The <science_configuration> is usually GAL9 or RAL3.2, depending on the nest resolution.

Each <nest_name> directory has the following subdirectories:

• ics → initial conditions
• lbcs → lateral boundary conditions
• um → model output data

For example, the model output data for the first cycle (20220226T0000Z) of the Lismore experiment (Lismore nested_region_name, using a RAL3P2 science_configuration and d0198 as a nest_name) can be found in /scratch/$PROJECT/$USER/cylc-run/<suite-ID>/share/cycle/20220226T0000Z/Lismore/d0198/RAL3P2/um.

The RNS output data files are typically in the UM fieldsfile format.

Tip

To change which output variables are produced, refer to Change the output variables

Edit ACCESS-rAM3 configuration

Tip

Due to the presence of two distinct suites (RAS and RNS), specific modifications might be required in either one or both.
For each of the following possible modifications, the relevant suite that needs editing is listed as a bullet point.

This section describes how to modify the ACCESS-rAM3 configuration.

The modifications discussed in this section can change the way the RAS and RNS are run, or how specific model components are configured and coupled together.

In general, ACCESS modelling suites can be edited either by directly modifying the configuration files within the suite directory, or by using the Rose GUI.

Warning

Unless you are an experienced user, directly modifying configuration files is usually discouraged to avoid encountering errors.

Rose GUI

To open the Rose GUI, run the following command from within the suite directory:

rose edit &

Tip

The & is optional. It allows the terminal prompt to remain active while running the Rose GUI as a separate process in the background.

Change run length

• RNS
  The RNS runs in multiple PBS jobs submissions, each one constituting a cycle. The job scheduler automatically resubmits the suite every chosen cycling frequency until the total run length is reached.
  

  Warning

  The cycling frequency is currently set to 24 hours (1 day) and should be left unchanged to avoid errors.
  This also means the model will run for a minimum of 1 day.

  The run length is calculated using the INITIAL_CYCLE_POINT and FINAL_CYCLE_POINT fields.
  Both these fields use ISO 8601 date format, with FINAL_CYCLE_POINT also accepting relative ISO 8601 Durations.

  Warning

  INITIAL_CYCLE_POINT and FINAL_CYCLE_POINT define all the Cylc cycle points that are set within the experiment run.
  The model will always run for a full cycling frequency (1 day) for each Cylc cycle point.
  This means, for example, that with INITIAL_CYCLE_POINT set to 20220226T0000Z, and FINAL_CYCLE_POINT set to +P1D (plus 1 day), 2 Cylc cycle points will be set (20220226T0000Z and 20220227T0000Z). Therefore, the model will run for a total of 2 days!
  To avoid running the model for longer that desired, we suggest adding -PT1S (minus 1 second) to the relative duration specified in the FINAL_CYCLE_POINT (refer to the example below).

  To modify these parameters within the Rose GUI, navigate to suite conf → Nesting Suite → Cycling options. Edit the related field and click the Save button .
  For example, to run the experiment for 2 days starting on the 5th April 2000, set INITIAL_CYCLE_POINT to 20000405T0000Z and FINAL_CYCLE_POINT to +P2D-PT1S (due to the run length mismatch).

Change the land-surface initial conditions source

• RNS
  To change the land-surface initial conditions source within the Rose GUI, navigate to suite conf → Nesting Suite → Driving model setup. Edit the NCI_HRES_ECCB field and click the Save button .

  For example, to get the land-surface initial conditions from the BARRA-R2 dataset, set the NCI_HRES_ECCB field to BARRA2-R.

Warning

When changing the land-surface initial conditions source, it is important to ensure that the configuration of the nested region aligns with the nest configuration requirements.

Change the simulation region

In ACCESS-rAM3, users can perform simulations for a particular region of the Earth by configuring specific parameters for each domain of interest (referred to as nested region).

In ACCESS-rAM3, the following parameters are supported to configure the nested regions:

• Nested region name
• Nested region position
• Nested region's nest configuration

Warning

Domain-specific changes need to be consistent between RAS and RNS. Therefore, for each of the configuration parameters listed above, consistent changes to both RAS and RNS will be required.

Change the nested region name

• RAS
  To change a nested region name within the Rose GUI, navigate to suite conf → Regional Ancillary Suite → Nested region 1 setup. Edit the rg01_name field and click the Save button .

  For example, to set the name of the nested region to Darwin, set the rg01_name field to Darwin.

• RNS
  Changing the RAS nested region name changes the RAS output path. As a consequence, the following changes are required within the RNS:

  • Ancillary directory
    To change the first nest ancillary directory within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup → Resolution 1 setup. Change the rg01_rs01_ancil_dir field by replacing Lismore with the chosen RAS nested region name, and click the Save button .
    The same step needs to be repeated for:

    • suite conf → Nesting Suite → Nested region 1 setup → Resolution 2 setup rg01_rs02_ancil_dir field
    • suite conf → Nesting Suite → Driving model setup dm_ec_lam_ancil_dir field

    For example, if the RAS nested region name was set to Darwin, replace Lismore in the rg01_rs01_ancil_dir, rg01_rs02_ancil_dir and dm_ec_lam_ancil_dir fields with Darwin.

  • RNS nested region name
    To change the nested region name within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup. Edit the rg01_name field and click the Save button .

    For example, to set the name of the nested region to Darwin, set the rg01_name field to Darwin.

    Tip

    Changing the RNS nested region name is not strictly necessary, but it affects the RNS outputs path. Therefore, for consistency, it is strongly recommended for RAS and RNS to have the same nested region names.

Change the nested region position

The nested region position is usually defined by the latitude and longitude coordinates of the nested region centre.

• RAS
  To change the nested region centre within the Rose GUI, navigate to suite conf → Regional Ancillary Suite → Nested region 1 setup. Edit the rg01_centre field and click the Save button .

  For example, to set the centre of the nested region to -12.4 / 130.8, set the rg01_centre field to -12.4 / 130.8.

Warning

When changing the land-surface initial conditions source, it is important to ensure that the configuration of the nested region aligns with the nest configuration requirements.

Change the nested region's nest configuration

Each nested region can contain multiple nests (referred to as Resolutions within the RAS and RNS), each of them being a separate domain where the simulation experiment is carried out.
Typically, nests within the same nested region are arranged concentrically, with increasingly smaller dimensions and higher resolutions towards the innermost nests.

Warning

Currently, ACCESS-rAM3 only supports specific nest configurations that meet the following criteria:

The grid points of the RAS first inner nest (i.e., Resolution 2, because Resolution 1 always corresponds to the outer ERA5 domain) must align with those of the land-surface initial conditions dataset. Thus, the configuration of the RAS first inner nest (Resolution 2), including its position, dimension and resolution, need to be modified accordingly. Note that the position of a nest is also influenced by the nested region position.

Change the output variables

UM outputs are usually provided as a list of STASH variables.
Manually specifying each STASH variable can be complex. To simplify the selection process for commonly used climate analysis variables, predefined groups of STASH variables are set up, known as stashpacks.

• RNS
  To toggle a stashpack within the Rose GUI, navigate to suite conf → Nesting Suite → Nested region 1 setup → Resolution 1 setup → Config 1 setup. Toggle a specific stashpack within the rg01_rs01_m01_stashpack field and click the Save button .
  Similar steps can be repeated for the suite conf → Nesting Suite → Nested region 1 setup → Resolution 2 setup → Config 2 setup rg01_rs02_m02_stashpack field.
  For example, to enable stashpack 6 (that includes variables such as wind gust, mean sea level pressure and rainfall amount, for every model timestep) in all nests, set the 6th button of both rg01_rs01_m01_stashpack and rg01_rs02_m02_stashpack fields to true.

Get Help

If you have questions or need help regarding ACCESS-rAM3, consider creating a topic in the Regional Nesting Suite category of the ACCESS-Hive Forum.
For assistance on how to request help from ACCESS-NRI, follow the guidelines on how to get help.

---

References

• https://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf
• https://code.metoffice.gov.uk/doc/um/latest/um-training/rose-gui.html
• https://opus.nci.org.au/display/DAE/Run+Cylc7+Suites
• https://opus.nci.org.au/display/Help/Persistent+Sessions
• https://gmd.copernicus.org/articles/13/1999/2020/

---

Last update: April 2, 2025