Run ACCESS-CM from ARE / Gadi
Run ACCESS-CM from accessdev
The tabs above let you choose the type of workflow you would like to follow.
If you are new to ACCESS-CM, we strongly suggest you follow the ARE/Gadi workflow, as the accessdev workflow will soon be interrupted.
Prerequisites
General prerequisites
Before running ACCESS-CM, you need to fulfil general prerequisites outlined in the First Steps section.
Model-specific prerequisites
-
MOSRS account
The Met Office Science Repository Service (MOSRS) is a server run by the UK Met Office (UKMO) to support collaborative development with other partners organisations. MOSRS contains the source code and configurations for some model components in ACCESS-CM (e.g., the UM).
To apply for a MOSRS account, please contact your local institutional sponsor. -
Join the access, hr22, ki32, and ki32_mosrs projects at NCI
To join these projects, request membership on the respective access, hr22 and ki32 and ki32_mosrs NCI project pages.To request membership for the ki32_mosrs subproject you need to be member of the ki32 project first.For more information on how to join specific NCI projects, please refer to How to connect to a project. -
Connection to an ARE VDI Desktop
To run ACCESS-CM, you need to be able to start an Australian Research Environment (ARE) VDI Desktop session.
If u are not familiar with ARE, please check the Getting Started on ARE section. -
Connection to accessdev
To run ACCESS-CM, you need to be able to connect to accessdev. This is an NCI server providing configuration and run control for ACCESS-CM.
You also need to ensure there is correct communication between accessdev and Gadi.
To complete these steps, refer to the SSH & SSH Agent section in the Getting Connected to Accessdev guide.
Setup for your chosen workflow
Launch ARE VDI Session
Go to the ARE VDI page and launch a session with the following directives:-
Walltime (hours) → ≈
4per simulated year
With the current state of the ARE/Gadi workflow, the ARE VDI session needs to remain active and running for the entirety of the ACCESS-CM simulation. If the ARE VDI session expires before the end of the simulation, the simulation itself will be terminated as well.
This means thatwalltimeneeds to be set according to the simulation length.
A good estimate to calculate thewalltimeneeded is 4 hours per simulated year.ARE VDI Session cannot be spun up for more than 48 consecutive hours. This means that ACCESS-CM simulations that need more than 48 hours to complete, at the current state, need to be broken down into multiple chunks running for up to 48 hours.
In the near future this will not be necessary anymore, as there will be long running servers in place for runnning ACCESS-CM simulations. -
Queue →
normalbw -
Compute Size →
tiny(1 CPU)
ACCESS-CM runs on a different Gadi node with respect to the one where the ARE VDI session is launched.
This means that the ARE VDI session only needs to carry out setup steps as well as starting the run itself. All these tasks can be easily done with only 1 CPU. -
Project → a project you belong, with allocated SU
The project, with allocated Service Units (SU), under which you want to run your simulation. Usually (but not always) this corresponds to your$PROJECT.
For more information, check how to join relevant NCI projects. -
Storage →
gdata/access+gdata/hh5+gdata/hr22+gdata/ki32(minimum)
This is the list (joined by+signs) of project data storage needed for the ACCESS-CM simulation. In ARE, storage locations need to be explicitly defined to access data from within a VDI instance.
Since every ACCESS-CM simulation can be unique and input data can come from various sources, if your specific simulation requires data coming from projects other thanaccess,hh5,hr22orki32, you need to add those projects to the storage path.
For example, if your ACCESS-CM simulation requires data coming from/g/data/tm70and/scratch/w40, your full storage path will be:gdata/access+gdata/hh5+gdata/hr22+gdata/ki32+/gdata/tm70+scratch/w40
Open the terminal
After the new tab opens you will see a Desktop with a few folders on the left.To open the terminal click on the black terminal icon at the top of the window.
As you can see from your terminal, you are now connected to a Gadi computing node.
ssh accessdevGet ACCESS-CM suite
ACCESS-CM comprises the model components UM, MOM, CICE, CABLE and OASIS. These components, which have different model parameters, input data and computer-related information, need to be packaged together as a suite in order to run.
Each ACCESS-CM suite has a suite-ID in the format u-<suite-name>, where <suite-name> is a unique identifier.
u-cy339, which is a preindustrial experiment suite.
Typically, an existing suite is copied and then edited as needed for a particular run.
u-br565, which is the CMIP6-release preindustrial experiment suite.
Typically, an existing suite is copied and then edited as needed for a particular run.
Copy ACCESS-CM suite with Rosie
Rosie is an SVN repository wrapper with a set of options specific for ACCESS-CM suites.
-
Get Cylc7 setup
To get the Cylc7 setup required to run ACCESS-CM, execute the following commands:module use /g/data/hr22/modulefiles module load cylc7module use /g/data/hr22/modulefiles module load cylc7 Loading cylc7/23.03 Loading requirement: mosrs-setup/1.0.1 -
MOSRS authentication
To authenticate using your MOSRS credentials, run:mosrs-authmosrs-auth INFO: You need to enter your MOSRS credentials here so that GPG can cache your password. Please enter the MOSRS password for <MOSRS-username>: INFO: Checking your credentials using Subversion. Please wait. INFO: Successfully accessed Subversion with your credentials. INFO: Checking your credentials using rosie. Please wait. INFO: Successfully accessed rosie with your credentials. -
Copy a suite
-
Local-only copy
To create a local copy of the<suite-ID>from the UKMO repository, run:rosie checkout <suite-ID>rosie checkout <suite-ID> [INFO] create: /home/565/<$USER>/roses [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> -
Remote and local copy
Alternatively, to create a new copy of an existing<suite-ID>both locally and remotely in the UKMO repository, run:rosie copy <suite-ID>rosie copy <suite-ID> Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> <suite-ID>is generated within the repository and populated with descriptive information about the suite and its initial configuration.
-
Local-only copy
rosie options, run:
rosie helpSuites are created in the user's home directory on Gadi under
~/roses/<suite-ID>.
-
MOSRS authentication
To authenticate using your MOSRS credentials, run:mosrs-authmosrs-auth Please enter the MOSRS password for <MOSRS-username>: Successfully authenticated with MOSRS as <MOSRS-username> -
Copy a suite
-
Local-only copy
To create a local copy of the<suite-ID>from the UKMO repository, run:rosie checkout <suite-ID>rosie checkout <suite-ID> [INFO] create: /home/565/<$USER>/roses [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<suite-ID> -
Remote and local copy
Alternatively, to create a new copy of an existing<suite-ID>both locally and remotely in the UKMO repository, run:rosie copy <suite-ID>rosie copy <suite-ID> Copy "<suite-ID>/trunk@<trunk-ID>" to "u-?????"? [y or n (default)] y [INFO] <new-suite-ID>: created at https://code.metoffice.gov.uk/svn/roses-u/<suite-n/a/m/e/> [INFO] <new-suite-ID>: copied items from <suite-ID>/trunk@<trunk-ID> [INFO] <suite-ID>: local copy created at /home/565/<$USER>/roses/<new-suite-ID> <suite-ID>is generated within the repository and populated with descriptive information about the suite and its initial configuration.
-
Local-only copy
rosie options, run:
rosie helpSuites are created in the user's home directory on accessdev under
~/roses/<suite-ID>.
Each suite directory usually contains two subdirectories and three files:
app→ directory containing the configuration files for various tasks within the suite.meta→ directory containing the GUI metadata.rose-suite.conf→ main suite configuration file.rose-suite.info→ suite information file.suite.rc→ Cylc control script file (Jinja2 language).
Edit ACCESS-CM suite configuration
Rose
Rose is a configuration editor which can be used to view, edit, or run an ACCESS-CM suite.
To edit a suite configuration, run the following command from within the suite directory (e.g., ~/roses/<suite-ID>) to open the Rose GUI:
rose edit && is optional. It allows the terminal prompt to remain active while running the Rose GUI as a separate process in the background.
Change NCI project
To ensure that your suite is run under the correct NCI project for which you are a member, edit the Compute project field in suite conf → Machine and Runtime Options, and click the Save button 
.
For example, to run an ACCESS-CM suite under the tm70 project (ACCESS-NRI), enter tm70 in the Compute project field:
Change run length and cycling frequency
ACCESS-CM suites are often run in multiple steps, each one constituting a cycle. The job scheduler resubmits the suite every chosen Cycling frequency until the Total Run length is reached.
To modify these parameters, navigate to suite conf → Run Initialisation and Cycling, edit the respective fields (using ISO 8601 Duration format) and click the Save button .
For example, to run a suite for a total of 50 years with a 1-year job resubmission, change Total Run length to P50Y and Cycling frequency to P1Y (the maximum Cycling frequency is currently two years):
Change wallclock time
The Wallclock time is the time requested by the PBS job to run a single cycle. If this time is insufficient for the suite to complete a cycle, your job will be terminated before completing the run. Hence, if you change the Cycling frequency, you may also need to change the Wallclock time accordingly. While the time required for a suite to complete a cycle depends on several factors, a good estimation is 4 hours per simulated year.
To modify the Wallclock time, edit the respective field in suite conf → Run Initialisation and Cycling (using ISO 8601 Duration format) and click the Save button .
Run ACCESS-CM suite
ACCESS-CM suites run on Gadi through a PBS job submission.
When the suite runs, its configuration files are copied on Gadi inside /scratch/$PROJECT/$USER/cylc-run/<suite-ID> and a symbolic link to this directory is also created in the $USER's home directory under ~/cylc-run/<suite-ID>.
An ACCESS-CM suite comprises several tasks, such as checking out code repositories, compiling and building the different model components, running the model, etc. The workflow of these tasks is controlled by Cylc.
Cylc
Cylc (pronounced ‘silk’) is a workflow manager that automatically executes tasks according to the model's main cycle script suite.rc. Cylc controls how the job will be run and manages the time steps of each submodel. It also monitors all tasks, reporting any errors that may occur.
To run an ACCESS-CM suite run the following command from within the suite directory:
rose suite-runAfter the initial tasks are executed, the Cylc GUI will open. You can now view and control the different tasks in the suite as they are run:
rose suite-run, if you get an error similar to the following:
[FAIL] Suite "<suite-ID>" appears to be running:
[FAIL] Contact info from: "/home/565/<$USER>/cylc-run/<suite-ID>/.service/contact"
[FAIL] CYLC_SUITE_HOST=accessdev.nci.org.au
[FAIL] CYLC_SUITE_OWNER=<$USER>
[FAIL] CYLC_SUITE_PORT=<port>
[FAIL] CYLC_SUITE_PROCESS=<PID> python2 /usr/local/cylc/cylc-7.8.3/bin/cylc-run <suite-ID>
[FAIL] Try "cylc stop '<suite-ID>'" first?rm /home/565/<$USER>/cylc-run/<suite-ID>/.service/contactrose suite-run command again.
You are done!!
If you do not get any errors, you can check the suite output files after the run is complete.
You can now close the Cylc GUI. To open it again, run the following command from within the suite directory:
rose suite-gcontrolMonitor ACCESS-CM runs
Check for errors
It is quite common, especially during the first few runs, to experience errors and job failures. Running an ACCESS-CM suite involves the execution of several tasks, and any of these tasks could fail. When a task fails, the suite is halted and a red icon appears next to the respective task name in the Cylc GUI.
To investigate the cause of a failure, we need to look at the logs job.err and job.out from the suite run. There are two main ways to do so:
-
Using the Cylc GUI
Right-click on the task that failed and click on View Job Logs (Viewer) → job.err or job.out.
To access a specific task, click on the arrow next to the task to extend the drop-down menu with all the subtasks.
-
Through the suite directory
The suite's log directories are stored in~/cylc-run/<suite-ID>aslog.<TIMESTAMP>, and the latest set of logs are also symlinked in the~/cylc-run/<suite-ID>/logdirectory.
The logs for the main job can be found in the~/cylc-run/<suite-ID>/log/jobdirectory.
Logs are separated into simulation cycles according to their starting dates, and then differentiated by task. They are then further separated into "attempts" (consecutive failed/successful tasks), whereNNis a symlink to the most recent attempt.
In the example above, a failure occurred for the 09500101 simulation cycle (i.e. starting date \: 1st January 950) in the coupled task. Hence, thejob.errandjob.outfiles can be found in the~/cylc-run/<suite-ID>/log/job/09500101/coupled/NNdirectory.cd ~/cylc-run/<suite-ID> ls app cylc-suite.db log log.20230530T051952Z meta rose-suite.info share suite.rc suite.rc.processed work cd log ls db job rose.conf rose-suite-run.conf rose-suite-run.locs rose-suite-run.log rose-suite-run.version suite suiterc cd job ls 09500101 cd 09500101 ls coupled fcm_make2_um fcm_make_um install_warm make2_mom make_mom fcm_make2_drivers fcm_make_drivers install_ancil make2_cice make_cice cd coupled ls 01 02 03 NN cd NN ls job job-activity.log job.err job.out job.status
Model Live Diagnostics
ACCESS-NRI developed the Model Live Diagnostics framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on Gadi.
For a complete documentation on how to use this framework, check the Model Diagnostics documentation.
Stop, restart and reload suites
In some cases, you may want to control the running state of a suite.
If your Cylc GUI has been closed and you are unsure whether your suite is still running, you can scan for active suites and reopen the GUI if desired.
To scan for active suites, run:
cylc scanTo reopen the Cylc GUI, run the following command from within the suite directory:
rose suite-gcontrolSTOP a suite
To shutdown a suite in a safe manner, run the following command from within the suite directory:
rose suite-stop -yAlternatively, you can directly kill the PBS job(s) connected to your run. To do so:
-
Check the status of all your PBS jobs:
qstat -u $USER -
Delete any job related to your run:
qdel <job-ID>
RESTART a suite
There are two main ways to restart a suite:
-
SOFT restart
To reinstall the suite and reopen Cylc in the same state it was prior to being stopped, run the following command from within the suite directory:rose suite-run --restartYou may need to manually trigger failed tasks from the Cylc GUI.cylc cd ~/roses/<suite-ID> rose suite-run --restart [INFO] export CYLC_VERSION=7.9.7 [INFO] export ROSE_ORIG_HOST=<gadi-cpu>.nci.org.au [INFO] export ROSE_SITE=nci [INFO] export ROSE_VERSION=2019.01.2 [INFO] delete: log/rose-suite-run.conf [INFO] symlink: rose-conf/<timestamp>-restart.conf <= log/rose-suite-run.conf [INFO] delete: log/rose-suite-run.version [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version [INFO] chdir: log/ [INFO] ._. [INFO] | | The Cylc Suite Engine [7.9.7] [INFO] ._____._. ._| |_____. Copyright (C) 2008-2019 NIWA [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; [INFO] .___! | see `cylc warranty`. It is free software, you [INFO] !_____! are welcome to redistribute it under certain [INFO] [INFO] *** listening on https://<gadi-cpu>.nci.org.au:<port>/ *** [INFO] [INFO] To view suite server program contact information: [INFO] $ cylc get-suite-contact <suite-ID> [INFO] [INFO] Other ways to see if the suite is still running: [INFO] $ cylc scan -n '<suite-ID>' <gadi-cpu>.nci.org.au [INFO] $ cylc ping -v --host=<gadi-cpu>.nci.org.au <suite-ID> [INFO] $ ps -opid,args <PID> # on <gadi-cpu>.nci.org.au
cylc cd ~/roses/<suite-ID> rose suite-run --restart [INFO] export CYLC_VERSION=7.8.3 [INFO] export ROSE_ORIG_HOST=accessdev.nci.org.au [INFO] export ROSE_SITE= [INFO] export ROSE_VERSION=2019.01.2 [INFO] delete: log/rose-suite-run.conf [INFO] symlink: rose-conf/<timestamp>-restart.conf <= log/rose-suite-run.conf [INFO] delete: log/rose-suite-run.version [INFO] symlink: rose-conf/<timestamp>-restart.version <= log/rose-suite-run.version [INFO] chdir: log/ [INFO] ._. [INFO] | | The Cylc Suite Engine [7.8.3] [INFO] ._____._. ._| |_____. Copyright (C) 2008-2019 NIWA [INFO] | .___| | | | | .___| & British Crown (Met Office) & Contributors. [INFO] | !___| !_! | | !___. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [INFO] !_____!___. |_!_____! This program comes with ABSOLUTELY NO WARRANTY; [INFO] .___! | see `cylc warranty`. It is free software, you [INFO] !_____! are welcome to redistribute it under certain [INFO] [INFO] *** listening on https://accessdev.nci.org.au:<port>/ *** [INFO] [INFO] To view suite server program contact information: [INFO] $ cylc get-suite-contact <suite-ID> [INFO] [INFO] Other ways to see if the suite is still running: [INFO] $ cylc scan -n '<suite-ID>' accessdev.nci.org.au [INFO] $ cylc ping -v --host=accessdev.nci.org.au <suite-ID> [INFO] $ ps -opid,args <PID> # on accessdev.nci.org.au
-
HARD restart
To overwrite any previous runs of the suite and start afresh, run the following command from within the suite directory:rose suite-run --newWARNING!! This will overwrite all existing model output and logs for the same suite.
RELOAD a suite
In some cases the suite needs to be updated without necessarily having to stop it (e.g., after fixing a typo in a file). Updating an active suite is called a reload, where the suite is re-installed and Cylc is updated with the changes. This is similar to a SOFT restart, except new changes are installed, so you may need to manually trigger failed tasks from the Cylc GUI.
To reload a suite, run the following command from within the suite directory:
rose suite-run --reloadACCESS-CM output files
All ACCESS-CM output files, together with work files, are available on Gadi inside /scratch/$PROJECT/$USER/cylc-run/<suite-ID>. They are also symlinked in ~/cylc-run/<suite-ID>.
While the suite is running, files are moved between the share and work directories.
At the end of each cycle, model output data and restart files are moved to /scratch/$PROJECT/$USER/archive/<suite-name>.
This directory contains two subdirectories:
historyrestart
Output data
/scratch/$PROJECT/$USER/archive/<suite-name>/history is the directory containing the model output data, which is grouped according to each model component:
-
atm→ atmosphere (UM) -
cpl→ coupler (OASIS3-MCT) -
ocn→ ocean (MOM) -
ice→ ice (CICE)
For the atmospheric output data, the files are typically a UM fieldsfile or netCDF file, formatted as <suite-name>a.p<output-stream-identifier><year><month-string>.
u-cy339 suite in this example, the atm directory contains:
u-br565 suite in this example, the atm directory contains:
Restart files
The restart files can be found in the /scratch/$PROJECT/$USER/archive/<suite-name>/restart directory, where they are categorised according to model components (similar to the history folder above).
The atmospheric restart files, which are UM fieldsfiles, are formatted as <suite-name>a.da<year><month><day>_00.
u-cy339 suite in this example, the atm directory contains:
u-br565 suite in this example, the atm directory contains:
Files formatted as <suite-name>a.xhist-<year><month><day> contain metadata information.
References
- https://confluence.csiro.au/display/ACCESS/Using+CM2+suites+in+Rose+and+Cylc
- https://confluence.csiro.au/display/ACCESS/Understanding+CM2+output
- https://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf
- https://code.metoffice.gov.uk/doc/um/latest/um-training/rose-gui.html
- https://opus.nci.org.au/display/DAE/Cylc+7+on+ARE