1. Introduction

The Unified Forecast System (UFS) is a community-based, coupled, comprehensive Earth modeling system. NOAA’s operational model suite for numerical weather prediction (NWP) is quickly transitioning to the UFS from a number of different modeling systems. The UFS enables research, development, and contribution opportunities within the broader weather enterprise (e.g. government, industry, and academia). For more information about the UFS, visit the UFS Portal.

The UFS includes multiple applications that span local to global domains and a range of predictive time scales. This documentation describes the UFS Medium-Range Weather (MRW) Application (App), which targets predictions of atmospheric behavior out to about two weeks. This MRW App release includes a prognostic atmospheric model, pre- and post-processing tools, and a community workflow. These components are documented within this User’s Guide and supported through a community forum. Additionally, the MRW App has transitioned from a CIME-based workflow to the global workflow. New and improved capabilities for the upcoming release include the addition of a verification package (METplus) for both deterministic and ensemble simulations and support for four Stochastically Perturbed Perturbation (SPP) schemes. Future work will expand the capabilities of the application to include data assimilation (DA) and a forecast restart/cycling capability.

The MRW App is available on GitHub and is designed to be code that the community can run and improve. It is portable to a set of commonly used platforms. A limited set of configurations of the release, such as specific model resolutions and physics options, are documented and supported. This documentation provides a Quick Start Guide for running the MRW Application and a detailed guide for running the MRW App on supported platforms. It also provides an overview of the release components and details on how to customize or modify different portions of the workflow.

1.1. How to Use This Document

This guide instructs both novice and experienced users on downloading, building, and running the MRW Application. Please post questions in the UFS Forum.

Throughout the guide, this presentation style indicates shell commands and options,
code examples, etc.

Variables presented as AaBbCc123 in this User’s Guide typically refer to variables in scripts, names of files, and directories.

File paths or code that include angle brackets (e.g., build_<platform>_<compiler>.env) indicate that users should insert options appropriate to their MRW App configuration (e.g., build_orion_intel.env).

Hint

Note

Variables presented as $VAR in this guide typically refer to variables in XML files in a MRW App experiment.

1.2. Prerequisites for Using the MRW Application

1.2.1. Background Knowledge Prerequisites

The instructions in this documentation assume that users have certain background knowledge:

  • Familiarity with LINUX/UNIX systems

  • Command line basics

  • System configuration knowledge (e.g., compilers, environment variables, paths, etc.)

  • Numerical Weather Prediction

  • Meteorology

Additional background knowledge in the following areas could be helpful: * High-Performance Computing (HPC) Systems for those running the MRW App on an HPC system * Programming (particularly Python) for those interested in contributing to the MRW App code * Creating an SSH Tunnel to access HPC systems from the command line * Containerization * Workflow Managers/Rocoto

1.2.2. Software/Operating System Requirements

The UFS MRW Application has been designed so that any sufficiently up-to-date machine with a UNIX-based operating system should be capable of running the application. NOAA Level 1 & 2 systems already have these prerequisites installed. However, users working on other systems must ensure that the following requirements are installed on their system:

Minimum Platform Requirements:

  • UNIX style operating system such as CNL, AIX, Linux, Mac

    • >40 GB disk space

    • 18 GB input data from GFS, RAP, and HRRR for “out-of-the-box” MRW App case described in Chapter 3

    • 6 GB for spack-stack full installation

    • 1 GB for ufs-mrweather-app installation

    • 11 GB for 120hr/5-day forecast

  • 4GB memory (25km domain)

  • Python 3.7+

  • Perl 5

  • Git client (1.8+)

  • Fortran compiler released since 2018

    • gfortran v9+ or ifort v18+ are the only ones tested, but others may work.

  • C compiler compatible with the Fortran compiler

    • gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang or LLVM clang) have been tested

The following software is also required to run the MRW Application, but spack-stack (which contains the software libraries necessary for building and running the MRW App) can be configured to build these requirements:

  • MPI (MPICH, OpenMPI, or other implementation)

    • Only MPICH can be built with spack-stack. Other options must be installed separately by the user.

  • CMake 3.15+

For MacOS systems, some additional software is needed. It is recommended that users install this software using the Homebrew package manager for MacOS:

Optional but recommended prerequisites for all systems:

  • Conda for installing/managing Python packages

  • Bash v4+

  • Rocoto Workflow Management System (1.3.1)

  • Python packages scipy, matplotlib, pygrib, cartopy, and pillow for graphics

  • Lmod

You are now ready to build the MRW App as documented in the Workflow Quick Start.

1.3. MRW App Components Overview

1.3.1. Pre-Processor Utilities and Initial Conditions

The MRW App requires input model data and the chgres_cube pre-processing software, which is part of the UFS_UTILS pre-processing utilities package, to initialize and prepare the model. Additional information about the pre-processor utilities can be found in Chapter 5.1 and in the UFS_UTILS User’s Guide.

1.3.2. Forecast Model

1.3.2.1. Atmospheric Model

The prognostic atmospheric model in the UFS MRW Application uses the Finite-Volume Cubed-Sphere (FV3) dynamical core. The dynamical core is the computational part of a model that solves the equations of fluid motion for the atmospheric component of the UFS Weather Model. A User’s Guide for the UFS Weather Model can be found here. Additional information about the FV3 dynamical core is available at here.

1.3.2.2. Common Community Physics Package

The Common Community Physics Package (CCPP) supports interoperable atmospheric physics and land surface model options. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. The upcoming MRW App release includes four physics suites and stochastic options to represent model uncertainty.

1.3.3. Unified Post-Processor

The Medium-Range Weather (MRW) Application is distributed with a post-processing tool, the Unified Post Processor (UPP). The UPP converts the native netCDF output from the model to GRIB2 format on standard isobaric coordinates in the vertical direction. The UPP can also be used to compute a variety of useful diagnostic fields, as described in the UPP User’s Guide. The UPP output can be used with visualization, plotting and verification packages, or for further downstream post-processing (e.g., statistical post-processing techniques).

1.3.4. METplus Verification Suite

The Model Evaluation Tools (MET) package is a set of statistical verification tools developed by the Developmental Testbed Center (DTC) for use by the NWP community to help them assess and evaluate the performance of numerical weather predictions. MET is the core component of the enhanced METplus verification framework. The suite also includes the associated database and display systems called METviewer and METexpress. METplus spans a wide range of temporal and spatial scales. It is intended to be extensible through additional capabilities developed by the community. More details about METplus can be found in Chapter 5.4 and on the METplus website.

1.3.5. Visualization Example

This release does not include support for model visualization. Four basic NCAR Command Language (NCL) scripts are provided to create a basic visualization of model output, but this capability is provided only as an example for users familiar with NCL. The scripts may be used to complete a visual check to verify that the application is producing reasonable results.

1.3.6. Workflow and Build System

The MRW Application has a portable CMake-based build system that packages together all the components required to build the MRW Application. Once built, users can generate a Rocoto-based workflow that will run each task in the proper sequence (see Rocoto documentation for more on workflow management).

This MRW Application release has been tested on a variety of platforms widely used by researchers, including NOAA High-Performance Computing (HPC) systems (e.g. Jet, Gaea), cloud environments, and generic Linux and macOS systems. Four levels of support have been defined for the MRW Application. Preconfigured (Level 1) systems already have the required software libraries (spack-stack) available in a central location. The MRW Application is expected to build and run out-of-the-box on these systems, and users can download the MRW App code without first installing prerequisites. On other platforms, the required libraries will need to be installed as part of the MRW Application build process. On Level 2 platforms, installation should be straightforward, and the MRW App should build and run successfully. On Level 3 & 4 platforms, users may need to perform additional troubleshooting since little or no pre-release testing has been conducted on these systems.

1.4. Code Repositories and Directory Structure

1.4.1. Hierarchical Repository Structure

1.5. User Support, Documentation, and Contributing Development

A forum-based online support system with topical sections provides a centralized location for UFS users and developers to post questions and exchange information. The forum complements the distributed documentation, summarized here for ease of use.

Table 1.1 Centralized list of documentation

Documentation

Location

MRW App v2.0 User’s Guide

https://ufs-mrweather-app.readthedocs.io/en/ufs-v1.1.0

chgres_cube User’s Guide

https://ufs-utils.readthedocs.io/en/ufs-v1.1.0

UFS Weather Model v2.0 User’s Guide

https://ufs-weather-model.readthedocs.io/en/ufs-v1.1.0

FV3 Documentation

https://noaa-emc.github.io/FV3_Dycore_ufs-v1.1.0/html/index.html

CCPP Scientific Documentation

https://dtcenter.org/GMTB/v4.1.0/sci_doc

CCPP Technical Documentation

https://ccpp-techdoc.readthedocs.io/en/v4.1.0

Stochastic Physics User’s Guide

https://stochastic-physics.readthedocs.io/en/ufs-v1.1.0

ESMF manual

http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_8_0_0/ESMF_refdoc

Common Infrastructure for Modeling the Earth

http://esmci.github.io/cime/versions/ufs_release_v1.1/html/index.html

Unified Post Processor

https://upp.readthedocs.io/en/upp_v10.1.0/

The UFS community is encouraged to contribute to the development effort of all related utilities, model code, and infrastructure. Users can post issues in the related GitHub repositories to report bugs or to announce upcoming contributions to the code base. For code to be accepted in the authoritative repositories, users must follow the code management rules of each UFS component repository, which are outlined in the respective User’s Guides listed in Table 1.1. In particular, innovations involving the UFS Weather Model need to be tested using the regression tests described in its User’s Guide. These tests are part of the official NOAA policy on accepting innovations into its code base, whereas the MRW App end-to-end tests are meant as a sanity check for users.

1.6. Future Direction

Users can expect to see incremental improvements and additional capabilities in upcoming releases of the MRW Application to enhance research opportunities and support operational forecast implementations.

Planned advancements include addition of:

  • component models for other Earth domains (such as oceans and sea ice)

  • cycled data assimilation for model initialization

  • expansion of supported platforms