Norman et al. (2025) PortUrb: a performance portable, high-order, moist atmospheric large eddy simulation model with variable-friction immersed boundaries

Identification

• Journal: Geoscientific model development
• Year: 2025
• Date: 2025-12-04
• Authors: Matthew Norman, Muralikrishnan Gopalakrishnan Meena, Kalyan Gottiparthi, Nicholson Koukpaizan, Stephen Nichols
• DOI: 10.5194/gmd-18-9605-2025

Research Groups

• Oak Ridge National Laboratory, Oak Ridge, TN, USA

Short Summary

This paper introduces "portUrb," a performance-portable, high-order, moist atmospheric Large Eddy Simulation (LES) model designed for urban building geometries using variable-friction immersed boundaries. The model demonstrates accuracy and robustness across various atmospheric boundary layer, supercell, and urban flow scenarios, closely matching experimental observations and other model comparisons.

Objective

• To introduce and investigate the properties of "portUrb," a moist, compressible, non-hydrostatic atmospheric Large Eddy Simulation model emphasizing portability, performance, accuracy, simplicity, readability, robustness, extensibility, and ensemble capabilities, particularly for urban building geometries with variable-friction immersed boundaries.

Study Configuration

• Spatial Scale: Microscale flows, resolving grid spacings in the range of 1 millimeter to 1 kilometer. Domain sizes varied from 4h × 4h × 10h (for cube arrays) up to 200 km × 200 km × 20 km (for supercell simulations).
• Temporal Scale: Simulation durations ranged from 5 seconds (for cube arrays) to 10 model hours (for atmospheric boundary layers and city flow).

Methodology and Data

• Models used: portUrb (a moist, compressible, non-hydrostatic atmospheric Large Eddy Simulation model). Comparisons were made against other LES models such as FastEddy, WRF, SOWFA, HiGrad, and Morrison 2-moment microphysics.
• Data sources:
  • Experimental observations: Sandia National Laboratories Scaled Wind Farm Technology (SWiFT) field experiment (for convective ABL), wind tunnel experiments (for staggered surface-mounted cube array).
  • Model comparisons: Data from FastEddy, WRF, SOWFA, HiGrad, and Morrison and Milbrandt (2011) supercell simulations.
  • Building geometry: OpenStreetMap data ingested via Blosm plugin for Blender.

Main Results

• Neutral and Convective Atmospheric Boundary Layers: portUrb accurately simulated mean and turbulent statistics, showing good agreement with other LES models and observations (e.g., supergeostrophic wind, TKE profiles). For the convective ABL, portUrb's velocity magnitude results were closest to SWiFT observations after tuning.
• Supercell Simulation: The model successfully simulated storm splitting and precipitation, with time traces of cold pool, accumulated precipitation, maximum updraft, and minimum downdraft showing similar trends and magnitudes to established microphysics schemes.
• Staggered Surface-Mounted Cube Array: The frictioned immersed boundary approach accurately captured time-averaged mean velocity and turbulent shear stress (u'w' correlations) compared to wind tunnel observations, demonstrating its suitability for realistic turbulent dynamics around obstacles.
• Coarsely Resolved Sphere: Numerical experiments with a sphere were used to tune the immersed boundary parameters for partially immersed cells, yielding accurate mean flow statistics at low resolution compared to high-resolution simulations.
• Flow Through Manhattan Buildings: The model demonstrated physically realizable flow through complex, unsmoothed building geometries from OpenStreetMap, free from significant numerical artifacts. It effectively showed how varying building surface roughness lengths influence mean flow speed and resolved/unresolved Turbulence Kinetic Energy (TKE) within urban canyons.
• City Flow Forced by Turbulent Precursor: The precursor forcing mechanism worked correctly, and the presence of the city led to a reduction in wind speed within the atmospheric boundary layer and an increase in the potential temperature inversion height due to mechanically generated turbulence. The stretched vertical grid showed no artifacts.
• Performance: portUrb achieved comparable performance to the FastEddy model on similar aggregate memory bandwidth (e.g., 7.3 time steps per wallclock second for a 2048x2048x122 grid on 16 Nvidia A100 GPUs for 5th-order accuracy without WENO limiter).

Contributions

• Introduction of portUrb, a novel moist, compressible, non-hydrostatic atmospheric LES model with a strong emphasis on performance portability across CPUs and GPUs (Nvidia, AMD, Intel) using C++ libraries (YAKL, Kokkos).
• Implementation of a high-order (eleventh-order) Weighted Essentially Non-Oscillatory (WENO) interpolation scheme combined with an upwind Riemann solver for accuracy and robustness, particularly near immersed boundaries.
• Development of a robust variable-friction immersed boundary method that handles complex urban building geometries directly from OpenStreetMap data without requiring pre-processing or smoothing, allowing for customizable roughness lengths.
• Integration of an ensemble capability within a single executable, enabling efficient exploration of parameter spaces and rapid prototyping of new model physics and surrogate models on High-Performance Computing (HPC) facilities.
• Validation of the model across a diverse set of atmospheric and urban flow test cases, demonstrating its accuracy against established models and experimental observations.

Funding

• Advanced Scientific Computing Research (grant no. DE-AC05-00OR22725)
• Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory (supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725)

Citation

@article{Norman2025PortUrb,
  author = {Norman, Matthew and Meena, Muralikrishnan Gopalakrishnan and Gottiparthi, Kalyan and Koukpaizan, Nicholson and Nichols, Stephen},
  title = {PortUrb: a performance portable, high-order, moist atmospheric large eddy simulation model with variable-friction immersed boundaries},
  journal = {Geoscientific model development},
  year = {2025},
  doi = {10.5194/gmd-18-9605-2025},
  url = {https://doi.org/10.5194/gmd-18-9605-2025}
}