Devi et al. (2025) A framework for the evaluation of flood inundation predictions over extensive benchmark databases

Identification

• Journal: Environmental Modelling & Software
• Year: 2025
• Date: 2025-11-20
• Authors: Dipsikha Devi, Supath Dhital, Dinuke Munasinghe, Sagy Cohen, Anupal Baruah, Yixian Chen, Dan Tian, Carson Pruitt
• DOI: 10.1016/j.envsoft.2025.106786

Research Groups

• Department of Geography and the Environment, The University of Alabama, Tuscaloosa, USA
• NOAA Office of Weather Prediction Affiliate, Tuscaloosa, USA

Short Summary

This paper introduces FIMeval, an open-source framework for automated, large-scale Flood Inundation Mapping (FIM) evaluation, integrating pixel-based and impact-based assessments. It demonstrates FIMeval's utility across diverse flood scenarios and benchmark datasets, highlighting the sensitivity of evaluation outcomes to flood extent delineation methods.

Objective

• To develop and demonstrate FIMeval, an open-source, modular framework for automated, large-scale Flood Inundation Mapping (FIM) evaluation.
• To integrate pixel-based (confusion matrix metrics) and impact-based (building footprint data) assessments into the evaluation process.
• To analyze the influence of different flood extent delineation methods (Smallest Extent, Convex Hull, User-defined Area of Interest) on FIM evaluation outcomes across historical and synthetic flood events.

Study Configuration

• Spatial Scale: Contiguous United States (CONUS) for the benchmarking database; case studies included a 35 km river reach of the Neuse River, North Carolina; the Arkansas River (2016 Midwest Flood); and 45 Hydrologic Unit Code-8 (HUC-8) watersheds in FEMA Region 6 (ranging from 1769.4 km² to 8429.22 km²). Spatial resolutions of FIMs varied from 40 cm (aerial imagery) to 10 m (satellite and model-predicted).
• Temporal Scale: Historical flood events (2016 Midwest Flood, Hurricane Matthew in October 2016) and synthetic flood scenarios (100-year and 500-year return period floods).

Methodology and Data

• Models used:
  • Evaluation Framework: Flood Inundation Mapping Evaluation Framework (FIMeval) – an open-source Python toolset, Jupyter Notebook, and ArcGIS Pro Toolbox.
  • Model-predicted FIM (M-FIM): NOAA Office of Water Prediction Height Above Nearest Drainage (OWP HAND-FIM), HEC-RAS, LISFLOOD-FP, FEMA Base Level Engineering (BLE) simulations.
• Data sources:
  • Benchmark FIM (B-FIM):
    • Tier 1: Hand-labelled rasters from NOAA’s Emergency Response Imagery (40 cm resolution).
    • Tier 2: High-resolution FIM from Planet satellite imagery (integrated with gap-filled algorithm).
    • Tier 3: FIM from Sentinel-1 (integrated with gap-filled algorithm).
    • Tier 4: Synthetic FIM from FEMA’s BLE (10 m resolution, HUC-8 scale).
    • Microsoft building footprint dataset (Open Data Commons Open Database License).
    • ESRI USA Detailed Water Bodies dataset (for Permanent Water Bodies removal).
  • Model-predicted FIM (M-FIM) inputs: US National Water Model (NWM) streamflow outputs, Relative Elevation Model (REM), Height Above Nearest Drainage (HAND) hydrofabric datasets, 10 m resolution Digital Elevation Models (DEMs).
  • Software libraries: rasterio, pyproj, shapely, geopandas, numpy, pandas, boto3, PySpark (for msfootprint).

Main Results

• FIMeval demonstrated consistent performance with existing tools (e.g., NOAA OWP's GVAL) but with reduced computational time (e.g., 100.33 seconds vs. 127.68 seconds for a historical flood case).
• The choice of flood extent delineation method significantly influenced evaluation metrics for historical, localized floods: the Convex Hull (CH) method showed an average of 7.7% difference in CSI, POD, and F1 scores compared to Smallest Extent (SE) and generally reduced False Positive Rate (FPR) by approximately 24.6%.
• For large-scale synthetic floods (100-year and 500-year return periods across 45 HUC-8 watersheds), differences between CH and SE methods were minimal (approximately 0.3% for CSI, POD, F1), due to the broad and continuous nature of inundation.
• Impact-based assessments using building footprints revealed that model-predicted FIMs often underpredicted building hits in historical cases (negative Building Deviation Ratio, BDR), with pixel-based building hit statistics showing low agreement (CSI, POD, F1 scores between 0.02 and 0.39). For synthetic floods, the CH method resulted in BDR values more clustered around zero, indicating better exclusion of low-confidence flooded areas.
• FIMeval processed pixel-based evaluation for 45 HUC-8 watersheds in approximately 1 hour and 4 minutes, with an additional 3 hours and 35 minutes for building footprint analysis, demonstrating scalability for large study areas.

Contributions

• Introduction of FIMeval, an open-source, modular, and cloud-compatible framework for automated, large-scale Flood Inundation Mapping (FIM) evaluation.
• Integration of a comprehensive benchmarking database with four tiers of FIM quality (remote sensing and high-fidelity models) across the Contiguous United States.
• Support for both pixel-based (confusion matrix metrics) and impact-based (building footprint data) assessments, addressing a critical need for emergency managers.
• Inclusion of automated flood extent delineation methods (Smallest Extent, Convex Hull, User-defined AOI) to manage varying spatial extents and data imbalances.
• Demonstration of the sensitivity of evaluation metrics to the chosen flood domain delineation method, especially for historical, localized floods.
• Provision of FIMeval as a Python package, Jupyter Notebook, and ArcGIS Pro Toolbox, enhancing accessibility for both code-savvy and non-code-savvy users.

Funding

• National Oceanic and Atmospheric Administration (NOAA)
• Cooperative Institute for Research to Operations in Hydrology (CIROH)
• NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003)
• NASA Commercial Smallsat Data Acquisition (CSDA) Program

Citation

@article{Devi2025framework,
  author = {Devi, Dipsikha and Dhital, Supath and Munasinghe, Dinuke and Cohen, Sagy and Baruah, Anupal and Chen, Yixian and Tian, Dan and Pruitt, Carson},
  title = {A framework for the evaluation of flood inundation predictions over extensive benchmark databases},
  journal = {Environmental Modelling & Software},
  year = {2025},
  doi = {10.1016/j.envsoft.2025.106786},
  url = {https://doi.org/10.1016/j.envsoft.2025.106786}
}