Yuan et al. (2025) Enhancing runoff prediction with causal lag-aware attention and multi-scale fusion in transformer models

Identification

• Journal: Journal of Hydrology
• Year: 2025
• Date: 2025-10-08
• Authors: Weizheng Yuan, Hua Yan
• DOI: 10.1016/j.jhydrol.2025.134369

Research Groups

College of Electronics and Information Engineering, Sichuan University, Chengdu, Sichuan 610065, China

Short Summary

This study addresses the non-causal issue in Transformer models for runoff prediction by proposing a novel Causal Lag-Aware Attention Mechanism, a multi-scale fusion module, and a frequency-domain-based loss function, achieving significant improvements in prediction accuracy over existing state-of-the-art Transformer models.

Objective

• To reveal and address the non-causal issue inherent in traditional Transformer attention mechanisms for runoff prediction and enhance model performance by effectively capturing causal and lagged dependencies, integrating multi-level temporal features, and improving training effectiveness.

Study Configuration

• Spatial Scale: Not explicitly specified for the proposed model, but runoff prediction generally operates at basin scales.
• Temporal Scale: Not explicitly specified for the proposed model, but addresses challenges in medium- and long-term predictions.

Methodology and Data

• Models used: Enhanced Transformer model incorporating a Causal Lag-Aware Attention Mechanism, a multi-scale fusion module, and a frequency-domain-based loss function.
• Data sources: Historical meteorological data, basin characteristics, soil moisture, vegetation coverage, and runoff data (observation, potentially reanalysis for meteorological data).

Main Results

• The proposed method effectively resolves causal issues in Transformer models for runoff prediction.
• Achieved an 11.56% improvement in mean Nash-Sutcliffe Efficiency (NSE) compared to existing state-of-the-art Transformer-based runoff prediction models.
• Achieved a 3.28% improvement in median NSE compared to existing state-of-the-art Transformer-based runoff prediction models.
• Demonstrated varying degrees of enhancement across other evaluation metrics.

Contributions

• First to reveal the non-causal issue in traditional Transformer attention mechanisms for runoff prediction.
• Proposed a novel Causal Lag-Aware Attention Mechanism that integrates causal and lag-aware attention.
• Designed a multi-scale fusion module to enhance the characterization of complex runoff patterns.
• Introduced a novel frequency-domain-based loss function leveraging spectral information for improved training.
• Achieved significant performance improvements over state-of-the-art Transformer-based runoff prediction models.

Funding

Not specified in the provided text.

Citation

@article{Yuan2025Enhancing,
  author = {Yuan, Weizheng and Yan, Hua},
  title = {Enhancing runoff prediction with causal lag-aware attention and multi-scale fusion in transformer models},
  journal = {Journal of Hydrology},
  year = {2025},
  doi = {10.1016/j.jhydrol.2025.134369},
  url = {https://doi.org/10.1016/j.jhydrol.2025.134369}
}