Farman et al. (2025) Activation function impact on rainfall prediction: comparative insights across ML and DL architectures

Identification

• Journal: Modeling Earth Systems and Environment
• Year: 2025
• Date: 2025-10-06
• Authors: Hira Farman, Muhammad Arif Hussain, Saif Hassan, Sarang Shaikh, Khurshed Ali
• DOI: 10.1007/s40808-025-02630-6

Research Groups

• Iqra University, Karachi, Pakistan
• Karachi Institute of Economics and Technology, Karachi, Pakistan
• Sukkur IBA University, Sukkur, Pakistan
• Norwegian University of Science and Technology, Gjovik, Norway

Short Summary

This study systematically compares the impact of various activation functions on deep learning (LSTM, BiLSTM, Transformer) and machine learning models for next-day rainfall prediction. It finds that BiLSTM with Leaky ReLU/ELU and Transformer with ELU/ReLU/Swish consistently achieve the highest accuracy (up to 99%) and stability, significantly outperforming traditional ML models.

Objective

• To systematically compare the performance of various deep learning architectures (LSTM, BiLSTM, Transformer) and traditional machine learning classifiers (Logistic Regression, SVM, KNN, Naïve Bayes, Passive Aggressive Classifier) for next-day rainfall prediction.
• To thoroughly assess the effects of different activation functions (Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, Swish) on model accuracy, convergence, stability, and generalization performance in the context of rainfall forecasting.

Study Configuration

• Spatial Scale: 20 major cities in the United States of America.
• Temporal Scale: Daily rainfall prediction for the next day, using meteorological observations from 2011 to 2023.

Methodology and Data

• Models used:
  • Deep Learning: Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM), Transformer (encoder-only architecture).
  • Machine Learning: Logistic Regression (LR), Support Vector Machines (SVM), K-Nearest Neighbor (KNN), Naïve Bayes, Passive Aggressive Classifier (PAC).
  • Activation Functions: Sigmoid, Tanh, Rectified Linear Unit (ReLU), Leaky ReLU, Exponential Linear Unit (ELU), Swish.
• Data sources: Publicly available Kaggle dataset containing daily meteorological observations from 20 major U.S. cities (2011-2023).
  • Input features: Temperature, Humidity, Wind Speed, Cloud Cover, Atmospheric Pressure, Precipitation.
  • Target variable: "Rain Tomorrow" (binary: 1 for rain, 0 for no rain).
  • Preprocessing: Handling missing values, Min-Max scaling for continuous variables, One-Hot encoding for categorical variables, Synthetic Minority Oversampling Technique (SMOTE) for class balancing on the training set.
  • Evaluation Protocol: 80:20 stratified train-test split, multi-seed evaluation (3 random seeds: 13, 37, 101) with 95% bootstrap confidence intervals, 5-fold cross-validation.
  • Performance Metrics: Accuracy, Precision, Recall, F1-score, Area Under the Receiver Operating Characteristic Curve (AUC-ROC).

Main Results

• Deep learning models consistently outperformed classical machine learning models for rainfall prediction.
• Classical ML models (Logistic Regression, SVM) achieved average accuracies of approximately 87%.
• LSTM: ReLU achieved the highest accuracy (0.996) and F1-score (0.991) but showed slight overfitting tendencies. Leaky ReLU provided the highest recall (1.000). Tanh offered a good balance of performance and generalization.
• BiLSTM: Leaky ReLU demonstrated the best performance with 0.994 accuracy, 0.975 precision, 0.999 recall, and 0.987 F1-score. ReLU showed similar high performance.
• Transformer: ELU and ReLU were the strongest in single-split tests (accuracy ~0.993, F1-score ~0.984). Swish and ReLU proved most robust in multi-seed evaluations (Swish accuracy 0.9695, AUC 0.9993; ReLU accuracy 0.9589, AUC 0.9988).
• Unsaturated activation functions (ReLU, Leaky ReLU, ELU, Swish) consistently outperformed saturated ones (Sigmoid, Tanh) across all deep learning architectures in terms of accuracy, stability, and generalization.
• "Precipitation" and "Humidity" were identified as the most influential meteorological features for next-day rainfall prediction.

Contributions

• Provides a comprehensive and unified comparative analysis of classical machine learning and state-of-the-art deep learning models (LSTM, BiLSTM, Transformer) under identical experimental conditions for next-day rainfall prediction.
• Systematically evaluates the impact of a wide range of activation functions (Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, Swish) and explicitly links their mathematical properties to empirical performance metrics and convergence dynamics.
• Introduces a novel dual-layered methodology combining SMOTE balancing with multi-seed robustness testing, providing more generalizable results and minimizing biases from single-split evaluations.
• Offers an activation-aware perspective in climate prediction research, highlighting the critical role of activation function selection for model stability and generalization.
• Provides practical recommendations for designing activation-sensitive and climatologically adaptive forecasting models, specifically identifying BiLSTM with Leaky ReLU/ELU and Transformer with ELU/ReLU/Swish as robust configurations.

Funding

• Open access funding provided by NTNU Norwegian University of Science and Technology (incl St. Olavs Hospital - Trondheim University Hospital).

Citation

@article{Farman2025Activation,
  author = {Farman, Hira and Hussain, Muhammad Arif and Hassan, Saif and Shaikh, Sarang and Ali, Khurshed},
  title = {Activation function impact on rainfall prediction: comparative insights across ML and DL architectures},
  journal = {Modeling Earth Systems and Environment},
  year = {2025},
  doi = {10.1007/s40808-025-02630-6},
  url = {https://doi.org/10.1007/s40808-025-02630-6}
}