Exploring potential of surrogate models

[EwoutH]

I came across the concept of surrogate models, and if sounds very interesting (and tricky), especially after having encountered a model with a runtime of an hour myself.

Especially this paper is promising:

• Angione, C., Silverman, E. S., & Yaneske, E. (2022). Using machine learning as a surrogate model for agent-based simulations. PLOS ONE, 17(2), e0263150. https://doi.org/10.1371/journal.pone.0263150

Discussion

We tested a range of machine learning tools towards replicating the outcome of a full ABM. Our results suggest that a deep learning approach (using multi-layered neural networks) is the most promising candidate to create a surrogate of the ABM. Neural networks can be trained on the data deriving from the ABM simulations, and can then replicate the ABM output with high accuracy. In general, we have shown that machine learning methods are able to approximate ABM predictions with high accuracy and dramatically reduced computational effort, and thus have great potential as a means for more efficient parameter calibration and sensitivity analysis.

GPT-4's take for some additional context:

Background

EMAworkbench is an invaluable tool for conducting exploratory modeling and analysis. It enables users to perform robust analyses with complex computational models. However, when dealing with high-dimensional or computationally expensive models, the analysis may be time-consuming or even infeasible.

Surrogate models, also known as metamodels or response surface models, provide an efficient alternative by approximating the original model with a simpler, computationally less expensive model. These surrogate models can be trained on a smaller set of data points and then used for analysis, thus reducing the computational burden.

Objective

The purpose of this issue is to explore the potential possibilities of incorporating surrogate models into EMAworkbench to improve its efficiency, scalability, and flexibility. We will consider various types of surrogate models and their applications, and evaluate the feasibility of integrating these models with the current EMAworkbench architecture.

Relevant Academic Papers

Several surrogate modeling techniques have been proposed in the literature, including but not limited to:

• Sacks, J., Welch, W. J., Mitchell, T. J., & Wynn, H. P. (1989). Design and analysis of computer experiments. Statistical Science, 4(4), 409-423. Link
• Forrester, A., Sobester, A., & Keane, A. (2008). Engineering design via surrogate modelling: a practical guide. John Wiley & Sons. Link
  Razavi, S., Tolson, B. A., & Burn, D. H. (2012). Review of surrogate modeling in water resources. Water Resources Research, 48(7). Link

Some popular surrogate modeling techniques are:

• Gaussian Process Regression (GPR)
• Radial Basis Function Networks (RBFN)
• Kriging
• Support Vector Regression (SVR)
• Artificial Neural Networks (ANN)
• Polynomial Regression
• Integration Approaches

Some possible integration approaches for incorporating surrogate models into EMAworkbench are:

1. Surrogate Model Creation: Provide functionality to create surrogate models from a set of training data generated by the original model.
2. Surrogate Model Evaluation: Enable users to evaluate the surrogate model performance using cross-validation or other validation techniques.
3. Surrogate Model Analysis: Allow users to conduct exploratory modeling and analysis using the surrogate models instead of the original models.
4. Surrogate Model Selection: Implement a model selection procedure to choose the best surrogate model based on performance metrics and user-defined criteria.

Discussion

Integration of surrogate models into EMAworkbench has the potential to significantly improve the efficiency of the analysis process, especially for computationally expensive models. The proposed integration approaches can provide users with the ability to create, evaluate, and use surrogate models for exploratory modeling and analysis. Furthermore, this issue aims to open a discussion on the most suitable surrogate modeling techniques and integration approaches for EMAworkbench.

We encourage the community to share their thoughts, experiences, and suggestions on this topic.