Surrogate model

The scientific challenge of surrogate modeling is the generation of a surrogate that is as accurate as possible, using as few simulation evaluations as possible. The process comprises three major steps which may be interleaved iteratively:

• Sample selection (also known as sequential design, optimal experimental design (OED) or active learning)
• Construction of the surrogate model and optimizing the model parameters (i.e., bias-variance tradeoff)
• Appraisal of the accuracy of the surrogate.

The accuracy of the surrogate depends on the number and location of samples (expensive experiments or simulations) in the design space. Various design of experiments (DOE) techniques cater to different sources of errors, in particular, errors due to noise in the data or errors due to an improper surrogate model.

Popular surrogate modeling approaches are: polynomial response surfaces; kriging; more generalized Bayesian approaches;^[1] gradient-enhanced kriging (GEK); radial basis function; support vector machines; space mapping;^[2] artificial neural networks and Bayesian networks.^[3] Other methods recently explored include Fourier surrogate modeling ^[4][5] and random forests.^[6]

For some problems, the nature of the true function is not known a priori, and therefore it is not clear which surrogate model will be the most accurate one. In addition, there is no consensus on how to obtain the most reliable estimates of the accuracy of a given surrogate. Many other problems have known physics properties. In these cases, physics-based surrogates such as space-mapping based models are commonly used.^[2][7]

Recently proposed comparison-based surrogate models (e.g., ranking support vector machines) for evolutionary algorithms, such as CMA-ES, allow preservation of some invariance properties of surrogate-assisted optimizers:^[8]

1. Invariance with respect to monotonic transformations of the function (scaling)
2. Invariance with respect to orthogonal transformations of the search space (rotation)

An important distinction can be made between two different applications of surrogate models: design optimization and design space approximation (also known as emulation).

In surrogate model-based optimization, an initial surrogate is constructed using some of the available budgets of expensive experiments and/or simulations. The remaining experiments/simulations are run for designs which the surrogate model predicts may have promising performance. The process usually takes the form of the following search/update procedure.

1. Initial sample selection (the experiments and/or simulations to be run)
2. Construct surrogate model
3. Search surrogate model (the model can be searched extensively, e.g., using a genetic algorithm, as it is cheap to evaluate)
4. Run and update experiment/simulation at new location(s) found by search and add to sample
5. Iterate steps 2 to 4 until out of time or design is "good enough"

Depending on the type of surrogate used and the complexity of the problem, the process may converge on a local or global optimum, or perhaps none at all.^[9]

In design space approximation, one is not interested in finding the optimal parameter vector, but rather in the global behavior of the system. Here the surrogate is tuned to mimic the underlying model as closely as needed over the complete design space. Such surrogates are a useful, cheap way to gain insight into the global behavior of the system. Optimization can still occur as a post-processing step, although with no update procedure (see above), the optimum found cannot be validated.

• Surrogate Modeling Toolbox (SMT: https://github.com/SMTorg/smt) is a Python package that contains a collection of surrogate modeling methods, sampling techniques, and benchmarking functions. This package provides a library of surrogate models that is simple to use and facilitates the implementation of additional methods. SMT is different from existing surrogate modeling libraries because of its emphasis on derivatives, including training derivatives used for gradient-enhanced modeling, prediction derivatives, and derivatives with respect to the training data. It also includes new surrogate models that are not available elsewhere: kriging by partial-least squares reduction and energy-minimizing spline interpolation.^[10]
• Surrogates.jl is a Julia packages which offers tools like random forests, radial basis methods and kriging.

• Linear approximation
• Response surface methodology
• Kriging
• Radial basis functions
• Gradient-enhanced kriging (GEK)
• OptiY
• Space mapping
• Surrogate endpoint
• Surrogate data
• Fitness approximation
• Computer experiment
• Conceptual model
• Bayesian regression
• Bayesian model selection

• Queipo, N.V., Haftka, R.T., Shyy, W., Goel, T., Vaidyanathan, R., Tucker, P.K. (2005), “Surrogate-based analysis and optimization,” Progress in Aerospace Sciences, 41, 1–28.
• D. Gorissen, I. Couckuyt, P. Demeester, T. Dhaene, K. Crombecq, (2010), “A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design," Journal of Machine Learning Research, Vol. 11, pp. 2051−2055, July 2010.
• T-Q. Pham, A. Kamusella, H. Neubert, “Auto-Extraction of Modelica Code from Finite Element Analysis or Measurement Data," 8th International Modelica Conference, 20–22 March 2011 in Dresden.
• Forrester, Alexander, Andras Sobester, and Andy Keane, Engineering design via surrogate modelling: a practical guide, John Wiley & Sons, 2008.
• Bouhlel, M. A. and Bartoli, N. and Otsmane, A. and Morlier, J. (2016) "Improving kriging surrogates of high-dimensional design models by Partial Least Squares dimension reduction", Structural and Multidisciplinary Optimization 53 (5), 935-952
• Bouhlel, M. A. and Bartoli, N. and Otsmane, A. and Morlier, J. (2016) "An improved approach for estimating the hyperparameters of the kriging model for high-dimensional problems through the partial least squares method", Mathematical Problems in Engineering

• Matlab code for surrogate modelling
• Matlab SUrrogate MOdeling Toolbox – Matlab SUMO Toolbox
• Surrogate Modeling Toolbox -- Python