Data-Driven Polynomial Ridge Approximation Using Variable Projection
Pages A1566 - A1589
Abstract
Inexpensive surrogates are useful for reducing the cost of science and engineering studies involving large-scale, complex computational models with many input parameters. A ridge approximation is one class of surrogate that models a quantity of interest as a nonlinear function of a few linear combinations of the input parameters. When used in parameter studies (e.g., optimization or uncertainty quantification), ridge approximations allow the low-dimensional structure to be exploited, reducing the effective dimension. We introduce a new, fast algorithm for constructing a ridge approximation where the nonlinear function is a polynomial. This polynomial ridge approximation is chosen to minimize least squares mismatch between the surrogate and the quantity of interest on a given set of inputs. Naively, this would require optimizing both the polynomial coefficients and the linear combination of weights, the latter of which define a low-dimensional subspace of the input space. However, given a fixed subspace the optimal polynomial can be found by solving a linear least-squares problem. Hence using variable projection the polynomial can be implicitly defined, leaving an optimization problem over the subspace alone. Here we develop an algorithm that finds this polynomial ridge approximation by minimizing over the Grassmann manifold of low-dimensional subspaces using a Gauss--Newton method. Our Gauss--Newton method has superior theoretical guarantees and faster convergence on our numerical examples than the alternating approach for polynomial ridge approximation earlier proposed by Constantine, Eftekhari, Hokanson, and Ward [Comput. Methods Appl. Mech. Engrg., 326 (2017), pp. 402--421] that alternates between (i) optimizing the polynomial coefficients given the subspace and (ii) optimizing the subspace given the coefficients.
References
[1]
P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds, Princeton University Press, Princeton, NJ, 2008.
[2]
K. P. Adragni and R. D. Cook, Sufficient dimension reduction and prediction in regression, Philos. Trans. Roy. Soc. A, 367 (2009), pp. 4385--4405, https://doi.org/10.1098/rsta.2009.0110.
[3]
H. Akaike, A new look at the statistical model identification, IEEE Trans. Automat. Control, 19 (1974), pp. 716--723, https://doi.org/10.1109/TAC.1974.1100705.
[4]
M. Bauerheim, A. Ndiaye, P. Constantine, S. Moreau, and F. Nicoud, Symmetry breaking of azimuthal thermoacoustic modes: The UQ perspective, J. Fluid Mech., 789 (2016), pp. 534--566, https://doi.org/10.1017/jfm.2015.730.
[5]
\AA. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.
[6]
S. L. Campbell and C. D. Meyer, Jr., Generalized Inverses of Linear Transformations, Pitman, Boston, 1979.
[7]
A. Cohen, I. Daubechies, R. DeVore, G. Kerkyacharian, and D. Picard, Capturing ridge functions in high dimensions from point queries, Constr. Approx., 35 (2012), pp. 225--243, https://doi.org/10.1007/s00365-011-9147-6.
[8]
P. G. Constantine, Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies, SIAM, Philadelphia, 2015.
[9]
P. G. Constantine and A. Doostan, Time-dependent global sensitivity analysis with active subspaces for a lithium ion battery model, Stat. Anal. Data Min., 10 (2017), pp. 243--262, https://doi.org/10.1002/sam.11347.
[10]
P. G. Constantine, E. Dow, and Q. Wang, Active subspace methods in theory and practice: Applications to kriging surfaces, SIAM J. Sci. Comput., 36 (2014), pp. A1500--A1524, https://doi.org/10.1137/130916138.
[11]
P. G. Constantine, A. Eftekhari, J. Hokanson, and R. A. Ward, A near-stationary subspace for ridge approximation, Comput. Methods Appl. Mech. Engrg., 326 (2017), pp. 402--421, https://doi.org/10.1016/j.cma.2017.07.038.
[12]
P. G. Constantine, M. Emory, J. Larsson, and G. Iaccarino, Exploiting active subspaces to quantify uncertainty in the numerical simulation of the HyShot II scramjet, J. Comput. Phys., 302 (2015), pp. 1--20, https://doi.org/10.1016/j.jcp.2015.09.001.
[13]
P. G. Constantine, B. Zaharatos, and M. Campanelli, Discovering an active subspace in a single-diode solar cell model, Stat. Anal. Data Min., 8 (2015), pp. 264--273, https://doi.org/10.1002/sam.11281.
[14]
R. D. Cook, Regression Graphics: Ideas for Studying Regressions Through Graphics, John Wiley and Sons, Hoboken, 1998, https://doi.org/10.1002/9780470316931.
[15]
P. Diaconis and M. Shahshahani, On nonlinear functions of linear combinations, SIAM J. Sci. Statist. Comput., 5 (1984), pp. 175--191, https://doi.org/10.1137/0905013.
[16]
T. D. Economon, F. Palacios, S. R. Copeland, T. W. Lukaczyk, and J. J. Alonso, SU2: An open-source suite for multiphysics simulation and design, AIAA J., 54 (2016), pp. 828--846, https://doi.org/10.2514/1.J053813.
[17]
A. Edelman, T. A. Arias, and S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl., 20 (1998), pp. 303--353, https://doi.org/10.1137/S0895479895290954.
[18]
M. Fornasier, K. Schnass, and J. Vybiral, Learning functions of few arbitrary linear parameters in high dimensions, Found. Comput. Math., 12 (2012), pp. 229--262, https://doi.org/10.1007/s10208-012-9115-y.
[19]
J. H. Friedman and W. Stuetzle, Projection pursuit regression, J. Amer. Statist. Assoc., 76 (1981), pp. 817--823, http://www.jstor.org/stable/2287576.
[20]
K. Fukumizu and C. Leng, Gradient-based kernel dimension reduction for regression, J. Amer. Statist. Assoc., 109 (2014), pp. 359--370, https://doi.org/10.1080/01621459.2013.838167.
[21]
J. M. Gilbert, J. L. Jefferson, P. G. Constantine, and R. M. Maxwell, Global spatial sensitivity of runoff to subsurface permeability using the active subspace method, Adv. Water Res., 92 (2016), pp. 30--42, https://doi.org/10.1016/j.advwatres.2016.03.020.
[22]
A. Glaws, P. G. Constantine, J. N. Shadid, and T. M. Wildey, Dimension reduction in magnetohydrodynamics power generation models: Dimensional analysis and active subspaces, Stat. Anal. Data Min., 10 (2017), pp. 312--325, https://doi.org/10.1002/sam.11355.
[23]
G. H. Golub and V. Pereyra, The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate, SIAM J. Numer. Anal., 10 (1973), pp. 413--432.
[24]
J. Hampton and A. Doostan, Coherence motivated sampling and convergence analysis of least squares polynomial chaos regression, Comput. Methods Appl. Mech. Engrg., 290 (2015), pp. 73--97, https://doi.org/10.1016/j.cma.2015.02.006.
[25]
T. Hastie, R. J. Tibshirani, and J. Friedman, Elements of Statistical Learning, 2nd ed., Springer Science+Business Media, New York, 2009, https://doi.org/10.1007/978-0-387-84858-7.
[26]
N. J. Higham, Accuracy and Stability of Numerical Algorithms, SIAM, Philadelphia, 2002.
[27]
X. Hu, G. T. Parks, X. Chen, and P. Seshadri, Discovering a one-dimensional active subspace to quantify multidisciplinary uncertainty in satellite system design, Adv. Space Res., 57 (2016), pp. 1268--1279, https://doi.org/10.1016/j.asr.2015.11.001.
[28]
P. J. Huber, Projection pursuit, Ann. Statist., 13 (1985), pp. 435--475, http://www.jstor.org/stable/2241175.
[29]
J. L. Jefferson, J. M. Gilbert, P. G. Constantine, and R. M. Maxwell, Active subspaces for sensitivity analysis and dimension reduction of an integrated hydrologic model, Comput. Geosci., 83 (2015), pp. 127--138, https://doi.org/10.1016/j.cageo.2015.11.002.
[30]
J. L. Jefferson, R. M. Maxwell, and P. G. Constantine, Exploring the sensitivity of photosynthesis and stomatal resistance parameters in a land surface model, J. Hydrometeorol., 18 (2017), pp. 897--915, https://doi.org/10.1175/JHM-D-16-0053.1.
[31]
D. Jones, A taxonomy of global optimization methods based on response surfaces, J. Global Optim., 21 (2001), pp. 345--383, https://doi.org/10.1023/A:1012771025575.
[32]
X. Liu and S. Guillas, Dimension reduction for Gaussian process emulation: An application to the influence of bathymetry on tsunami heights, SIAM/ASA J. Uncertain. Quantif., 5 (2017), pp. 787--812, https://doi.org/10.1137/16M1090648.
[33]
T. Loudon and S. Pankavich, Mathematical analysis and dynamic active subspaces for a long term model of HIV, Math. Biosci. Eng., 14 (2017), pp. 709--733, https://doi.org/10.3934/mbe.2017040.
[34]
T. W. Lukaczyk, Surrogate Modeling and Active Subspaces for Efficient Optimization of Supersonic Aircraft, Ph.D. thesis, Stanford University, 2015, https://purl.stanford.edu/xx611nd3190.
[35]
T. W. Lukaczyk, P. Constantine, F. Palacios, and J. J. Alonso, Active subspaces for shape optimization, in 10th AIAA Multidisciplinary Design Optimization Conference, National Harbor, MD, AIAA SciTech Forum, 2014, https://doi.org/10.2514/6.2014-1171.
[36]
J. J. Moré and S. M. Wild, Estimating computational noise, SIAM J. Sci. Comput., 33 (2011), pp. 1292--1314.
[37]
R. H. Myers and D. C. Montgomery, Response Surface Methodology: Process and Product Optimization Using Designed Experiments, John Wiley and Sons, New York, 1995.
[38]
J. Nocedal and S. J. Wright, Numerical Optimization, Springer, New York, 2006.
[39]
C. Othmer, T. W. Lukaczyk, P. Constantine, and J. J. Alonso, On active subspaces in car aerodynamics, in 17th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Washington, DC, AIAA AVIATION Forum, 2016, https://doi.org/10.2514/6.2016-4294.
[40]
V. Y. Pan, How bad are Vandermonde matrices?, SIAM J. Matrix Anal. Appl., 37 (2016), pp. 676--694.
[41]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python, J. Mach. Learn. Res., 12 (2011), pp. 2825--2830.
[42]
R. R. Picard and R. D. Cook, Cross-validation of regression models, J. Amer. Statist. Assoc., 79 (1984), pp. 575--583, https://doi.org/10.1080/01621459.1984.10478083.
[43]
A. Pinkus, Ridge Functions, Cambridge University Press, Cambridge, 2015, https://doi.org/10.1017/CBO9781316408124.
[44]
C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning, The MIT Press, Boston, 2006, http://www.gaussianprocess.org/gpml/.
[45]
A. Ruhe and P. A. Wedin, Algorithms for separable nonlinear least squares problems, SIAM Rev., 22 (1980), pp. 318--337, https://doi.org/10.1137/1022057.
[46]
T. J. Santner, B. J. Williams, and W. I. Notz, The Design and Analysis of Computer Experiments, Springer, New York, 2003, https://doi.org/10.1007/978-1-4757-3799-8.
[47]
P. Seshadri, S. Shahpar, P. Constantine, G. Parks, and M. Adams, Turbomachinery active subspace performance maps, in ASME Turbo Expo 2017: Turbomachinery Technical Conference and Exposition, no. 50787, Charlotte, NC, International Gas Turbine Institute, 2017, p. V02AT39A034, https://doi.org/10.1115/GT2017-64528.
[48]
S. Shan and G. G. Wang, Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions, Struct. Multidiscip. Optim., 41 (2010), pp. 291--241, https://doi.org/10.1007/s00158-009-0420-2.
[49]
R. C. Smith, Uncertainty Quantification: Theory, Implementation, and Applications, SIAM, Philadelphia, 2013.
[50]
T. J. Sullivan, Introduction to Uncertainty Quantification, Springer, New York, 2015, https://doi.org/10.1007/978-3-319-23395-6.
[51]
J. Townsend, N. Koep, and S. Weichwald, Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation, J. Mach. Learn. Res., 17 (2016), pp. 1--5, http://jmlr.org/papers/v17/16-177.html.
[52]
R. Tripathy, I. Bilionis, and M. Gonzalez, Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation, J. Comput. Phys., 321 (2016), pp. 191--223, https://doi.org/10.1016/j.jcp.2016.05.039.
[53]
H. Tyagi and V. Cevher, Learning non-parametric basis independent models from point queries via low-rank methods, Appl. Comput. Harmon. Anal., 37 (2014), pp. 389--412, https://doi.org/10.1016/j.acha.2014.01.002.
[54]
F. Vivarelli and C. K. I. Williams, Discovering hidden features with gaussian processes regression, in Advances in Neural Information Processing Systems 11, M. J. Kearns, S. A. Solla, and D. A. Cohn, eds., MIT Press, Cambridge, MA, 1999, pp. 613--619, http://papers.nips.cc/paper/1579-discovering-hidden-features-with-gaussian-processes-regression.pdf.
[55]
G. G. Wang and S. Shan, Review of metamodeling techniques in support of engineering design optimization, J. Mech. Design, 129 (2006), pp. 370--380, https://doi.org/10.1115/1.2429697.
[56]
S. Weisberg, Applied Linear Regression, 3rd ed., John Wiley and Sons, Hoboken, 2005, https://doi.org/https://doi.org/10.1002/0471704091.
[57]
Y.-C. Wong, Differential geometry of Grassmann manifolds, Proc. Nat. Acad. Sci. USA, 57 (1967), pp. 589--594, https://doi.org/10.1073/pnas.57.3.589.
[58]
Y. Xia, A multiple-index model and dimension reduction, J. Amer. Statist. Assoc., 103 (2008), pp. 1631--1640, http://www.jstor.org/stable/27640210.
[59]
Y. Xia, H. Tong, W. K. Li, and L.-X. Zhu, An adaptive estimation of dimension reduction space, J. R. Stat. Soc. Ser. B. Stat. Methodol., 64 (2002), pp. 363--410, https://doi.org/10.1111/1467-9868.03411.
Recommendations
Convergence of the Gauss-Newton method for convex composite optimization problems under majorant condition on Riemannian manifolds
AbstractIn this paper, we consider convex composite optimization problems on Riemannian manifolds, and discuss the semi-local convergence of the Gauss-Newton method with quasi-regular initial point and under the majorant condition. As special cases, we ...
Highlights- Study of convex composite optimization problems on Riemannian manifolds.
- Convergence of the Gauss-Newton method under the majorant condition.
- Special cases are studied under (i) Lipschitz-type condition, (ii) γ-condition.
A New Descent Method for Symmetric Non-monotone Variational Inequalities with Application to Eigenvalue Complementarity Problems
In this paper, a modified Josephy---Newton direction is presented for solving the symmetric non-monotone variational inequality. The direction is a suitable descent direction for the regularized gap function. In fact, this new descent direction is ...
Direct Discriminant Locality Preserving Projection With Hammerstein Polynomial Expansion
Discriminant locality preserving projection (DLPP) is a linear approach that encodes discriminant information into the objective of locality preserving projection and improves its classification ability. To enhance the nonlinear description ability of ...
Comments
Information & Contributors
Information
Published In
© 2018, Society for Industrial and Applied Mathematics.
Publisher
Society for Industrial and Applied Mathematics
United States
Publication History
Published: 01 January 2018
Author Tags
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 26 Jan 2025