Article

Ultrascalable Implicit Finite Element Analyses in Solid Mechanics with over a Half a Billion Degrees of Freedom

Authors:

Harun H. Bayraktar,

Tony M. Keaveny,

Panayiotis PapadopoulosAuthors Info & Claims

SC '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing

Page 34

https://doi.org/10.1109/SC.2004.62

Published: 06 November 2004 Publication History

Abstract

The solution of elliptic diffusion operators is the computational bottleneck in many simulations in a wide range of engineering and scientific disciplines. We present a truly scalable-ultrascalable-algebraic multigrid (AMG) linear solver for the diffusion operator in unstructured elasticity problems. Scalability is demonstrated with speedup studies of a non-linear micro-finite element analyses of a human vertebral body with over a half of a billion degrees of freedom on up to 4088 processors on the ACSI White machine. This work is significant because in the domain of unstructured implicit finite element analysis in solid mechanics with complex geometry, this is the first demonstration of a highly parallel and efficient application of a mathematically optimal linear solution method on a common large scale computing platform - the IBM SP Power3.

References

[1]

{1} M.F. ADAMS, A distributed memory unstructured Gauss-Seidel algorithm for multigrid smoothers, in ACM/IEEE Proceedings of SC2001: High Performance Networking and Computing, Denver, Colorado, November 2001.

Digital Library

[2]

{2} M.F. ADAMS, Evaluation of three unstructured multigrid methods on 3D finite element problems in solid mechanics, International Journal for Numerical Methods in Engineering, 55 (2002), pp. 519-534.

[3]

{3} M.F. ADAMS, H. BAYRAKTAR, T. KEAVENY, AND P. PAPADOPOULOS, Applications of algebraic multigrid to large-scale finite element analysis of whole bone micro-mechanics on the IBM SP, in ACM/IEEE Proceedings of SC2003: High Performance Networking and Computing, 2003.

Digital Library

[4]

{4} M.F. ADAMS, M. BREZINA, J.J. HU, AND R.S. TUMINARO, Parallel multigrid smoothing: polynomial versus Gauss-Seidel, J. Comp. Phys., 188 (2003), pp. 593-610.

Digital Library

[5]

{5} S. BALAY, W.D. GROPP, L.C. MCINNES, AND B.F. SMITH, PETSc 2.0 users manual, tech. report, Argonne National Laboratory, 1996.

[6]

{6} R. BANK AND T. DUPONT, An optimal order process for solving finite element equations, Math. Comp., 36 (1981), pp. 35-51.

[7]

{7} H.H. BAYRAKTAR, M. ADAMS, P. HOFFMANN, D. LEE, A. GUPTA, P. PAPADOPOULOS, AND T. KEAVENY, Micromechanics of the human vertebral body, in Transactions of the Orthopaedic Research Society, vol. 29, San Francisco, 2004, p. 1129.

[8]

{8} H.H. BAYRAKTAR, M.F. ADAMS, A. GUPTA, P. PAPADOPOULOS, AND T.M. KEAVENY, The role of large deformations in trabecular bone mechanical behavior, in ASME Bioengineering Conference, Key Biscayne, FL, 2003, pp. 31-32.

[9]

{9} H. H. BAYRAKTAR, J.M. BUCKLEY, M.F. ADAMS, A. GUPTA, P.F. HOFFMANN, D.C. LEE, P. PAPADOPOULOS, AND T.M. KEAVENY, Cortical shell thickness and its contribution to vertebral body stifiness, in ASME Bioengineering Conference, Key Biscayne, FL, 2003, pp. 115-116.

[10]

{10} H.H. BAYRAKTAR, A. GUPTA, R. KWON, P. PAPADOPOULOS, AND T.M. KEAVENY, The modified super-ellipsoid yield criterion for human trabecular bone, Journal of Biomechanical Engineering, In Press (2004).

[11]

{11} H.H. BAYRAKTAR, E.F. MORGAN, G.L. NIEBUR, G. MORRIS, E. WONG, AND T.M. KEAVENY, Comparison of the elastic and yield properties of human femoral trabecular and cortical bone tissue, Journal of Biomechanics, 37 (2004), pp. 27-35.

[12]

{12} D. BRAESS, On the combination of the multigrid method and conjugate gradients, in Multigrid Methods II, W. Hackbusch and U. Trottenberg, eds., Berlin, 1986, Springer-Verlag, pp. 52-64.

[13]

{13} J. BRAMBLE, Multigrid methods, Longman Scientific and Technical, 1993.

[14]

{14} A. BRANDT, Multi-level adaptive solutions to boundary value problems, Math. Comput., 31 (1977), pp. 333-390.

[15]

{15} W.L. BRIGGS, V.E. HENSON, AND S. MCCORMICK, A multigrid tutorial, Second Edition, SIAM, Philadelphia, 2000.

Digital Library

[16]

{16} V.E. BULGAKOV AND G. KUHN, High-performance multilevel iterative aggregation solver for large finite-element structural analysis problems, International Journal for Numerical Methods in Engineering, 38 (1995), pp. 3529-3544.

[17]

{17} S.C. COWIN, ed., Bone Mechanics Handbook, CRC Press, Boca Raton, 2 ed., 2001.

[18]

{18} C.C. DOUGLAS, Multi-grid algorithms with applications to elliptic boundary-value problems, SIAM J. Numer. Anal., 21 (1984), pp. 236-254.

[19]

{19} S.C. EISENSTAT AND H.F. WALKER, Choosing the forcing terms in an inexact Newton method, SIAM J. Sci. Comput., 17 (1996), pp. 16-32.

Digital Library

[20]

{20} FEAP www.ce.berkeley.edu/~rlt.

[21]

{21} S. FORTUNE AND J. WYLLIE, Parallelism in random access machines, in ACM Symp. on Theory of Computing, 1978, pp. 114-118.

Digital Library

[22]

{22} R. GULDBERG, S. HOLLISTER, AND G. CHARRAS, The accuracy of digital image-based finite element models., Journal of Biomechanical Engineering, 120 (1998), pp. 289-295.

[23]

{23} W. HACKBUSCH, Multigrid methods and applications, vol. 4 of Computational Mathematics, Springer-Verlag, Berlin, 1985.

[24]

{24} J. HOMMINGA, B. VAN-RIETBERGEN, E.M. LOCHMULLER, H. WEINANS, F. ECKSTEIN, AND R. HUISKES, The osteoporotic vertebral structure is well adapted to the loads of daily life, but not to infrequent "error" loads, Bone, 34 (2004), pp. 510-516.

[25]

{25} T.J.R. HUGHES, R.M. FERENCZ, AND J.O. HALLQUIST, Large-scale vectorized implicit calculation in solid mechanics on a cray x-mp/48 utilizing ebe preconditioned conjugate gradients, Computer Methods in Applied Mechanics and Engineering, 61 (1987), pp. 215-248.

Digital Library

[26]

{26} G. KARYPIS AND V. KUMAR, Parallel multilevel k-way partitioning scheme for irregular graphs, ACM/IEEE Proceedings of SC1996: High Performance Networking and Computing, (1996).

Digital Library

[27]

{27} T.M. KEAVENY, Bone Mechanics Handbook, CRC Press, Boca Raton, 2 ed., 2001, ch. 16, pp. 1-42.

[28]

{28} C. KELLY, Iterative Methods for Linear and Nonlinear Equations, Frontiers in Applied Mathematics, SIAM, Philadelphia, 1995.

[29]

{29} M. M. BHARDWAJ, K. PIERSON, G. REESE, T. WALSH, D. DAY, K. ALVIN, AND J. PEERY, Salinas: A scalable software for high-performance structural and solid mechanics simulation, in ACM/IEEE Proceedings of SC2002: High Performance Networking and Computing, 2002. Gordon Bell Award.

Digital Library

[30]

{30} E.F. MORGAN, H.H. BAYRAKTAR, O.C. YEH, S. MAJUMDAR, A. BURGHARDT, AND T.M. KEAVENY, Contribution of inter-site variations in architecture to trabecular bone apparent yield strain, Journal of Biomechanics, In Press (2004).

[31]

{31} G.L. NIEBUR, M.J. FELDSTEIN, AND T.M. KEAVENY, Biaxial failure behavior of bovine tibial trabecular bone, Journal of Biomechanical Engineering, 124 (2002), pp. 699-705.

[32]

{32} G.L. NIEBUR, M.J. FELDSTEIN, J.C. YUEN, T.J. CHEN, AND T.M. KEAVENY, High-resolution finite element models with tissue strength asymmetry accurately predict failure of trabecular bone, Journal of Biomechanics, 33 (2000), pp. 1575-1583.

[33]

{33} G.L. NIEBUR, J.C. YUEN, A.C. HSIA, AND T.M. KEAVENY, Convergence behavior of high-resolution finite element models of trabecular bone, Journal of Biomechanical Engineering, 121 (1999), pp.629-635.

[34]

{34} G. POOLE, Y. LIU, AND J. MANDEL, Advancing analysis capabilities in ANSYS through solver technology, Electronic Transactions in Numerical Analysis, 15 (2003), pp. 106-121.

[35]

{35} P. RÜEGSEGGER, B. KOLLER, AND R. MÜLLER, A microtomographic system for the nondestructive evaluation of bone architecture, Calcified Tissue International, 58 (1996), pp. 24-29.

[36]

{36} J.S. STOLKEN AND J.H. KINNEY, On the importance of geometric nonlinearity in finite-element simulations of trabecular bone failure, Bone, 33 (2003), pp. 494-504.

[37]

{37} X. SUN AND L. Hi, Scalable problems and memory-bounded speedup, J. of Parallel and Distibuted Comp., 19 (1993), pp. 27-37.

Digital Library

[38]

{38} U. TROTTENBERG, C. OOSTERLEE, AND A. SCHÜLLER, Multigrid, Academic Press, London, 2001.

Digital Library

[39]

{39} D. ULRICH, B. VAN RIETBERGEN, A. LAIB, AND P. RÜEGSEGGER, The ability of three-dimensional structural indices to refiect mechanical aspects of trabecular bone., Bone, 25 (1999), pp. 55-60.

[40]

{40} D. ULRICH, B. VAN RIETBERGEN, H. WEINANS, AND P. RUEGSEGGER, Finite element analysis of trabecular bone structure: a comparison of image-based meshing techniques, Journal of Biomechanics, 31 (1998), pp. 1187-1192.

[41]

{41} B. VAN RIETBERGEN, R. HUISKES, F. ECKSTEIN, AND P. RUEGSEGGER, Trabecular bone tissue strains in the healthy and osteoporotic human femur, Journal of Bone Mineral Research, 18 (2003), pp. 1781-1788.

[42]

{42} B. VAN RIETBERGEN, H. WEINANS, R. HUISKES, AND A. ODGAARD, A new method to determine trabecular bone elastic properties and loading using micromechanical finite element models, Journal of Biomechanics, 28 (1995), pp. 69-81.

[43]

{43} P. VANEK, M. BREZINA, AND J. MANDEL, Convergence of algebraic multigrid based on smoothed aggregation, Numerische Mathematik, 88 (2001), pp. 559-579.

[44]

{44} P. VANEK, J. MANDEL, AND M. BREZINA, Algebraic multigrid by smoothed aggregation for second and fourth order elliptic problems, in 7th Copper Mountain Conference on Multigrid Methods, 1995.

Cited By

Yoda RBolten MNakajima KFujii A(2022)Assignment of idle processors to spatial redistributed domains on coarse levels in multigrid reduction in timeInternational Conference on High Performance Computing in Asia-Pacific Region10.1145/3492805.3492810(41-51)Online publication date: 7-Jan-2022
https://dl.acm.org/doi/10.1145/3492805.3492810
Jolivet PTournier PWest J(2016)Block iterative methods and recycling for improved scalability of linear solversProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3014904.3014927(1-14)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3014904.3014927
May DSanan PRupp KKnepley MSmith B(2016)Extreme-Scale Multigrid Components within PETScProceedings of the Platform for Advanced Scientific Computing Conference10.1145/2929908.2929913(1-12)Online publication date: 8-Jun-2016
https://dl.acm.org/doi/10.1145/2929908.2929913
Show More Cited By

Recommendations

Axisymmetric solid-of-revolution finite elements with rotational degrees of freedom

Two new solid-of-revolution axisymmetric finite elements, which account for hoop fibre rotations, are introduced. The first is based on an irreducible formulation, with only displacement and rotation fields assumed independently. The second element, ...
Numerical evaluation of discontinuous and nonconforming finite element methods in nonlinear solid mechanics

This work presents a systematic study of discontinuous and nonconforming finite element methods for linear elasticity, finite elasticity, and small strain plasticity. In particular, we consider new hybrid methods with additional degrees of freedom on ...
A $C^1$ Tetrahedral Finite Element without Edge Degrees of Freedom

A composite $C^1$ tetrahedral finite element is developed which does not have any edge degrees of freedom. This eliminates the need to associate a basis for the planes perpendicular to each edge; such a basis cannot depend continuously upon the edge ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing

November 2004

724 pages

ISBN:0769521533

General Chair:
Jeff Huskamp

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

IEEE Computer Society

United States

Publication History

Published: 06 November 2004

Check for updates

Qualifiers

Article

Conference

SC '04

Sponsor:

SIGARCH
IEEE-CS

SC '04: International Conference for High Performance Computing, Networking, Storage and Analysis

November 6 - 12, 2004

Acceptance Rates

SC '04 Paper Acceptance Rate 60 of 200 submissions, 30%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
31
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yoda RBolten MNakajima KFujii A(2022)Assignment of idle processors to spatial redistributed domains on coarse levels in multigrid reduction in timeInternational Conference on High Performance Computing in Asia-Pacific Region10.1145/3492805.3492810(41-51)Online publication date: 7-Jan-2022
https://dl.acm.org/doi/10.1145/3492805.3492810
Jolivet PTournier PWest J(2016)Block iterative methods and recycling for improved scalability of linear solversProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3014904.3014927(1-14)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3014904.3014927
May DSanan PRupp KKnepley MSmith B(2016)Extreme-Scale Multigrid Components within PETScProceedings of the Platform for Advanced Scientific Computing Conference10.1145/2929908.2929913(1-12)Online publication date: 8-Jun-2016
https://dl.acm.org/doi/10.1145/2929908.2929913
Sundar HBiros GBurstedde CRudi JGhattas OStadler GHollingsworth J(2012)Parallel geometric-algebraic multigrid on unstructured forests of octreesProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389055(1-11)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.5555/2388996.2389055
Flaig CArbenz P(2011)A scalable memory efficient multigrid solver for micro-finite element analyses based on CT imagesParallel Computing10.1016/j.parco.2011.08.00137:12(846-854)Online publication date: 1-Dec-2011
https://dl.acm.org/doi/10.1016/j.parco.2011.08.001
Kassiotis CIbrahimbegovic ANiekamp RMatthies H(2011)Nonlinear fluid---structure interaction problem. Part IIComputational Mechanics10.1007/s00466-010-0544-747:3(335-357)Online publication date: 1-Mar-2011
https://dl.acm.org/doi/10.1007/s00466-010-0544-7
Arbenz PFlaig C(2009)On Smoothing Surfaces in Voxel Based Finite Element Analysis of Trabecular BoneLarge-Scale Scientific Computing10.1007/978-3-540-78827-0_6(69-77)Online publication date: 26-Mar-2009
https://dl.acm.org/doi/10.1007/978-3-540-78827-0_6
Sampath RAdavani SSundar HLashuk IBiros GTeller P(2008)DendroProceedings of the 2008 ACM/IEEE conference on Supercomputing10.5555/1413370.1413389(1-12)Online publication date: 15-Nov-2008
https://dl.acm.org/doi/10.5555/1413370.1413389
Sundar HSampath RAdavani SDavatzikos CBiros GVerastegui B(2007)Low-constant parallel algorithms for finite element simulations using linear octreesProceedings of the 2007 ACM/IEEE conference on Supercomputing10.1145/1362622.1362656(1-12)Online publication date: 16-Nov-2007
https://dl.acm.org/doi/10.1145/1362622.1362656
Arbenz PVan Lenthe GMennel UMüller RSala M(2006)Multi-level µ-finite element analysis for human bone structuresProceedings of the 8th international conference on Applied parallel computing: state of the art in scientific computing10.5555/1775059.1775095(240-250)Online publication date: 18-Jun-2006
https://dl.acm.org/doi/10.5555/1775059.1775095
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten