research-article

Public Access

Scalable simulation of realistic volume fraction red blood cell flows through vascular networks

Authors:

Matthew J. Morse,

Abtin Rahimian,

Denis ZorinAuthors Info & Claims

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 6, Pages 1 - 30

https://doi.org/10.1145/3295500.3356203

Published: 17 November 2019 Publication History

Abstract

High-resolution blood flow simulations have potential for developing better understanding biophysical phenomena at the microscale, such as vasodilation, vasoconstriction and overall vascular resistance. To this end, we present a scalable platform for the simulation of red blood cell (RBC) flows through complex capillaries by modeling the physical system as a viscous fluid with immersed deformable particles. We describe a parallel boundary integral equation solver for general elliptic partial differential equations, which we apply to Stokes flow through blood vessels. We also detail a parallel collision avoiding algorithm to ensure RBCs and the blood vessel remain contact-free. We have scaled our code on Stampede2 at the Texas Advanced Computing Center up to 34,816 cores. Our largest simulation enforces a contact-free state between four billion surface elements and solves for three billion degrees of freedom on one million RBCs and a blood vessel composed from two million patches.

References

[1]

N. Al Quddus, W. A. Moussa, and S. Bhattacharjee. "Motion of a spherical particle in a cylindrical channel using arbitrary Lagrangian-Eulerian method". In: Journal of colloid and interface science 317.2 (2008), pp. 620--630.

[2]

S. Balay, S. Abhyankar, M. Adams, J. Brown, P. Brune, K. Buschelman, L. Dalcin, V. Eijkhout, W Gropp, D. Kaushik, et al. Petsc users manual revision 3.8. Tech. rep. Argonne National Lab.(ANL), Argonne, IL (United States), 2017.

[3]

P. Balogh and P. Bagchi. "A computational approach to modeling cellular-scale blood flow in complex geometry". In: Journal of Computational Physics 334 (2017), pp. 280--307.

Digital Library

[4]

P. Balogh and P. Bagchi. "Direct numerical simulation of cellular-scale blood flow in 3D microvascular networks". In: Biophysical journal 113.12 (2017), pp. 2815--2826.

[5]

H. H. Billett. "Hemoglobin and hematocrit". In: Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd edition. Butterworths, 1990.

[6]

O. P. Bruno and S. K. Lintner. "A high-order integral solver for scalar problems of diffraction by screens and apertures in three-dimensional space". In: Journal of Computational Physics 252 (2013), pp. 250--274.

Digital Library

[7]

C. Burstedde, L. C. Wilcox, and O. Ghattas. "p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees". In: SIAM Journal on Scientific Computing 33.3 (2011), pp. 1103--1133.

Digital Library

[8]

P. B. Canham. "The minimum energy of bending as a possible explanation of the biconcave shape of the human red blood cell". In: Journal of theoretical biology 26.1 (1970), pp. 61--81.

[9]

C. G. Caro, T. Pedley, R. Schroter, and W. Seed. The mechanics of the circulation. Cambridge University Press, 2012.

[10]

P. Du, J. Zhao, W. Cao, and Y. Wang. "DCCD: Distributed N-Body Rigid Continuous Collision Detection for Large-Scale Virtual Environments". In: Arabian Journal for Science and Engineering 42.8 (2017), pp. 3141--3147.

[11]

S. Fang. "A linearization method for generalized complementarity problems". In: IEEE transactions on automatic control 29.10 (1984), pp. 930--933.

[12]

J. B. Freund. "Numerical simulation of flowing blood cells". In: Annual review of fluid mechanics 46 (2014), pp. 67--95.

[13]

J. Gounley, M. Vardhan, and A. Randles. "A Computational Framework to Assess the Influence of Changes in Vascular Geometry on Blood Flow". In: Proceedings of the Platform for Advanced Scientific Computing Conference. ACM. 2017, p. 2.

Digital Library

[14]

I. G. Graham and I. H. Sloan. "Fully discrete spectral boundary integral methods for Helmholtz problems on smooth closed surfaces in R³". In: Numerische Mathematik 92.2 (2002), pp. 289--323.

[15]

L. Greengard and V. Rokhlin. "A fast algorithm for particle simulations". In: Journal of computational physics 73.2 (1987), pp. 325--348.

Digital Library

[16]

L. Grinberg, J. A. Insley, V. Morozov, M. E. Papka, G. E. Karniadakis, D. Fedosov, and K. Kumaran. "A new computational paradigm in multiscale simulations: Application to brain blood flow". In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM. 2011, p. 5.

Digital Library

[17]

D. Harmon, D. Panozzo, O. Sorkine, and D. Zorin. "Interference-aware geometric modeling". In: ACM Transactions on Graphics 30.6 (Dec. 2011), p. 1.

Digital Library

[18]

W. Helfrich. "Elastic properties of lipid bilayers: theory and possible experiments". In: Zeitschrift für Naturforschung C 28.11--12 (1973), pp. 693--703.

[19]

K. Iglberger and U. Rüde. "A Parallel Rigid Body Dynamics Algorithm". In: Euro-Par 2009 Parallel Processing. Ed. by H. Sips, D. Epema, and H.-X. Lin. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 760--771.

[20]

D. Kim, J.-P. Heo, J. Huh, J. Kim, and S.-e. Yoon. "HPCCD: Hybrid Parallel Continuous Collision Detection using CPUs and GPUs". In: Computer Graphics Forum (2009).

[21]

L. af Klinteberg and A.-K. Tornberg. "A fast integral equation method for solid particles in viscous flow using quadrature by expansion". In: Journal of Computational Physics 326 (2016), pp. 420--445.

[22]

A. Klöckner, A. Barnett, L. Greengard, and M. O'Neil. "Quadrature by expansion: A new method for the evaluation of layer potentials". In: Journal of Computational Physics 252 (2013), pp. 332--349.

Digital Library

[23]

F. Liu, T. Harada, Y. Lee, and Y. J. Kim. "Real-time Collision Culling of a Million Bodies on Graphics Processing Units". In: ACM SIGGRAPH Asia 2010 Papers. SIGGRAPH ASIA '10. Seoul, South Korea: ACM, 2010, 154:1--154:8.

Digital Library

[24]

L. Lu, A. Rahimian, and D. Zorin. "Contact-aware simulations of particulate Stokesian suspensions". In: Journal of Computational Physics 347C (Nov. 2017), pp. 160--182. arXiv: 1612.02057.

Digital Library

[25]

L. Lu, A. Rahimian, and D. Zorin. "Parallel contact-aware simulations of deformable particles in 3D Stokes flow". In: arXiv preprint arXiv:1812.04719 (2018).

[26]

D. Malhotra and G. Biros. "PVFMM: A Parallel Kernel Independent FMM for Particle and Volume Potentials". In: Communications in Computational Physics 18 (2015), pp. 808--830.

[27]

D. Malhotra and G. Biros. "Algorithm 967: A distributed-memory fast multipole method for volume potentials". In: ACM Transactions on Mathematical Software (TOMS) 43.2 (2016), p. 17.

Digital Library

[28]

D. Malhotra, A. Rahimian, D. Zorin, and G. Biros. "A parallel algorithm for long-timescale simulation of concentrated vesicle suspensions in three dimensions". In: (2017).

[29]

H. Mazhar, T. Heyn, and D. Negrut. "A scalable parallel method for large collision detection problems". In: 26 (June 2011), pp. 37--55.

[30]

E. Nazockdast, A. Rahimian, D. Zorin, and M. Shelley. "Fast and high-order methods for simulating fiber suspensions applied to cellular mechanics". preprint. 2015.

[31]

S. Pabst, A. Koch, and W. Strasser. "Fast and Scalable CPU/GPU Collision Detection for Rigid and Deformable Surfaces". In: Computer Graphics Forum (2010).

[32]

P. Perdikaris, L. Grinberg, and G. E. Karniadakis. "An effective fractal-tree closure model for simulating blood flow in large arterial networks". In: Annals of biomedical engineering 43.6 (2015), pp. 1432--1442.

[33]

P. Perdikaris, L. Grinberg, and G. E. Karniadakis. "Multiscale modeling and simulation of brain blood flow". In: Physics of Fluids 28.2 (2016), p. 021304.

[34]

M. Peyrounette, Y. Davit, M. Quintard, and S. Lorthois. "Multiscale modelling of blood flow in cerebral microcirculation: Details at capillary scale control accuracy at the level of the cortex". In: PloS one 13.1 (2018), e0189474.

[35]

H. Power and G. Miranda. "Second kind integral equation formulation of Stokes' flows past a particle of arbitrary shape". In: SIAM Journal on Applied Mathematics 47.4 (1987), p. 689.

Digital Library

[36]

C. Pozrikidis. Boundary integral and singularity methods for linearized viscous flow. Cambridge Texts in Applied Mathematics. Cambridge University Press, Cambridge, 1992.

[37]

A. Rahimian, I. Lashuk, S. Veerapaneni, A. Chandramowlishwaran, D. Malhotra, L. Moon, R. Sampath, A. Shringarpure, J. Vetter, R. Vuduc, et al. "Petascale direct numerical simulation of blood flow on 200k cores and heterogeneous architectures". In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2010, pp. 1--11.

Digital Library

[38]

A. Rahimian, S. K. Veerapaneni, D. Zorin, and G. Biros. "Boundary integral method for the flow of vesicles with viscosity contrast in three dimensions". In: Journal of Computational Physics 298 (2015), pp. 766--786.

Digital Library

[39]

A. Randles, E. W. Draeger, T. Oppelstrup, L. Krauss, and J. A. Gunnels. "Massively parallel models of the human circulatory system". In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM. 2015, p. 1.

Digital Library

[40]

D. Rossinelli, Y.-H. Tang, K. Lykov, D. Alexeev, M. Bernaschi, P. Hadjidoukas, M. Bisson, W. Joubert, C. Conti, G. Karniadakis, et al. "The in-silico lab-on-a-chip: petascale and high-throughput simulations of microfluidics at cell resolution". In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM. 2015, p. 2.

[41]

A. Saadat, C. J. Guido, and E. S. Shaqfeh. "Simulation of Red Blood Cell Migration in Small Arterioles: Effect of Cytoplasmic Viscosity". In: bioRxiv (2019), p. 572933.

[42]

A. Saadat, C. J. Guido, G. Iaccarino, and E. S. Shaqfeh. "Immersed-finite-element method for deformable particle suspensions in viscous and viscoelastic media". In: Physical Review E 98.6 (2018), p. 063316.

[43]

C. Sorgentone and A.-K. Tornberg. "A highly accurate boundary integral equation method for surfactant-laden drops in 3D". In: Journal of Computational Physics 360 (2018), pp. 167--191.

[44]

C. Sorgentone, A.-K. Tornberg, and P. M. Vlahovska. "A 3D boundary integral method for the electrohydrodynamics of surfactant-covered drops". In: Journal of Computational Physics (2019).

[45]

H. Sundar, D. Malhotra, and G. Biros. "HykSort: A New Variant of Hypercube Quicksort on Distributed Memory Architectures". In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing. ICS '13. Eugene, Oregon, USA: ACM, 2013, pp. 293--302.

Digital Library

[46]

L. N. Trefethen. Approximation theory and approximation practice. Vol. 128. Siam, 2013.

[47]

S. K. Veerapaneni, D. Gueyffier, G. Biros, and D. Zorin. "A numerical method for simulating the dynamics of 3D axisymmetric vesicles suspended in viscous flows". In: Journal of Computational Physics 228.19 (Apr. 2009), pp. 7233--7249.

Digital Library

[48]

S. K. Veerapaneni, A. Rahimian, G. Biros, and D. Zorin. "A fast algorithm for simulating vesicle flows in three dimensions". In: Journal of Computational Physics 230.14 (2011), pp. 5610--5634.

Digital Library

[49]

M. Wala and A. Klöckner. "A Fast Algorithm with Error Bounds for Quadrature by Expansion". In: arXiv preprint arXiv:1801.04070 (2018).

[50]

M. Wala and A. Klöckner. "Optimization of Fast Algorithms for Global Quadrature by Expansion Using Target-Specific Expansions". In: arXiv preprint arXiv:1811.01110 (2018).

[51]

W. Wang, T. G. Diacovo, J. Chen, J. B. Freund, and M. R. King. "Simulation of platelet, thrombus and erythrocyte hydrodynamic interactions in a 3D arteriole with in vivo comparison". In: PLoS One 8.10 (2013), e76949.

[52]

D. Xu, E. Kaliviotis, A. Munjiza, E. Avital, C. Ji, and J. Williams. "Large scale simulation of red blood cell aggregation in shear flows". In: Journal of Biomechanics 46.11 (2013), pp. 1810--1817.

[53]

W. Yan, H. Zhang, and M. J. Shelley. "Computing collision stress in assemblies of active spherocylinders: Applications of a fast and generic geometric method". In: The Journal of chemical physics 150.6 (2019), p. 064109.

[54]

T. Ye, L. Peng, and Y. Li. "Three-dimensional motion and deformation of a red blood cell in bifurcated microvessels". In: Journal of Applied Physics 123.6 (2018), p. 064701.

[55]

T. Ye, N. Phan-Thien, and C. T. Lim. "Particle-based simulations of red blood cells - A review". In: Journal of biomechanics 49.11 (2016), pp. 2255--2266.

[56]

T. Ye, N. Phan-Thien, C. T. Lim, L. Peng, and H. Shi. "Hybrid smoothed dissipative particle dynamics and immersed boundary method for simulation of red blood cells in flows". In: Physical Review E 95.6 (2017), p. 063314.

[57]

L. Ying, G. Biros, and D. Zorin. "A high-order 3D boundary integral equation solver for elliptic PDEs in smooth domains". In: Journal of Computational Physics 219.1 (2006), pp. 247--275.

Digital Library

[58]

L. Ying, G. Biros, and D. Zorin. "A high-order 3D boundary integral equation solver for elliptic PDEs in smooth domains". In: Journal of Computational Physics 219.1 (2006), pp. 247--275.

Digital Library

[59]

H. Zhao, A. H. Isfahani, L. N. Olson, and J. B. Freund. "A spectral boundary integral method for flowing blood cells". In: Journal of Computational Physics 229.10 (May 2010), pp. 3726--3744.

Digital Library

Cited By

Puthumana VChen PLeonetti MLasserre RJaeger M(2024)Assessment of coupled bilayer–cytoskeleton modelling strategy for red blood cell dynamics in flowJournal of Fluid Mechanics10.1017/jfm.2023.1092979Online publication date: 22-Jan-2024
https://doi.org/10.1017/jfm.2023.1092
Broms ABarnett ATornberg A(2024)Accurate close interactions of Stokes spheres using lubrication-adapted image systemsJournal of Computational Physics10.1016/j.jcp.2024.113636(113636)Online publication date: Nov-2024
https://doi.org/10.1016/j.jcp.2024.113636
Broms ATornberg A(2024)A barrier method for contact avoiding particles in Stokes flowJournal of Computational Physics10.1016/j.jcp.2023.112648497(112648)Online publication date: Jan-2024
https://doi.org/10.1016/j.jcp.2023.112648
Show More Cited By

Recommendations

Computational simulation of red blood cell motion in microvascular flows
Blood Rheology and Red Blood Cell Migration in Microchannel Flow
Rheology of red blood cell aggregation by computer simulation

The aggregation of red blood cells (RBC) induced by the interactions between RBCs is a dominant factor of the in vitro rheological properties of blood, and existing models of blood do not contain full cellular information. In this work, we introduce a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2019

1921 pages

ISBN:9781450362290

DOI:10.1145/3295500

General Chair:
Michela Taufer,
Program Chairs:
Pavan Balaji,
Antonio J. Peña

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SC '19

Sponsor:

SIGHPC

SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis

November 17 - 19, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
1,112
Total Downloads

Downloads (Last 12 months)222
Downloads (Last 6 weeks)34

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Puthumana VChen PLeonetti MLasserre RJaeger M(2024)Assessment of coupled bilayer–cytoskeleton modelling strategy for red blood cell dynamics in flowJournal of Fluid Mechanics10.1017/jfm.2023.1092979Online publication date: 22-Jan-2024
https://doi.org/10.1017/jfm.2023.1092
Broms ABarnett ATornberg A(2024)Accurate close interactions of Stokes spheres using lubrication-adapted image systemsJournal of Computational Physics10.1016/j.jcp.2024.113636(113636)Online publication date: Nov-2024
https://doi.org/10.1016/j.jcp.2024.113636
Broms ATornberg A(2024)A barrier method for contact avoiding particles in Stokes flowJournal of Computational Physics10.1016/j.jcp.2023.112648497(112648)Online publication date: Jan-2024
https://doi.org/10.1016/j.jcp.2023.112648
Triebold CBarber J(2024)The effect of the endothelial surface layer on cell–cell interactions in microvessel bifurcationsBiomechanics and Modeling in Mechanobiology10.1007/s10237-024-01863-123:5(1695-1721)Online publication date: 7-Jun-2024
https://doi.org/10.1007/s10237-024-01863-1
Roychowdhury SDraeger ERandles A(2023)Establishing metrics to quantify spatial similarity in spherical and red blood cell distributionsJournal of Computational Science10.1016/j.jocs.2023.10206071(102060)Online publication date: Jul-2023
https://doi.org/10.1016/j.jocs.2023.102060
Huet DWachs A(2023)A Cartesian-octree adaptive front-tracking solver for immersed biological capsules in large complex domainsJournal of Computational Physics10.1016/j.jcp.2023.112424(112424)Online publication date: Aug-2023
https://doi.org/10.1016/j.jcp.2023.112424
Roychowdhury SDraeger ERandles A(2022)Establishing Metrics to Quantify Underlying Structure in Vascular Red Blood Cell DistributionsComputational Science – ICCS 202210.1007/978-3-031-08751-6_7(89-102)Online publication date: 15-Jun-2022
https://doi.org/10.1007/978-3-031-08751-6_7
Akerkouch LLe T(2021)A Hybrid Continuum-Particle Approach for Fluid-Structure Interaction Simulation of Red Blood Cells in Fluid FlowsFluids10.3390/fluids60401396:4(139)Online publication date: 2-Apr-2021
https://doi.org/10.3390/fluids6040139
Wang BFerguson ZSchneider TJiang XAttene MPanozzo D(2021)A Large-scale Benchmark and an Inclusion-based Algorithm for Continuous Collision DetectionACM Transactions on Graphics10.1145/346077540:5(1-16)Online publication date: 24-Sep-2021
https://dl.acm.org/doi/10.1145/3460775
Ferguson ZLi MSchneider TGil-Ureta FLanglois TJiang CZorin DKaufman DPanozzo D(2021)Intersection-free rigid body dynamicsACM Transactions on Graphics10.1145/3450626.345980240:4(1-16)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459802
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten