Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

High-Performance Ptychographic Reconstruction with Federated Facilities

  • Conference paper
  • First Online:
Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation (SMC 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1512))

Included in the following conference series:

Abstract

Beamlines at synchrotron light source facilities are powerful scientific instruments used to image samples and observe phenomena at high spatial and temporal resolutions. Typically, these facilities are equipped only with modest compute resources for the analysis of generated experimental datasets. However, high data rate experiments can easily generate data in volumes that take days (or even weeks) to process on those local resources. To address this challenge, we present a system that unifies leadership computing and experimental facilities by enabling the automated establishment of data analysis pipelines that extend from edge data acquisition systems at synchrotron beamlines to remote computing facilities; under the covers, our system uses Globus Auth authentication to minimize user interaction, funcX to run user-defined functions on supercomputers, and Globus Flows to define and execute workflows. We describe the application of this system to ptychography, an ultra-high-resolution coherent diffraction imaging technique that can produce 100s of gigabytes to terabytes in a single experiment. When deployed on the DGX A100 ThetaGPU cluster at the Argonne Leadership Computing Facility and a microscopy beamline at the Advanced Photon Source, our system performs analysis as an experiment progresses to provide timely feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. APS Upgrade. https://www.aps.anl.gov/APS-Upgrade. Accessed May 2021

  2. Tike toolbox. https://tike.readthedocs.io/en/latest/

  3. Advanced Photon Source, Argonne National Laboratory: Research and Engineering Highlights, APS Science (2020). https://www.aps.anl.gov/Science/APS-Science. Accessed May 2021

  4. Allan, D., Caswell, T., Campbell, S., Rakitin, M.: Bluesky’s ahead: a multi-facility collaboration for an a la carte software project for data acquisition and management. Synchrotron Radiat. News 32(3), 19–22 (2019)

    Article  Google Scholar 

  5. Ananthakrishnan, R., et al.: Globus platform services for data publication. In: Proceedings of the Practice and Experience on Advanced Research Computing, pp. 1–7 (2018)

    Google Scholar 

  6. APS: Early science at the upgraded Advanced Photon Source. Technical report, Advanced Photon Source, Argonne National Laboratory (2015). https://www.aps.anl.gov/APS-Facility-Documents

  7. Argonne National Laboratory: Braid: Data Flow Automation for Scalable and FAIR Science. https://github.com/ANL-Braid/Flows. Accessed Jun 2021

  8. Argonne National Laboratory: Advanced Photon Source, An Office of Science National User Facility. https://www.aps.anl.gov. Accessed May 2021

  9. Aslan, S., Liu, Z., Nikitin, V., Bicer, T., Leyffer, S., Gursoy, D.: Distributed optimization with tunable learned priors for robust ptycho-tomography. arXiv preprint arXiv:2009.09498 (2020)

  10. Aslan, S., Nikitin, V., Ching, D.J., Bicer, T., Leyffer, S., Gürsoy, D.: Joint ptycho-tomography reconstruction through alternating direction method of multipliers. Opt. Express 27(6), 9128–9143 (2019)

    Article  Google Scholar 

  11. Basham, M., et al.: Data Analysis WorkbeNch (DAWN). J. Synchrotron Radiat. 22(3), 853–858 (2015). https://doi.org/10.1107/S1600577515002283

    Article  Google Scholar 

  12. Batey, D.J., et al.: X-ray ptychography with a laboratory source. Phys. Rev. Lett. 126(19), 193902 (2021)

    Article  Google Scholar 

  13. Ben-Nun, T., Gamblin, T., Hollman, D.S., Krishnan, H., Newburn, C.J.: Workflows are the new applications: challenges in performance, portability, and productivity. In: 2020 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pp. 57–69. IEEE Computer Society, Los Alamitos (2020). https://doi.org/10.1109/P3HPC51967.2020.00011

  14. Bicer, T., Gursoy, D., Kettimuthu, R., De Carlo, F., Agrawal, G., Foster, I.T.: Rapid tomographic image reconstruction via large-scale parallelization. In: Träff, J.L., Hunold, S., Versaci, F. (eds.) Euro-Par 2015. LNCS, vol. 9233, pp. 289–302. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-48096-0_23

    Chapter  Google Scholar 

  15. Bicer, T., et al.: Real-time data analysis and autonomous steering of synchrotron light source experiments. In: IEEE 13th International Conference on e-Science (e-Science), pp. 59–68. IEEE (2017)

    Google Scholar 

  16. Bicer, T., Gürsoy, D., Kettimuthu, R., De Carlo, F., Foster, I.T.: Optimization of tomographic reconstruction workflows on geographically distributed resources. J. Synchrotron Radiat. 23(4), 997–1005 (2016)

    Article  Google Scholar 

  17. Chang, H., Enfedaque, P., Marchesini, S.: Blind ptychographic phase retrieval via convergent alternating direction method of multipliers. SIAM J. Imag. Sci. 12(1), 153–185 (2019)

    Article  MathSciNet  Google Scholar 

  18. Chantzialexiou, G., Luckow, A., Jha, S.: Pilot-streaming: a stream processing framework for high-performance computing. In: 2018 IEEE 14th International Conference on e-Science (e-Science), pp. 177–188. IEEE (2018)

    Google Scholar 

  19. Chard, K., Tuecke, S., Foster, I.: Globus: recent enhancements and future plans. In: XSEDE16 Conference on Diversity, Big Data, and Science at Scale, pp. 1–8 (2016)

    Google Scholar 

  20. Chard, R., et al.: FuncX: a federated function serving fabric for science. In: 29th International Symposium on High-Performance Parallel and Distributed Computing, pp. 65–76 (2020)

    Google Scholar 

  21. Chen, Z., et al.: Electron ptychography achieves atomic-resolution limits set by lattice vibrations. Science 372(6544), 826–831 (2021). https://doi.org/10.1126/science.abg2533

    Article  Google Scholar 

  22. Computational Science Initiative, Brookhaven National Laboratory. https://www.bnl.gov/compsci/c3d/programs/NSLS.php. Accessed Jun 2021

  23. Deelman, E., et al.: Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46, 17–35 (2015). https://doi.org/10.1016/j.future.2014.10.008

    Article  Google Scholar 

  24. Deng, J., et al.: Simultaneous cryo X-ray ptychographic and fluorescence microscopy of green algae. Proc. Natl. Acad. Sci. 112(8), 2314–2319 (2015)

    Article  Google Scholar 

  25. Donatelli, J., et al.: Camera: the center for advanced mathematics for energy research applications. Synchrotron Radiat. News 28(2), 4–9 (2015)

    Article  MathSciNet  Google Scholar 

  26. Dong, Z., et al.: High-performance multi-mode ptychography reconstruction on distributed GPUs. In: 2018 New York Scientific Data Summit (NYSDS), pp. 1–5 (2018). https://doi.org/10.1109/NYSDS.2018.8538964

  27. Enders, B., et al.: Cross-facility science with the superfacility project at LBNL, pp. 1–7 (2020). https://doi.org/10.1109/XLOOP51963.2020.00006

  28. Enfedaque, P., Chang, H., Enders, B., Shapiro, D., Marchesini, S.: High performance partial coherent X-ray ptychography. In: Rodrigues, J.M.F., et al. (eds.) ICCS 2019. LNCS, vol. 11536, pp. 46–59. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22734-0_4

    Chapter  Google Scholar 

  29. Hammer, M., Yoshii, K., Miceli, A.: Strategies for on-chip digital data compression for X-ray pixel detectors. J. Instrum. 16(01), P01025 (2021)

    Article  Google Scholar 

  30. Hidayetoğlu, M., et al.: MemXCT: memory-centric X-ray CT reconstruction with massive parallelization. In: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–56 (2019)

    Google Scholar 

  31. Hidayetoglu, M., et al.: Petascale XCT: 3D image reconstruction with hierarchical communications on multi-GPU nodes. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 510–522. IEEE Computer Society (2020)

    Google Scholar 

  32. Holler, M., et al.: X-ray ptychographic computed tomography at 16 nm isotropic 3D resolution. Sci. Rep. 4(1), 1–5 (2014)

    Google Scholar 

  33. Huang, P., Du, M., Hammer, M., Miceli, A., Jacobsen, C.: Fast digital lossy compression for X-ray ptychographic data. J. Synchrotron Radiat. 28(1), 292–300 (2021)

    Article  Google Scholar 

  34. Intelligence Advanced Research Projects Activity: Rapid Analysis of Various Emerging Nanoelectronics. https://www.iarpa.gov/index.php/research-programs/raven. Accessed May 2021

  35. Jesse, S., et al.: Big data analytics for scanning transmission electron microscopy ptychography. Sci. Rep. 6(1), 1–8 (2016)

    Article  Google Scholar 

  36. Kalinin, S.V., et al.: Big, deep, and smart data in scanning probe microscopy (2016)

    Google Scholar 

  37. Klein, M., Martinasso, M., Leong, S.H., Alam, S.R.: Interactive supercomputing for experimental data-driven workflows. In: Juckeland, G., Chandrasekaran, S. (eds.) HUST/SE-HER/WIHPC - 2019. CCIS, vol. 1190, pp. 164–178. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44728-1_10

    Chapter  Google Scholar 

  38. Liu, Z., Bicer, T., Kettimuthu, R., Foster, I.: Deep learning accelerated light source experiments. In: 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS), pp. 20–28. IEEE (2019)

    Google Scholar 

  39. Liu, Z., Bicer, T., Kettimuthu, R., Gursoy, D., De Carlo, F., Foster, I.: TomoGAN: low-dose synchrotron X-ray tomography with generative adversarial networks: discussion. JOSA A 37(3), 422–434 (2020)

    Article  Google Scholar 

  40. Marchesini, S., et al.: SHARP: a distributed GPU-based ptychographic solver. J. Appl. Crystallogr. 49(4), 1245–1252 (2016)

    Article  Google Scholar 

  41. Nashed, Y.S.G., Vine, D.J., Peterka, T., Deng, J., Ross, R., Jacobsen, C.: Parallel ptychographic reconstruction. Opt. Express 22(26), 32082–32097 (2014)

    Article  Google Scholar 

  42. Naughton, T., et al.: Software framework for federated science instruments. In: Nichols, J., Verastegui, B., Maccabe, A.B., Hernandez, O., Parete-Koon, S., Ahearn, T. (eds.) SMC 2020. CCIS, vol. 1315, pp. 189–203. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63393-6_13

    Chapter  Google Scholar 

  43. Nikitin, V., et al.: Photon-limited ptychography of 3D objects via Bayesian reconstruction. OSA Continuum 2(10), 2948–2968 (2019)

    Article  Google Scholar 

  44. Ossyra, J.R., Sedova, A., Baker, M.B., Smith, J.C.: Highly interactive, steered scientific workflows on HPC systems: optimizing design solutions. In: Weiland, M., Juckeland, G., Alam, S., Jagode, H. (eds.) ISC High Performance 2019. LNCS, vol. 11887, pp. 514–527. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34356-9_39

    Chapter  Google Scholar 

  45. Pandolfi, R.J., et al.: Xi-cam: a versatile interface for data visualization and analysis. J. Synchrotron Radiat. 25(4), 1261–1270 (2018). https://doi.org/10.1107/S1600577518005787

    Article  Google Scholar 

  46. Peterka, T., Goodell, D., Ross, R., Shen, H.W., Thakur, R.: A configurable algorithm for parallel image-compositing applications. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, pp. 1–10. IEEE (2009)

    Google Scholar 

  47. Pfeiffer, F.: X-ray ptychography. Nat. Photonics 12(1), 9–17 (2018)

    Article  Google Scholar 

  48. Salim, M., Uram, T., Childers, J.T., Vishwanath, V., Papka, M.: Balsam: near real-time experimental data analysis on supercomputers. In: 2019 IEEE/ACM 1st Annual Workshop on Large-scale Experiment-in-the-Loop Computing (XLOOP), pp. 26–31 (2019). https://doi.org/10.1109/XLOOP49562.2019.00010

  49. Shapiro, D.A., et al.: Ptychographic imaging of nano-materials at the advanced light source with the nanosurveyor instrument. In: Journal of Physics: Conference Series, vol. 849, p. 012028. IOP Publishing (2017)

    Google Scholar 

  50. da Silva, R.F., et al.: Workflows community summit: bringing the scientific workflows community together. In: Workflows Community Summit: Bringing the Scientific Workflows Community Together (WorkflowsRI). Zenodo (2021)

    Google Scholar 

  51. Somnath, S., et al.: Feature extraction via similarity search: application to atom finding and denoising in electron and scanning probe microscopy imaging. Adv. Struct. Chem. Imag. 4(1), 1–10 (2018). https://doi.org/10.1186/s40679-018-0052-y

    Article  Google Scholar 

  52. Tuecke, S., et al.: Globus Auth: a research identity and access management platform. In: 12th International Conference on e-Science, pp. 203–212. IEEE (2016)

    Google Scholar 

  53. Turilli, M., Balasubramanian, V., Merzky, A., Paraskevakos, I., Jha, S.: Middleware building blocks for workflow systems. Comput. Sci. Eng. 21(4), 62–75 (2019)

    Article  Google Scholar 

  54. Ushizima, D.M., et al.: Ideal: images across domains, experiments, algorithms and learning. JOM 68(11), 2963–2972 (2016)

    Article  Google Scholar 

  55. Venkatakrishnan, S.V., et al.: Robust X-ray phase ptycho-tomography. IEEE Signal Process. Lett. 23(7), 944–948 (2016)

    Article  Google Scholar 

  56. Venkatakrishnan, S., Mohan, K.A., Ziabari, A.K., Bouman, C.A.: Algorithm-driven advances for scientific CT instruments: from model-based to deep learning-based approaches. arXiv preprint arXiv:2104.08228 (2021)

  57. Vine, D.J., et al.: Simultaneous X-ray fluorescence and ptychographic microscopy of Cyclotella meneghiniana. Opt. Express 20(16), 18287–18296 (2012)

    Article  Google Scholar 

  58. Wang, C., et al.: Deploying the Big Data Science Center at the Shanghai Synchrotron Radiation Facility: the first superfacility platform in China. Mach. Learn.: Sci. Technol. 2(3), 035003 (2021)

    Google Scholar 

  59. Wang, X., et al.: Consensus equilibrium framework for super-resolution and extreme-scale CT reconstruction. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3295500.3356142

  60. Wilke, R., et al.: Hard X-ray imaging of bacterial cells: nano-diffraction and ptychographic reconstruction. Opt. Express 20(17), 19232–19254 (2012)

    Article  Google Scholar 

  61. Wolstencroft, K., et al.: The Taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucl. Acids Res. 41, W557–W561 (2013). p. gkt328

    Google Scholar 

  62. Wu, Z., Bicer, T., Liu, Z., De Andrade, V., Zhu, Y., Foster, I.T.: Deep learning-based low-dose tomography reconstruction with hybrid-dose measurements. arXiv preprint arXiv:2009.13589 (2020)

  63. Yu, X., Bicer, T., Kettimuthu, R., Foster, I.: Topology-aware optimizations for multi-GPU ptychographic image reconstruction. In: International Conference on Supercomputing, ICS 2021, pp. 354–366. ACM (2021)

    Google Scholar 

  64. Yu, X., Nikitin, V., Ching, D.J., Aslan, S., Gursoy, D., Bicer, T.: Scalable and accurate multi-GPU based image reconstruction of large-scale ptychography data (2021)

    Google Scholar 

  65. Ziabari, A., et al.: Beam hardening artifact reduction in X-ray CT reconstruction of 3D printed metal parts leveraging deep learning and cad models. In: ASME International Mechanical Engineering Congress and Exposition, vol. 84492, p. V02BT02A043. American Society of Mechanical Engineers (2020)

    Google Scholar 

Download references

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Basic Energy Sciences and Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. This research used resources of the Argonne Leadership Computing Facility and Advanced Photon Source, which are U.S. Department of Energy (DOE) Office of Science User Facilities operated for the DOE Office of Science by Argonne National Laboratory under the same contract. Authors acknowledge ASCR funded Braid project at Argonne National Laboratory for the workflow system research and development activities. Authors also acknowledge Junjing Deng, Yudong Yao, Yi Jiang, Jeffrey Klug, Nick Sirica, and Jeff Nguyen from Argonne National Laboratory for providing the experimental data. This work is partially supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract D2019-1903270004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government.

Author information

Authors and Affiliations

  1. Data Science and Learning Division, CELS, Argonne National Laboratory, Lemont, IL, 60439, USA

    Tekin Bicer, Xiaodong Yu, Ryan Chard, Rajkumar Kettimuthu & Ian T. Foster

  2. Mathematics and Computer Science Division, CELS, Argonne National Laboratory, Lemont, IL, 60439, USA

    Bogdan Nicolae

  3. X-ray Science Division, APS, Argonne National Laboratory, Lemont, IL, 60439, USA

    Tekin Bicer, Daniel J. Ching & Mathew J. Cherukara

Authors
  1. Tekin Bicer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaodong Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel J. Ching
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ryan Chard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mathew J. Cherukara
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bogdan Nicolae
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rajkumar Kettimuthu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ian T. Foster
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tekin Bicer .

Editor information

Editors and Affiliations

  1. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Jeffrey Nichols

  2. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Arthur ‘Barney’ Maccabe

  3. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    James Nutaro

  4. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Swaroop Pophale

  5. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Pravallika Devineni

  6. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Theresa Ahearn

  7. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Becky Verastegui

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bicer, T. et al. (2022). High-Performance Ptychographic Reconstruction with Federated Facilities. In: Nichols, J., et al. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. SMC 2021. Communications in Computer and Information Science, vol 1512. Springer, Cham. https://doi.org/10.1007/978-3-030-96498-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-96498-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-96497-9

  • Online ISBN: 978-3-030-96498-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation