Cited By
View all- Pradheep Kumar K(2024)Optical Neural Networks – A Strategy for Secure Quantum ComputingApplications and Techniques in Information Security10.1007/978-981-97-9743-1_2(23-32)Online publication date: 3-Nov-2024
LightRidge: An End-to-end Agile Design Framework for Diffractive Optical Neural Networks
Authors: 
Yingjie Li, Ruiyang Chen, Minhan Lou, Berardi Sensale-Rodriguez, Weilu Gao, Cunxi YuAuthors Info & Claims
Yingjie Li, Ruiyang Chen, Minhan Lou, Berardi Sensale-Rodriguez, Weilu Gao, Cunxi YuAuthors Info & ClaimsPages 202 - 218
Abstract
To lower the barrier to diffractive optical neural networks (DONNs) design, exploration, and deployment, we propose LightRidge, the first end-to-end optical ML compilation framework, which consists of (1) precise and differentiable optical physics kernels that enable complete explorations of DONNs architectures, (2) optical physics computation kernel acceleration that significantly reduces the runtime cost in training, emulation, and deployment of DONNs, and (3) versatile and flexible optical system modeling and user-friendly domain-specific-language (DSL). As a result, LightRidge framework enables efficient end-to-end design and deployment of DONNs, and significantly reduces the efforts for programming, hardware-software codesign, and chip integration. Our results are experimentally conducted with physical optical systems, where we demonstrate: (1) the optical physics kernels precisely correlated to low-level physics and systems, (2) significant speedups in runtime with physics-aware emulation workloads compared to the state-of-the-art commercial system, (3) effective architectural design space exploration verified by the hardware prototype and on-chip integration case study, and (4) novel DONN design principles including successful demonstrations of advanced image classification and image segmentation task using DONNs architecture and topology.
References
[1]
J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv:1607.06450, 2016.
[2]
A. L. Beam and I. S. Kohane, "Big data and machine learning in health care" Jama, vol. 319, no. 13, pp. 1317--1318, 2018.
[3]
S. Cao, W. Lu, and Q. Xu, "Deep neural networks for learning graph representations," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, 2016.
[4]
S. Cass, "Taking ai to the edge: Google's tpu now comes in a maker-friendly package," IEEE Spectrum, vol. 56, no. 5, pp. 16--17, 2019.
[5]
H. Chen, J. Feng, M. Jiang, Y. Wang, J. Lin, J. Tan, and P. Jin, "Diffractive deep neural networks at visible wavelengths," Engineering, vol. 7, no. 10, pp. 1483--1491, 2021.
[6]
R. Chen, Y. Li, M. Lou, J. Fan, Y. Tang, B. Sensale-Rodriguez, C. Yu, and W. Gao, "Physics-aware complex-valued adversarial machine learning in reconfigurable diffractive all-optical neural network," arXiv preprint arXiv:2203.06055, 2022.
[7]
R. Chen, Y. Li, M. Lou, C. Yu, and W. Gao, "Complex-valued reconfigurable diffractive optical neural networks using cost-effective spatial light modulators," in CLEO: Applications and Technology. Optica Publishing Group, 2022, pp. JTh3B-56.
[8]
T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha, "Deep learning for classical Japanese literature," arXiv preprint arXiv:1812.01718, 2018.
[9]
G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik, "Emnist: Extending mnist to handwritten letters," in 2017 international joint conference on neural networks (IJCNN). IEEE, 2017, pp. 2921--2926.
[10]
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The cityscapes dataset for semantic urban scene understanding," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213--3223.
[11]
P. Covington, J. Adams, and E. Sargin, "Deep neural networks for youtube recommendations," in Proceedings of the 10th ACM conference on recommender systems, 2016, pp. 191--198.
[12]
J. Dean, "1.1 the deep learning revolution and its implications for computer architecture and chip design," in 2020 IEEE International Solid-State Circuits Conference-(ISSCC). IEEE, 2020, pp. 8--14.
[13]
N. U. Dinc, J. Lim, E. Kakkava, C. Moser, and D. Psaltis, "Computer generated optical volume elements by additive manufacturing," Nanophotonics, vol. 9, no. 13, pp. 4173--4181, 2020.
[14]
N. U. Dinc, D. Psaltis, and D. Brunner, "Optical neural networks: the 3d connection," Photoniques, no. 104, pp. 34--38, 2020.
[15]
D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov, "Scalable object detection using deep neural networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 2147--2154.
[16]
E. Fathi and B. M. Shoja, "Deep neural networks for natural language processing," in Handbook of statistics. Elsevier, 2018, vol. 38, pp. 229--316.
[17]
J. Feldmann, N. Youngblood, M. Karpov, H. Gehring, X. Li, M. Stappers, M. Le Gallo, X. Fu, A. Lukashchuk, A. S. Raja et al., "Parallel convolutional processing using an integrated photonic tensor core," Nature, vol. 589, no. 7840, pp. 52--58, 2021.
[18]
W. Gao, C. Yu, and R. Chen, "Artificial intelligence accelerators based on graphene optoelectronic devices," Advanced Photonics Research, vol. 2, no. 6, p. 2100048, 2021.
[19]
H. Genc, S. Kim, A. Amid, A. Haj-Ali, V. Iyer, P. Prakash, J. Zhao, D. Grubb, H. Liew, H. Mao et al., "Gemmini: Enabling systematic deep-learning architecture evaluation via full-stack integration," in 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 2021, pp. 769--774.
[20]
E. Goi, X. Chen, Q. Zhang, B. P. Cumming, S. Schoenhardt, H. Luan, and M. Gu, "Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a cmos chip," Light: Science & Applications, vol. 10, no. 1, pp. 1--11, 2021.
[21]
J. W. Goodman, "Introduction to fourier optics. 3rd," Roberts and Company Publishers, 2005.
[22]
J. Gu, Z. Zhao, C. Feng, M. Liu, R. T. Chen, and D. Z. Pan, "Towards area-efficient optical neural networks: an fft-based architecture," in 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2020, pp. 476--481.
[23]
R. Hamerly, L. Bernstein, A. Sludds, M. Soljačić, and D. Englund, "Large-scale optical neural networks based on photoelectric multiplication," Physical Review X, vol. 9, no. 2, p. 021032, 2019.
[24]
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770--778.
[25]
E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," arXiv preprint arXiv:1611.01144, 2016.
[26]
N. P. Jouppi, C. Young, N. Patil, and D. Patterson, "A domain-specific architecture for deep neural networks," Communications of the ACM, vol. 61, no. 9, pp. 50--59, 2018.
[27]
D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
[28]
Y. LeCun, "The mnist database of handwritten digits," http://yann.lecun.com/exd-b/mnist/, 1998.
[29]
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436--444, 2015.
[30]
Y. Li, R. Chen, W. Gao, and C. Yu, "Physics-aware differentiable discrete codesign for diffractive optical neural networks," in Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, 2022, pp. 1--9.
[31]
Y. Li, R. Chen, B. Sensale-Rodriguez, W. Gao, and C. Yu, "Real-time multi-task diffractive deep neural networks via hardware-software co-design," Scientific reports, vol. 11, no. 1, pp. 1--9, 2021.
[32]
Y. Li, W. Gao, and C. Yu, "Rubik's optical neural networks: Multi-task learning with physics-aware rotation architecture," arXiv preprint arXiv:2304.12985, 2023.
[33]
Y. Li and C. Yu, "Late breaking results: physical adversarial attacks of diffractive deep neural networks," in DAC, 2021.
[34]
X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, "All-optical machine learning using diffractive deep neural networks," Science, vol. 361, no. 6406, pp. 1004--1008, 2018.
[35]
M. Lou, Y. Li, C. Yu, B. Sensale-Rodriguez, and W. Gao, "Effects of interlayer reflection and interpixel interaction in diffractive optical neural networks," Optics Letters, vol. 48, no. 2, pp. 219--222, 2023.
[36]
X. Luo, Y. Hu, X. Ou, X. Li, J. Lai, N. Liu, X. Cheng, A. Pan, and H. Duan, "Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible," Light: Science & Applications, vol. 11, no. 1, pp. 1--11, 2022.
[37]
J. W. Massey et al., "A comprehensive comparison of fft-accelerated integral equation methods vs. fdtd for bioelectromagnetics," Ph.D. dissertation, 2015.
[38]
D. Mengu, Y. Rivenson, and A. Ozcan, "Scale-, shift-, and rotation-invariant diffractive optical networks," ACS Photonics, vol. 8, no. 1, pp. 324--334, 2020.
[39]
D. Mengu, Y. Zhao, A. Tabassum, M. Jarrahi, and A. Ozcan, "Diffractive interconnects: All-optical permutation operation using diffractive networks," arXiv preprint arXiv:2206.10152, 2022.
[40]
R. Okuta, Y. Unno, D. Nishino, S. Hido, and C. Loomis, "Cupy: A numpy-compatible library for nvidia gpu calculations," in Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS), 2017. [Online]. Available: http://learningsys.org/nips17/assets/papers/paper_16.pdf
[41]
P. Prettenhofer and G. Louppe, "Gradient boosted regression trees in scikit-learn," 2014.
[42]
C. Qian, X. Lin, X. Lin, J. Xu, Y. Sun, E. Li, B. Zhang, and H. Chen, "Performing optical logic operations by a diffractive neural network," Light: Science & Applications, vol. 9, no. 1, pp. 1--7, 2020.
[43]
J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and S. Amarasinghe, "Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines," Acm Sigplan Notices, vol. 48, no. 6, pp. 519--530, 2013.
[44]
M. S. S. Rahman, J. Li, D. Mengu, Y. Rivenson, and A. Ozcan, "Ensemble learning of diffractive optical networks," Light: Science & Applications, vol. 10, no. 1, pp. 1--13, 2021.
[45]
N. Rao, "Beyond the cpu or gpu: Why enterprise-scale artificial intelligence requires a more holistic approach," See https://newsroom.intel.com/editorials/artificial-intelligence-requires-holistic-approach (accessed 5 November 2018), 2018.
[46]
B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernández-Lobato, G.-Y. Wei, and D. Brooks, "Minerva: Enabling low-power, highly-accurate deep neural network accelerators," in 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2016, pp. 267--278.
[47]
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter," arXiv preprint arXiv:1910.01108, 2019.
[48]
Y. Shen, N. C. Harris et al., "Deep learning with coherent nanophotonic circuits," Nature Photonics, vol. 11, no. 7, pp. 441--446, 2017.
[49]
W. A. Simon, Y. M. Qureshi, A. Levisse, M. Zapater, and D. Atienza, "Blade: A bitline accelerator for devices on the edge," in Proceedings of the 2019 on Great Lakes Symposium on VLSI, 2019, pp. 207--212.
[50]
E. Strubell, A. Ganesh, and A. McCallum, "Energy and policy considerations for deep learning in nlp," arXiv preprint arXiv:1906.02243, 2019.
[51]
C. Szegedy, A. Toshev, and D. Erhan, "Deep neural networks for object detection," Advances in neural information processing systems, vol. 26, 2013.
[52]
T. Tambe, C. Hooper, L. Pentecost, T. Jia, E.-Y. Yang, M. Donato, V. Sanh, P. Whatmough, A. M. Rush, D. Brooks et al., "Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference," in MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, 2021, pp. 830--844.
[53]
Y. Tang, P. T. Zamani, R. Chen, J. Ma, M. Qi, C. Yu, and W. Gao, "Device-system end-to-end design of photonic neuromorphic processor using reinforcement learning," Laser & Photonics Reviews, vol. 17, no. 2, p. 2200381, 2023.
[54]
M. S. Tobin, "Introduction to fourier optics," American Scientist, vol. 85, no. 6, pp. 581--584, 1997.
[55]
G. Vdovin, H. van Brug, and F. van Goor, "Lightpipes: Software for education in coherent optics," in https://github.com/opticspy/lightpipes, 2019.
[56]
M. Veettikazhy, A. K. Hansen, D. Marti, S. M. Jensen, A. L. Borre, E. R. Andresen, K. Dholakia, and P. E. Andersen, "Bpm-matlab: an open-source optical propagation simulation tool in matlab," Optics Express, vol. 29, no. 8, pp. 11 819--11 832, 2021.
[57]
M. Verhelst and B. Moons, "Embedded deep neural network processing: Algorithmic and processor techniques bring deep learning to iot and edge devices," IEEE Solid-State Circuits Magazine, vol. 9, no. 4, pp. 55--65, 2017.
[58]
C.-J. Wu, D. Brooks, K. Chen, D. Chen, S. Choudhury, M. Dukhan, K. Hazelwood, E. Isaac, Y. Jia, B. Jia et al., "Machine learning at facebook: Understanding inference at the edge," in 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2019, pp. 331--344.
[59]
H. Xiao, K. Rasul, and R. Vollgraf, "Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms," arXiv preprint arXiv:1708.07747, 2017.
[60]
X. Xu, M. Tan, B. Corcoran, J. Wu, A. Boes, T. G. Nguyen, S. T. Chu, B. E. Little, D. G. Hicks, R. Morandotti et al., "11 tops photonic convolutional accelerator for optical neural networks," Nature, vol. 589, no. 7840, pp. 44--51, 2021.
[61]
T. Yan, J. Wu, T. Zhou, H. Xie, F. Xu, J. Fan, L. Fang, X. Lin, and Q. Dai, "Fourier-space diffractive deep neural network," Physical review letters, vol. 123, no. 2, p. 023901, 2019.
[62]
A. Yazdanbakhsh, K. Seshadri, B. Akin, J. Laudon, and R. Narayanaswami, "An evaluation of edge tpu accelerators for convolutional neural networks," arXiv e-prints, pp. arXiv-2102, 2021.
[63]
K. Yee, "Numerical solution of initial boundary value problems involving maxwell's equations in isotropic media," IEEE Transactions on antennas and propagation, vol. 14, no. 3, pp. 302--307, 1966.
[64]
Z. Ying, C. Feng, Z. Zhao, S. Dhar, H. Dalir, J. Gu, Y. Cheng, R. Soref, D. Z. Pan, and R. T. Chen, "Electronic-photonic arithmetic logic unit for high-speed computing," Nature communications, vol. 11, no. 1, pp. 1--9, 2020.
[65]
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, "Places: A 10 million image database for scene recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[66]
S. Zhou, Y. Li, M. Lou, W. Gao, Z. Shi, C. Yu, and C. Ding, "Physics-aware roughness optimization for diffractive optical neural networks," arXiv preprint arXiv:2304.01500, 2023.
[67]
T. Zhou, L. Fang, T. Yan, J. Wu, Y. Li, J. Fan, H. Wu, X. Lin, and Q. Dai, "In situ optical backpropagation training of diffractive optical neural networks," Photonics Research, vol. 8, no. 6, pp. 940--953, 2020.
[68]
T. Zhou, X. Lin, J. Wu, Y. Chen, H. Xie, Y. Li, J. Fan, H. Wu, L. Fang, and Q. Dai, "Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit," Nature Photonics, vol. 15, no. 5, pp. 367--373, 2021.
Index Terms
- LightRidge: An End-to-end Agile Design Framework for Diffractive Optical Neural Networks
Index terms have been assigned to the content through auto-classification.
Recommendations
Optical Interconnect for High-End Computer Systems
Editor's note:Advances in silicon photonic technology have made possible the use of optical communication in large-scale chip arrays. This article shows how such a structure utilizes the high bandwidth of the optical links through optical proximity ...
Hardware-software co-design of slimmed optical neural networks
ASPDAC '19: Proceedings of the 24th Asia and South Pacific Design Automation ConferenceOptical neural network (ONN) is a neuromorphic computing hardware based on optical components. Since its first on-chip experimental demonstration, it has attracted more and more research interests due to the advantages of ultra-high speed inference with ...
Comments
Information & Contributors
Information
Published In
March 2023
430 pages
ISBN:9798400703942
DOI:10.1145/3623278
- Chair:
- Tor Aamodt,
- Program Chair:
- Michael M Swift,
- Program Co-chair:
- Natalie Enright Jerger
Copyright © 2023 Owner/Author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 February 2024
Check for updates
Badges
Qualifiers
- Research-article
Funding Sources
- NSF
Conference
ASPLOS '23: 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4
March 25 - 29, 2023
BC, Vancouver, Canada
Acceptance Rates
Overall Acceptance Rate 535 of 2,713 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 322Total Downloads
- Downloads (Last 12 months)322
- Downloads (Last 6 weeks)72
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
Cited By
View all- Pradheep Kumar K(2024)Optical Neural Networks – A Strategy for Secure Quantum ComputingApplications and Techniques in Information Security10.1007/978-981-97-9743-1_2(23-32)Online publication date: 3-Nov-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in