Towards a Task-agnostic Distillation Methodology for Creating Edge Foundation Models
Authors: 
Swarnava Dey, Arijit Mukherjee, 
Arijit Ukil, and 
Arpan PalAuthors Info & Claims
Swarnava Dey, Arijit Mukherjee, Arijit Ukil, and Arpan PalAuthors Info & ClaimsJune 2024
Pages 10 - 15
Abstract
In recent years, AI has undergone significant changes. Firstly, there is a growing recognition of the need to deploy inference models based on Deep Neural Networks (DNNs) on edge devices. Secondly, there is an increasing demand for low-energy inferencing and continuous online learning, particularly in dynamic environments. Thirdly, foundation models, trained on broad datasets for diverse applications, are gaining prominence. In closed-loop systems like robotics, there is a need to use foundation models at the edge due to practical constraints in training new models for every environment or data type. This article addresses issues in current edge computing scenarios and proposes Edge Foundation models as a solution. We introduce a task-agnostic distillation method for generating compact yet generalized models and present preliminary proof-of-concept results, demonstrating the potential of Edge Foundation models to accelerate Edge AI adoption.
References
[1]
Modassir Afzal. 2023. ResNet-9. https://tinyurl.com/5cn97rew
[2]
Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning Representations by Maximizing Mutual Information Across Views. arXiv preprint arXiv:1906.00910 (2019).
[3]
Brian Bailey. 2022. AI Power Consumption Exploding. https://semiengineering.com/ai-power- consumption- exploding/.
[4]
Jane Bromley et al. 1993. Signature Verification Using a "Siamese" Time Delay Neural Network. In NeurIPS (Denver, Colorado). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 737--744.
[5]
Keyan Cao, Yefan Liu, Gongjie Meng, and Qimeng Sun. 2020. An overview on edge computing research. IEEE access 8 (2020), 85714--85728.
[6]
Mathilde Caron et al. 2021. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. arXiv preprint arXiv:2006.09882 (2021).
[7]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709 (2020).
[8]
Xinlei Chen et al. 2020. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.04297 (2020).
[9]
Xinlei Chen and Kaiming He. 2020. Exploring Simple Siamese Representation Learning. arXiv preprint arXiv:2011.10566 (2020).
[10]
Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang, Yin Fang, Jeff Z Pan, and Huajun Chen. 2023. Duet: Cross-modal semantic grounding for contrastive zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 405--413.
[11]
Po-Yung Chou, Yu-Yung Kao, and Cheng-Hung Lin. 2023. Fine-grained Visual Classification with High-temperature Refinement and Background Suppression. arXiv preprint arXiv:2303.06442 (2023).
[12]
Semiconductor Research Corporation. 2022. Decadal Plan for Semiconductors. https://www.src.org/about/decadal-plan/decadal-plan-full-report.pdf.
[13]
Swarnava Dey, Avik Ghose, and Soumik Das. 2023. Challenges of Accurate and Efficient AutoML. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, Los Alamitos, CA, USA, 1834--1839.
[14]
Swarnava Dey, Jayeeta Mondal, and Arijit Mukherjee. 2019. Offloaded Execution of Deep Learning Inference at Edge: Challenges and Insights. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE Computer Society, Los Alamitos, CA, USA, 855--861.
[15]
Swarnava Dey, Arijit Mukherjee, Arpan Pal, and P. Balamuralidhar. 2018. Partitioning of CNN Models for Execution on Fog Devices. In Proceedings of the 1st ACM International Workshop on Smart Cities and Fog Computing (Shenzhen, China) (CitiFog'18). ACM, New York, NY, USA, 19--24.
[16]
Swarnava Dey, Arijit Mukherjee, Arpan Pal, and Balamuralidhar P. 2019. Embedded Deep Inference in Practice: Case for Model Partitioning. In Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems (New York, NY, USA) (SenSys-ML 2019). Association for Computing Machinery, New York, NY, USA, 25--30. https://doi.org/10.1145/3362743.3362964
[17]
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2022. CLAP: Learning Audio Concepts From Natural Language Supervision. arXiv preprint arXiv 2206.04769 (2022).
[18]
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. 2021. SEED: Self-supervised Distillation For Visual Representation. arXiv preprint arXiv:2101.04731 (2021).
[19]
Jean-Bastien Grill et al. 2020. Bootstrap your own latent: A new approach to self-supervised Learning. arXiv preprint arXiv:2006.07733 (2020).
[20]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv preprint arXiv:1911.05722 (2020).
[21]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
[22]
R Devon Hjelm et al. 2019. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2019).
[23]
Olivier J. Hénaff et al. 2020. Data-Efficient Image Recognition with Contrastive Predictive Coding. arXiv preprint arXiv:1905.09272 (2020).
[24]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023).
[25]
Mustafa Taha Kocyigit et al. 2023. Accelerating Self-Supervised Learning via Efficient Training Strategies. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision 2023 (WACV) (IEEE Workshop on Applications of Computer Vision (WACV)). 5643--5653. https://doi.org/10.1109/WACV56688.2023.00561
[26]
Ishan Misra and Laurens van der Maaten. 2019. Self-Supervised Learning of Pretext-Invariant Representations. arXiv preprint arXiv:1912.01991 (2019).
[27]
Arijit Mukherjee, Jayeeta Mondal, and Swarnava Dey. 2022. Accelerated Fire Detection and Localization at Edge. ACM Trans. Embed. Comput. Syst. 21, 6, Article 70 (oct 2022), 27 pages.
[28]
Shalini Mukhopadhyay, Swarnava Dey, and Avik Ghose. 2023. Demo: On-Device Puff Detection System for Smoking Cessation. In MobiSys (Helsinki, Finland). ACM, New York, NY, USA, 586--587.
[29]
Shalini Mukhopadhyay, Swarnava Dey, and Avik Ghose. 2023. TinyPuff: Automated Design of Tiny Smoking Puff Classifiers for Body Worn Devices. In Proceedings of the 8th Workshop on Body-Centric Computing Systems (BodySys '23). ACM, New York, NY, USA, 7--12.
[30]
Shalini Mukhopadhyay, Swarnava Dey, Avik Ghose, Pragya Singh, and Pallab Dasgupta. 2023. Generating Tiny Deep Neural Networks for ECG Classification on Micro-Controllers. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). IEEE, USA, 392--397.
[31]
Shalini Mukhopadhyay, Swarnava Dey, Avik Ghose, and Aakash Tyagi. 2023. Automated Generation of Tiny Model for Real-Time ECG Classification on Tiny Edge Devices. In SenSys (Boston, Massachusetts). ACM, New York, NY, USA, 756--757.
[32]
OpenAI. 2024. CLIP: Connecting text and images. https://openai.com/research/clip.
[33]
Alec Radford et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748--8763.
[34]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. International Conference on Learning Representations.
[35]
Ishan Sahu, Arijit Ukil, Sundeep Khandelwal, and Arpan Pal. 2022. LTH-ECG: Lottery Ticket Hypothesis-based Deep Learning Model Compression for Atrial Fibrillation Detection from Single Lead ECG On Wearable and Implantable Devices. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, USA, 1655--1658.
[36]
Varsha Sharma, Shalini Mukhopadhyay, Sakyajit Bhattacharya, Swarnava Dey, and Avik Ghose. 2023. PuffConv: A System for Online and On-device Puff Detection for Smoking Cessation. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). IEEE, USA, 595--600.
[37]
Duncan Stewart, Jeff Loucks, Mark Casey, and Craig Wigginton. 2019. Bringing AI to the device: Edge AI chips come into their own. https://www2.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2020/ai-chips.html.
[38]
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive Multiview Coding. arXiv preprint arXiv:1906.05849 (2020).
[39]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748 (2019).
[40]
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. 2018. Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination. arXiv preprint arXiv:1805.01978 (2018).
[41]
Mang Ye et al. 2019. Unsupervised Embedding Learning via Invariant and Spreading Instance Feature. arXiv preprint arXiv:1904.03436 (2019).
Index Terms
- Towards a Task-agnostic Distillation Methodology for Creating Edge Foundation Models
Recommendations
Foundation Models in Healthcare: Opportunities, Biases and Regulatory Prospects in Europe
Electronic Government and the Information Systems PerspectiveTowards a foundation model for geospatial artificial intelligence (vision paper)
SIGSPATIAL '22: Proceedings of the 30th International Conference on Advances in Geographic Information SystemsLarge pre-trained models, also known as foundation models (FMs), are trained in a task-agnostic manner on large-scale data and can be adapted to a wide range of downstream tasks by fine tuning, few-shot, or even zero-shot learning. Despite their ...
Lecture-style Tutorial: Towards Graph Foundation Models
WWW '24: Companion Proceedings of the ACM on Web Conference 2024Emerging as fundamental building blocks for diverse artificial intelligence applications, foundation models have achieved notable success across natural language processing and many other domains. Concurrently, graph machine learning has gradually ...
Comments
Information & Contributors
Information
Published In
June 2024
44 pages
ISBN:9798400706639
DOI:10.1145/3662006
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 June 2024
Accepted: 05 June 2009
Revised: 12 March 2009
Received: 20 February 2007
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
MOBISYS '24
Sponsor:
MOBISYS '24: The 22nd Annual International Conference on Mobile Systems, Applications and Services
June 3 - 7, 2024
Tokyo, Minato-ku, Japan
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 12Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)12
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in