research-article

Incremental On-Device Tiny Machine Learning

Authors:

Simone Disabato,

Manuel RoveriAuthors Info & Claims

AIChallengeIoT '20: Proceedings of the 2nd International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things

Pages 7 - 13

https://doi.org/10.1145/3417313.3429378

Published: 16 November 2020 Publication History

Abstract

Tiny Machine Learning (TML) is a novel research area aiming at designing and developing Machine Learning (ML) techniques meant to be executed on Embedded Systems and Internet-of-Things (IoT) units. Such techniques, which take into account the constraints on computation, memory, and energy characterizing the hardware platform they operate on, exploit approximation and pruning mechanisms to reduce the computational load and the memory demand of Machine and Deep Learning (DL) algorithms.

Despite the advancement of the research, TML solutions present in the literature assume that Embedded Systems and IoT units support only the inference of ML and DL algorithms, whereas their training is confined to more-powerful computing units (due to larger computational load and memory demand). This also prevents such pervasive devices from being able to learn in an incremental way directly from the field to improve the accuracy over time or to adapt to new working conditions.

The aim of this paper is to address such an open challenge by introducing an incremental algorithm based on transfer learning and k-nearest neighbor to support the on-device learning (and not only the inference) of ML and DL solutions on embedded systems and IoT units. Moreover, the proposed solution is general and can be applied to different application scenarios. Experimental results on image/audio benchmarks and two off-the-shelf hardware platforms show the feasibility and effectiveness of the proposed solution.

References

[1]

Cesare Alippi, Simone Disabato, and Manuel Roveri. 2018. Moving Convolutional Neural Networks to Embedded Systems: The AlexNet and VGG-16 Case. In 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, Porto, 212--223.

Digital Library

[2]

Cesare Alippi and Manuel Roveri. 2008. Just-in-time adaptive classifiers---Part II: Designing the classifier. IEEE Transactions on Neural Networks 19, 12 (2008), 2053--2064.

Digital Library

[3]

Naomi S Altman. 1992. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46, 3 (1992), 175--185.

[4]

Colby R Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, et al. 2020. Benchmarking TinyML Systems: Challenges and Direction. arXiv preprint arXiv: 2003.04821 (2020).

[5]

Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. 2017. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 527--536.

Digital Library

[6]

Johannes Buchner. 2017. Synthetic Speech Commands: A public dataset for single-word speech recognition. Dataset available from https://www.kaggle.com/jbuchner/synthetic-speech-commands-dataset/ (2017).

[7]

Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. 2017. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5918--5926.

[8]

Lukas Cavigelli and Luca Benini. 2016. Origami: A 803-gop/s/w convolutional network accelerator. IEEE Transactions on Circuits and Systems for Video Technology 27, 11 (2016), 2461--2475.

[9]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273--297.

[10]

Henggang Cui, Hao Zhang, Gregory R Ganger, Phillip B Gibbons, and Eric P Xing. 2016. Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proceedings of the Eleventh European Conference on Computer Systems. 1--16.

Digital Library

[11]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.

[12]

Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems. 1269--1277.

[13]

Simone Disabato and Manuel Roveri. 2018. Reducing the Computation Load of Convolutional Neural Networks through Gate Classification. In 2018 International Joint Conference on Neural Networks (IJCNN), Vol. 2018-July. IEEE, 208--215.

[14]

Simone Disabato and Manuel Roveri. 2019. Learning Convolutional Neural Networks in presence of Concept Drift. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.

[15]

Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar. 2015. Learning in nonstationary environments: A survey. IEEE Computational Intelligence Magazine 10, 4 (2015), 12--25.

Digital Library

[16]

Aysegul Dundar, Jonghoon Jin, Berin Martini, and Eugenio Culurciello. 2016. Embedded streaming deep neural networks accelerator with applications. IEEE transactions on neural networks and learning systems 28, 7 (2016), 1572--1583.

[17]

Igor Fedorov, Ryan P Adams, Matthew Mattina, and Paul Whatmough. 2019. Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers. In Advances in Neural Information Processing Systems. 4977--4989.

[18]

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015. Deep learning with limited numerical precision. In International Conference on Machine Learning. 1737--1746.

Digital Library

[19]

Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 1510.00149 (2015).

[20]

Peter Hart. 1968. The condensed nearest neighbor rule (Corresp.). IEEE transactions on information theory 14, 3 (1968), 515--516.

[21]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]

Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel Pruning for Accelerating Very Deep Neural Networks. In The IEEE International Conference on Computer Vision (ICCV).

[23]

Ashish Kumar, Saurabh Goyal, and Manik Varma. 2017. Resource-efficient machine learning in 2 KB RAM for the internet of things. In International Conference on Machine Learning. 1935--1944.

[24]

Dawei Li, Xiaolong Wang, and Deguang Kong. 2018. Deeprebirth: Accelerating deep neural network execution on mobile devices. In Thirty-second AAAI conference on artificial intelligence.

[25]

Shaohui Lin, Rongrong Ji, Chao Chen, Dacheng Tao, and Jiebo Luo. 2018. Holistic cnn compression via low-rank decomposition with knowledge transfer. IEEE transactions on pattern analysis and machine intelligence 41, 12 (2018), 2889--2905.

[26]

Xiaofan Lin, Cong Zhao, and Wei Pan. 2017. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems. 345--353.

[27]

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision. Springer, 525--542.

[28]

Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. 2013. Learning separable filters. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2754--2761.

Digital Library

[29]

Ramon Sanchez-Iborra and Antonio F Skarmeta. 2020. TinyML-Enabled Frugal Smart Objects: Challenges and Opportunities. IEEE Circuits and Systems Magazine 20, 3 (2020), 4--18.

[30]

Jie Tang, Dawei Sun, Shaoshan Liu, and Jean-Luc Gaudiot. 2017. Enabling deep learning on IoT devices. Computer 50, 10 (2017), 92--96.

Digital Library

[31]

Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 2464--2469.

[32]

Ivan Tomek. 1976. Two Modifications of CNN. IEEE Transactions on Systems, Man, and Cybernetics SMC-6, 11 (1976), 769--772.

[33]

Yu Emma Wang, Gu-Yeon Wei, and David Brooks. 2019. Benchmarking TPU, GPU, and CPU platforms for deep learning. arXiv preprint arXiv: 1907.10701 (2019).

[34]

Pete Warden. 2018. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 1804.03209 (2018).

[35]

Dennis L Wilson. 1972. Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics 3 (1972), 408--421.

[36]

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems. 3320--3328.

[37]

Chen Zhang, Peng Li, Guangyu Sun, Yijin Guan, Bingjun Xiao, and Jason Cong. 2015. Optimizing fpga-based accelerator design for deep convolutional neural networks. In Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 161--170.

Digital Library

Cited By

Fragkou EKatsaros D(2024)A Joint Survey in Decentralized Federated Learning and TinyML: A Brief Introduction to Swarm LearningFuture Internet10.3390/fi1611041316:11(413)Online publication date: 8-Nov-2024
https://doi.org/10.3390/fi16110413
Ren HAnicic DLi XRunkler T(2024)On-device Online Learning and Semantic Management of TinyML SystemsACM Transactions on Embedded Computing Systems10.1145/366527823:4(1-32)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3665278
Pandey SNair SLi XHandl J(2024)Onboard Class Incremental Learning for Resource-Constrained scenarios using Genetic Algorithm and TinyMLProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654392(299-302)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654392
Show More Cited By

Index Terms

Incremental On-Device Tiny Machine Learning
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Sensor networks
2. Computing methodologies
  1. Machine learning
    1. Learning paradigms

Recommendations

Survey of continuous deep learning methods and techniques used for incremental learning
Abstract
Neural networks and deep learning algorithms are designed to function similarly to biological synaptic structures. However, classical deep learning algorithms fail to fully capture the need for continuous learning; this has led to the advent of ...
TyBox: An Automatic Design and Code Generation Toolbox for TinyML Incremental On-Device Learning
Incremental on-device learning is one of the most relevant and interesting challenges in the field of Tiny Machine Learning (TinyML). Indeed, differently from traditional TinyML solutions where the training is typically carried out on the Cloud and ...
Multithreading Incremental Learning Scheme for Embedded System to Realize a High-Throughput
Proceedings of the 23rd International Conference on Neural Information Processing - Volume 9948

Recent improvement of the microcomputer enables it to execute complex intelligent algorithms on embedded systems. However, when using conventional incremental learning methods, its resources are often increased with learning, and continuing the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AIChallengeIoT '20: Proceedings of the 2nd International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things

November 2020

74 pages

ISBN:9781450381345

DOI:10.1145/3417313

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SenSys '20

Sponsor:

SenSys '20: The 18th ACM Conference on Embedded Networked Sensor Systems

November 16 - 19, 2020

Virtual Event, Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
1,174
Total Downloads

Downloads (Last 12 months)180
Downloads (Last 6 weeks)8

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fragkou EKatsaros D(2024)A Joint Survey in Decentralized Federated Learning and TinyML: A Brief Introduction to Swarm LearningFuture Internet10.3390/fi1611041316:11(413)Online publication date: 8-Nov-2024
https://doi.org/10.3390/fi16110413
Ren HAnicic DLi XRunkler T(2024)On-device Online Learning and Semantic Management of TinyML SystemsACM Transactions on Embedded Computing Systems10.1145/366527823:4(1-32)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3665278
Pandey SNair SLi XHandl J(2024)Onboard Class Incremental Learning for Resource-Constrained scenarios using Genetic Algorithm and TinyMLProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654392(299-302)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654392
Andrade PSilva IDiniz MFlores TCosta DSoares E(2024)Online Processing of Vehicular Data on the Edge Through an Unsupervised TinyML Regression TechniqueACM Transactions on Embedded Computing Systems10.1145/359135623:3(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3591356
Disabato SRoveri M(2024)Tiny Machine Learning for Concept DriftIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.322989735:6(8470-8481)Online publication date: Jun-2024
https://doi.org/10.1109/TNNLS.2022.3229897
Shalby HPavan MRoveri M(2024)StreamTinyNet: video streaming analysis with spatial-temporal TinyML2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651090(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651090
Cioflan CCavigelli LRusci Mde Prado MBenini L(2024)On-Device Domain Learning for Keyword Spotting on Low-Power Extreme Edge Embedded Systems2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS)10.1109/AICAS59952.2024.10595987(6-10)Online publication date: 22-Apr-2024
https://doi.org/10.1109/AICAS59952.2024.10595987
Giménez NLee JFreitag FVandierendonck H(2024)The Effects of Weight Quantization on Online Federated Learning for the IoT: A Case StudyIEEE Access10.1109/ACCESS.2024.334955712(5490-5502)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3349557
Lin TChang CZhuang TPutranto A(2024)Real-time hollow defect detection in tiles using on-device tiny machine learningMeasurement Science and Technology10.1088/1361-6501/ad266535:5(056006)Online publication date: 22-Feb-2024
https://doi.org/10.1088/1361-6501/ad2665
Epifani LCaruso A(2024)A Survey on Deep Learning in UAV imagery for Precision Agriculture and Wild Flora Monitoring: Datasets, Models and ChallengesSmart Agricultural Technology10.1016/j.atech.2024.100625(100625)Online publication date: Oct-2024
https://doi.org/10.1016/j.atech.2024.100625
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten