A Review on the emerging technology of TinyML

NNs tend to have several parameters with significant redundancy regarding the models, ultimately leading to more computation power and memory usage than required [52]. As stated above, TinyML is a paradigm that allows ML models to fit into constrained hardware without compromising their energy efficiency [55]. To achieve model inference in devices with limited resources, specifically MCUs, it is necessary to heavily optimize and compress the models being used. Optimizing algorithms and ML models is a challenging issue. As described previously, it is not just a software or hardware problem, but hardware and software co-design is a prerequisite to obtain the targeted result [58]. Over the past decades, many methods, frameworks, and techniques have been proposed for compression. O’Neill’s work [147] provides an analytical review of those methods. Furthermore, recent research also reports the first attempts to optimize deep RL designed for resource-constrained systems [198]. A brief description of the most commonly used techniques follows, aiming to highlight the preferred ones in TinyML.

A network trained on high-end GPUs has values stored in 32-bit FP single precision. For faster inference and lower computational needs, researchers have focused on training lower-precision networks with 1-bit or 2-bit representations able to run on other types of hardware such as FPGAs and ASICs. The process of representing network values with fewer bits is known as quantization. One necessity of the aforementioned method is to retain accuracy or have the lowest possible tradeoff. A typical quantization technique lowers the values from FP32 to 8-bit representations [53, 86]. Gupta et al. [85] analyzed that stochastic rounding methods can maintain accuracy when an FP32 model is quantized to FP16. Cambier et al. [33] proposed a shifted and squeezed 8-bit (S2FP-8) to avoid stochastic rounding, while Mellempudi et al. [138] showcased a different approach where this work proposed different Floating Representations for each layer eg FP64, FP32, FP16 than the typical FP32. Park et al. [157] proposed 3-bit activations where weights are also quantized to 4-bit and 16-bit scaling factors for approximately 1% of the network, resulting in only 1% of accuracy loss. Migacz proposed different approaches [140], which used relative entropy to measure information loss, and Banner et al. [26], who used noise and clipping distortion. Merolla et al. [139] tried to understand how different distortions affect the networks. Their results revealed that NNs are robust to distortions but have more extensive convergence times. References [54, 111] suggested that post-training quantization on small networks can hit accuracy when utilizing 8-bit formats or lower. In an attempt to weather the aforementioned issue, Zhou et al. [236] proposed the quantization of gradients to 6-bit and stochastically propagated using estimators. By minimizing the loss regarding binarized weights, Hou et al. [98] proposed a Newton algorithm combined with Hessian approximation. Explicit Loss-aware Quantization (ELQ) is another method of quantization proposed by Zhou et al. [235], which focuses on minimizing the loss for very low precision. Similarly, Park et al. [157] achieved reduced precision by intensively narrowing the range weight values. Stock et al. [197], to overcome quantization drift, used iterative quantization starting with low layers and performed gradient updates to the rest. On the contrary, Jacob et al. [54] used an estimator to backpropagate through activations and weights during the training process instead of exploiting the aforementioned iterative approach. Finally, Fan et al. [69] suggested that utilizing the previous two approaches is not advisable when operating with ternary or binary precision representations. They proposed a simulation of quantization noise, which is random for a subset of the network, and then the performance of backward weights passes on to the rest.

Quantization is one of the most often used optimization strategies for TinyML compression on MCUs. As described previously, the former attempts to map 64-bit or 32-bit weights to a smaller bit width to store the required models in MCUs [58], while there is no focus or strategies in the initial architecture and instead relies on techniques such as transfer learning after the final model is ready [3]. In Reference [168], the Quantised Latent Replay-based Continual Learning method is proposed, which relies on low bandwidth quantization that can reduce the memory requirements of the layer but also increase the overall speed of the network. This work is one of the first attempts at creating a hardware and software platform for TinyML continual learning.

Network pruning is one of the oldest and most used techniques for optimization. It is based on the human brain, where irrelevant and unimportant past experiences are removed to make room for newer [219]. Network pruning removes synapses and neurons that fall under a certain weight threshold [90] to improve performance and reduce the computational needs of a network without sacrificing a significant amount of accuracy. Han and Qiao [92] and Narasimha et al. [146] suggested a different perspective by combining different techniques, such as the addition of new neurons and a predefined percentage of weights that need to be pruned instead of the standard version of setting a threshold. Magnitude-based Pruning (MBP) is used due to its high performance while it keeps its simplicity for ML and other tasks [185] and even tends to outperform layer-wise MBP [90, 107, 120, 171]. In References [94, 119, 171], the authors measure the importance of the pruned units and try to remove only those with minor importance that ultimately will lead to the least loss. Mozer and Smolensky [144] employed skeletonization, a method to eliminate the least important units during training. Karnin [107] measured the sensitivity of the loss function so that the network can be efficiently pruned, while Engelbrecht [66] recommended assessing whether the variance of sensitivity is statistically different from zero before proceeding with weight removal. Additionally, LeCun et al. [119] proposed that weight importance can be estimated utilizing the Taylor series.

Based on this technique, Molchanov et al. [142] used a Taylor expansion to prune the weights with the lowest change regarding the cost function. Theis et al. [205] extended Molchanov’s work by providing cost estimates for network pruning. Mallya and Lazebnik [132] suggested that a dynamic mask can learn to adapt a dense network to a different sparse network. A different approach is structured sparsity learning, which can be found in Reference [224]. Louizos et al. [129] suggested that Bayesian methods can also be utilized for structured pruning. Dai et al. [47] proposed an alternative method known as pruning via variational information bottleneck, where the authors aimed to remove mutual information between layers.

Another approach was introduced by Lin et al. [124] by applying a soft mask to minimize the mean squared error. This work is an extension of a previous work [123], in which the authors achieved optimization using binary masks and hard thresholding. Genetic Algorithms (GAs) are another approach that can be utilized for the pruning method by keeping the best performance parameters they generate and mixing them until they achieve the desired result. Several researchers attempted the usage of GAs to prune a network, and some examples can be studied in References [34, 100, 225].

Noy et al. [150] tried to reduce the required time for searching neural architecture by implementing pruning via simulated pruning. Their work is based on the DARTS approach suggested in Reference [125]. Particle filtering for pruning is another method where sequential Monte Carlo estimation is utilized to identify the crucial weights [9]. Particle swarm optimization for the pruning method was also proposed in Reference [210]. Furthermore, AutoML [95] was proposed to improve the compression performance and the overall efficiency in a more automated pruning approach. Recently, pruning before training where the architecture is adequately initialized and designed shows that the network utilizing this method can achieve the same accuracy as an entire network [73, 127].

One of the first attempts to reduce network size is associated with layer weight sharing. One key point of this method is that the number of weights that should be shared is not always clear before it starts affecting the overall performance and accuracy of the network. To overcome this issue, some recent works attempted to apply the technique after the initial training instead of prior to training [22, 36, 211]. Nowlan and Hinton [149] proposed a different approach, where the authors tried a Gaussian mixture to assign the weights.

An extension of this work was provided later by Ullrich et al. [211], where the optimization method was based on soft-weight sharing. A different approach was seen in the works of Zhang et al. [231] and Plummer et al. [162], where they tried to learn which weights or parameters must be shared. Parameter hashing is another option where parameters are grouped to share weights randomly [36] or share weights of the same value [187, 223]. Another option for weight sharing is using transformers, as Dabre and Fujita [46] and Xiao et al. [227] proposed. Bai et al. [22] proposed deep equilibrium models to find the network’s point where backpropagation can be utilized. One more type of parameter hashing is when recursively reusing layers by feeding again into the input the result of the output layer [64, 110, 178]. The work of Kim et al. [109] showcased the usage of residual connection among the output and input layers. Tai et al. [202] proposed an extension of this work, where the authors used an element-wise addition regarding the intermediate outputs before passing the result to the final layer. Relevant works that seek to overcome Vanishing or Exploding Gradients (VEGs) issues were proposed by Zhang et al. [233] and Guo et al. [84].

The Neural Architecture Search (NAS) is another method utilized by data scientists who attempt to build NNs under stringent limitations, ready to be implemented in MCUs. It is a process used to select the model with the highest possible accuracy from a predefined space of CNNs [65]. The typical procedure of a NAS system includes the search algorithm, the search space, and an evaluation strategy [65]. According to Heim et al. [96], NAS techniques could be divided into four different categories: hardware and software co-design, hardware-aware and usage of perceptible metrics, no hardware influence, and finally, the usage of proxies characterized in advance.

Fedorov et al. introduced a NAS method, SpArSe, capable of finding dense and sparse CNNs for MCUs [70]. Another NAS with the name TinyNAS was proposed by Lin et al. [65]. This method includes a search approach that is used to optimize the search space to fit constrained hardware and then continues with the optimization for the search space. One more example was proposed by Heim et al. [96], where their proposed method utilizes generic and hardware-based optimizations to decrease the memory requirements of NNs and speed up the inference latency.

Despite software-oriented optimizations, hardware acceleration is also employed when specialized hardware could be optimized to improve speed and parallel processing [38]. The main focus of hardware accelerators is accelerating mathematical matrix operations [186].

The authors, in Reference [198], presented a block-based SC architecture focusing on improvements in energy and latency by dividing inputs into blocks that will be executed next in parallel. The experiments show significant savings in power consumption while retaining high accuracy.

The researchers of Reference [39] proposed PRIME framework, where NN applications could be accelerated by utilizing RAM due to its efficiency and potential to perform matrix-vector multiplications. With this architecture and the design provided by the authors, the overall performance is enhanced by 2,360 \(\times\) while the consumption is decreased by 895 \(\times\) .

Another team [122] introduced SmartShuttle for off-chip memory accesses to run as an accelerator for DL applications. The scheme can switch among different data reuse schemes and achieve a match for different layers dynamically. The results revealed a better performance than other state-of-the-art approaches, such as Eyeriss [37], which is also an accelerator for CNNs and can optimize energy efficiency by including a hardware accelerator and a chip for Dynamic RAM (DRAM).

One more proposal regarding hardware can be read in Reference [145], where the authors try to overcome three main design issues: data access, energy consumption, and data movement. Those three key components, especially the third one, data movement, can result in difficulties with bandwidth and latency. Their main focus is to bring computation closer to data by utilizing a technique called Processing In Memory (PIM) [74]. By implementing this method, the overall data movement between memory and the computation units could be reduced or eliminated by a great margin.

Exploiting FPGAs acceleration for matrix multiplications, the authors in Reference [160] presented an implementation of the transformer model and proper hardware scheduling for DL models utilizing also the method of weight pruning. The results reveal low latency regarding the inference on the hardware with speeds up to 10.96 times compared to CPUs and 2.08 when compared to GPUs. Some more examples related to hardware acceleration designs and architectures can be studied in References [105, 163, 228].

As mentioned earlier, another popular way for training, optimizing, implementing, or even all the procedures could be achieved by utilizing frameworks, libraries, and other toolkits. Software-based solutions such as Google’s TensorFlow (TF) Lite for MCUs [49] and Microsoft’s Embedded Learning Library (ELL) [79] enable the design and deployment of ML models on power-efficient devices and even single-board computers such as Arduino and Raspberry Pi [165]. Additionally, the research community has already developed several software-based solutions, such as libraries and diverse sets of tools, for automatically converting pre-trained AI algorithms, such as NNs and ML models, and integrating the generated optimized code into researchers’ boards. A more detailed overview of the software-based solutions exploited to implement models into MCUs will be provided in Section 4. The infrastructure of the laboratory and datasets provided by the project “ParICT-CENG” were leveraged to evaluate the frameworks described in this work. In future development, benchmarking results will be presented.

Figure 1 showcases a summary of the technology of TinyML, with information regarding the technology’s main optimization methods, the main benefits and challenges, a framework example, some of the most valuable educational resources, and finally, three different boards fully compatible with TinyML.

Another interesting statistic is that MCUs are currently used in vehicles, consumer electronics, home devices, and industrial equipment. It is expected to have a rise in sales by the end of 2021, followed by higher increases by the year 2023 [101]. This translates to an environment that is mature and prepared to implement the necessary software to create even more innovative devices using TinyML technology. Additionally, one of the most common applications of TinyML is keyword spotting [232]. Many technological champions such as Apple, Google, and Amazon already use a hybrid of keyword spotting applications by inferring ML models into their smart home devices but also depend on the cloud for better results [6, 11, 83]. Another example of utilizing the technology is anomaly detection, a valuable mechanism for security applications [35]. Moreover, since the need for TinyML operations hardware is publicly available, as stated above, and the recipe for ML applications is very close to the technology of TinyML, a new road for jobs and opportunities is shining. Educational programs for TinyML are extremely necessary to achieve the training mentioned earlier. One of the first collaborations to create educational material was conducted between Harvard University [93] and Google [82] on the edX platform [61]. The course aims to train and prepare students on ways to overcome many challenges, such as the insufficient hardware resources required to run the ML models, and close the growing gap between industry and academia due to the industry’s rapid progress. The course teaches the fundamentals of TinyML technology, trains students on real-world TinyML applications, and also informs them about ethical and life-cycle considerations regarding development and deployment.

Additionally, a TinyML kit was co-designed with Arduino [16] to offer low-cost project-based learning to students. The kit contains the Arduino Nano 33 BLE Sense [14], a camera module [13], and a TinyML shield for simplified sensor integration [62, 170]. Another course worth mentioning is hosted on the Coursera learning platform [43]. The course was developed with some of the most valuable key actors of the TinyML sector: Edge Impulse [60], Arm [20], Arduino [16], and the TinyML Foundation [206]. The course starts with an introduction to ML and embedded systems, continues with NNs and how to train them, and concludes with teaching how to deploy those networks to MCUs. The concepts, demonstrations, and tutorials can be done with the user’s smartphone device or Arduino Nano 33 BLE Sense board [44, 59]. Warden and Situnayake offer another great educational resource in the form of a book. The authors introduce ML and embedded systems, while the rest of the book enables the reader to create a series of TinyML projects. The book is ideal for hardware or software developers who want to infer ML models in MCUs [207].

In conclusion, TinyML is the miniaturization of intelligence machinery that can collect data, process it, and extract valuable information in a safe, energy-efficient, reliable, and low-cost way; and has applications in most scientific and industry sectors. There is still significant interest in improving NN capabilities and accuracy [51, 131, 168]. However, with the advances and opportunities TinyML is capable of offering, it is evident that there is now a need to keep accuracy on high levels while also optimizing and compressing the models to be utilized by power constraint hardware, identified as MCUs. Finally, there is a great need to democratize the technology of AI. Nowadays, current AI implementations are developed with businesses and end-consumers in mind and are built to run on research workstations without considering how this could also be achieved in real-world applications and hardware, and even more in constrained hardware. TinyML could bring AI algorithms to more sectors, connect communities, and achieve a digital revolution with more tailored results extracted from data collected near the fountain of each sector and user. As mentioned above, MCUs are among the best-suited hardware devices due to their low cost and worldwide accessibility from everyone [161, 217].

Table 1 provides additional educational material. These educational materials are categorized according to their type, book, course, tutorial, and so on. The content the development boards utilize and their creators are also mentioned.

TinyML offers tangible solutions to the urgent energy consumption problems and computational limitations plaguing conventional ML deployment. TinyML employs tiny, energy-efficient processors to perform complex computations on the device itself, unlike conventional methods that often require powerful servers and significant energy resources. This eliminates the need for constant communication with central servers, thereby decreasing latency and energy consumption. Additionally, sensitive data can remain on the device, allowing for increased privacy and security. TinyML represents a significant paradigm shift in implementing and deploying ML, bridging the divide between cutting-edge technology and practical, real-world applications. To achieve this, various frameworks and techniques have been employed to allow the deployment of ML models on constrained hardware devices, specifically MCUs. These tools seek to optimize and efficiently execute ML models, considering these devices’ limited memory and power. The frameworks include model conversion, inference execution, hardware-specific optimizations, and support for multiple DL frameworks. Figure 3 is an infographic that visually classifies and illustrates these components of the TinyML ecosystem, providing a comprehensive overview of the various categories and frameworks. The available software can be categorized according to the primary functionality or purpose of each framework/tool as follows:

The advantages of embedding ML models on hardware are undeniably numerous, and as mentioned earlier, security, instant, tailored results, and the creation of autonomous devices are some of the most resounding examples. Several researchers are also exploring this new sector and analyzing why the industry should start inferring models on hardware, how it is achievable, and finally, why the technology of ML must become more accessible to everyone [49, 176, 217, 232]. Two of the most popular boards ready to run any type of software, specifically the technology in consideration, are Raspberry Pi and Jetson Nano from Nvidia [151]. Jetson is a great example of a tiny computer able to fit in a palm. It is powerful and most commonly used in automotive and robotics. In those two sectors, any application built requires a vast amount of power source, which can also power the demanding board from Nvidia. For smaller, innovative, and not that demanding applications such as wearable devices, the need for an external power supply makes using the board prohibitive [222]. The solution is the use of another constrained hardware, the MCU. An MCU is a hardware platform that operates below 1 mWatt and has a few kilobytes of RAM, typically with the same amount of flash memory for data storage. Nowadays, most MCUs switched to 32-bit CPUs, thanks to ARM. To be able to operate at the cost of a single small battery, MCUs do not always have an operating system, are single-threaded, and avoid the usage of dynamic location functions [24, 72, 199, 222]. With the advances in technology and by understanding the need for more ingenious IoT devices, intelligent machines, and wearable health-related devices, many manufacturers have developed small-factor and low-cost boards specifically designed for ML inference.

A popular board is from Sparkfun called Edge, utilizing an Ambiq Apollo3 MCU (the MCU’s schematic is visualized in Figure 4), which is powered by Arm Cortex-M4, operates at 48 MHZ, and has a TurboSPOT burst mode for reaching 96 MHZ. The board requires an ultra-low supply current at 6 \(\mu\) A/MHz executing from flash at 3.3 V; it has a flash memory of 1 MB and up to 384 KB of RAM [7]. Nano 33 BLE Sense [14] is the board of preference for many educational courses, since it has a unique TinyML Kit equipped with components and hardware required for students to start embedding ML models and is also used for research and small projects [18, 19, 48, 170, 172].

Table 3 summarizes the development boards mentioned above, as well as additional boards that use TinyML technology. The type of MCU and CPU, the CPU clock, the capacity of RAM and flash memory, and the dimensions of each board are reported.

Paul et al. [159] developed a real-time TinyML-based American sign language system. The proposed system is a highly efficient CNN-based fingerspelling recognition system that embedded TinyML in a Cortex-M7 with a 158 KB size board. The proposed solution reduces quantization’s accuracy drop and generalizes the model via interpolation augmentation. TinySpeech [226] are low-precision DNNs that are designed for limited-vocabulary speech recognition. The experimental results of the proposed TinyML NN showed significantly lower architectural and computational complexity compared with other DNNs for limited-vocabulary speech recognition.

The authors of Reference [71] implemented recurrent NNs for hearing aid hardware. Several TinyML techniques, such as pruning and integer quantization of weights/activations, were utilized.

A wearable system based on regular shoes for foot gesture recognition was implemented by Orfanidis et al. [154]. The system can track the user’s specific foot gestures in the environment of a smart city, and when the user is in danger, the system notifies his/her familiars. The overall process was implemented with an embedded NN in an MCU. The evaluating results have shown a 98% accuracy among five different activities and foot gestures.

A wearable TinyML-based wristband for hand gesture recognition was proposed in Reference [28]. The experimental results show that the proposed TinyML-based wristband can recognize seven hand gestures with 96.4% accuracy. Also, a hand gesture recognition approach for wearable devices is presented in Reference [234]. The proposed approach uses accelerometer and electromyogram signals for a dual-stage classification in a memory-efficient way. The TinyML-based approach achieved a 93.34% classification accuracy, a 17.79% improvement of the trained model, and a significant decrease in the device’s memory footprint.

A cloud computing system for high-level supporting systems combined with TinyML for prognostics and health management was proposed in Reference [237]. The authors of this work investigate sensor-based applications and predict health status while combining TinyML with cloud computing to adapt the system’s diverse requirements, such as power, latency, and communication.

A non-invasive TinyML-based system for real-time activity tracking for elderly people and their nurses was proposed by T’Jonck et al. [208]. The system provides real-time elderly concerns information such as incontinence, night wandering, and pressure ulcers. Furthermore, a healthcare body-pose estimation platform-agnostic framework was proposed in Reference [215]. The proposed application monitors patients and alerts when they fall off their bed, have an accident, or experience restricted movement.

Ooko et al. [153] used TinyML-based NN models to predict in real-time chronic obstructive pulmonary disease. The proposed approach improves the inference accuracy of portable and non-invasive self-diagnostic kits for respiratory diseases. Also, a low-cost mechanical ventilator, namely, A-Vent, was developed by Cabacungan et al. in the study [32]. The proposed ventilator uses TinyML models to detect patient-ventilator asynchrony. The experiment results showed that TinyML could be trained to detect breathing anomalies and provide low-cost and real-time remote monitoring.

Most of the aforementioned healthcare systems reported improved system accuracy because of the use of TinyML. Furthermore, healthcare devices and systems are crucial to providing real-time and tailored results. Finally, due to the local data processing, sensitive and private data are kept secure and private.

In Reference [115] a wearable TinyML-based alcohol sensor was proposed. The usage of the proposed wearable system is to detect the consumed level of alcohol. The system’s ML model was trained by the sensors’ collected data, and the TinyML model was uploaded to an nRF52480 MCU for the prediction of test data to account for the impact of environmental conditions on the alcohol sensor. The results indicate that the method is useful for refining sensor response implementation in a variety of heat and humidity conditions.

The authors of Reference [174] presented a novel approach for automating traffic scheduling based on the density of vehicles waiting in line. The proposed system detects vehicles with embedded sensors across the road’s lanes, and a TinyML model was built to predict the green signal timings and to control the traffic system efficiently. In Reference [148] a TinyML-based sensor for the classification of the vehicle’s type and the speed range was designed. The presented sensor is powered by the battery and mounted in the pavement for the vehicles’ classification with the use of recurrent NNs. The experimental results show an accuracy of 96% in vehicle’s type classification and 89% in vehicle’s speed classification.

A TinyML approach for detecting road anomalies (potholes, bumps, and obstacles) on vehicles was proposed in Reference [8]. The unsupervised TinyML technique is embedded in an Arduino Nano BLE 33.

Raza et al. [169] presented a novel approach for more considerable autonomy and intelligence in micro aerial vehicles. This is achieved using TinyML technology via OpenMV for lower latency, energy efficiency, offline inference, and data security in drones. The experimental results of this study revealed that integrating TinyML-based MCU into the drones makes the system viable in a practical context.

The authors of Reference [50] presented the deployment of TinyML models for autonomous driving mini-vehicles. The authors, with the use of high-throughput TinyCNNs having access to an on-board camera, control mini-vehicles and succeed in minimizing the inference energy consumption.

In the automotive field, TinyML-based systems succeed in significant improvements in systems energy efficiency, which are meaningful advantages in the modern automotive.

Vuppalapati et al. [216] presented a novel framework for TinyML-based agriculture sensors. The proposed framework exploits Azure DevOps automation techniques and TinyML to provide high-quality products, in the most cost-efficient way, ensuring farmers’ cost savings. Also, Reference [218] proposed a TinyML-based framework for rural areas and small farmers. The system’s TinyML IoT edge devices collect real-time data from a real production environment, contributing to creating a sustainable food future.

The authors of Reference [5] presented a real-time embedded prediction weather system. The system was implemented using tiny DNNs in an STM32 MCU to predict real-time environmental conditions. The main advantage of this system, rather than the others that have been proposed, is the performance of prediction close to the environmental sensor to avoid data traffic to the cloud.

Reference [209] illustrates an end-to-end strategy for enhancing the security of the food supply chain and, consequently, boosting the credibility of the food sector. The system seeks to increase the transparency of food supply chain monitoring systems by securing their constituent parts. A universal information monitoring strategy based on blockchain technology protects the integrity of gathered data, while a self-sovereign identification approach for all supply chain participants reduces single points of failure. Finally, monitoring devices are fitted with a security mechanism based on TinyML’s fledgling technology to mitigate a major amount of harmful supply-chain actor activity.

TinyML-based agriculture systems provide farmers with tailored results and autonomous, low-cost systems. Moreover, using TinyML in this area reduces access to cloud services, resulting in low network bandwidth and lower costs.

Dutta and Kant [57] designed a TinyML-based framework to predict potential threats that propagate to smart devices. The proposed framework copes with the various security challenges in the different protocols’ layers of IoT devices. Furthermore, a TinyML-based framework for detecting devices’ physical anomalies was proposed in Reference [128]. With the use of an Arduino Nano 33 BLE that embedded DL TinyML models mounted in a washing machine, physical anomalies were detected via a battery-powered embedded device with no network connection.

In the security field, the TinyML approach can shield IoT devices in an automated way while requiring low power consumption.

Acharjee and Deb, in Reference [3], used TinyML strategies, such as post-training quantization, to generate cartoonized versions of real-world images. The proposed method’s testing results showed that the implemented model with post-training quantization achieves a high compression level. An embedded system for TinyML-based audio classification was presented in Reference [114]. The proposed technique improves the operation speed and decreases the time and the energy consumption. Also, the experiments have proved that co-designing both hardware and software reduces execution overhead when compared to optimizations used only on the software side.

Kamal et al. [106] proposed an architectural design for checking and quality auditing machine objects using a TinyML DNN. The proposed design consists of (i) the machine object module, (ii) the edge computing unit to connect, configure, and control the module, (iii) the edge server that embeds the TinyML, and (iv) the cloud services for the quality audit and predictive maintenance of the machine objects. Giordano et al. [76] developed a TinyML-based wireless camera for face recognition. The TinyML algorithm for face recognition was hosted in an ARM Cortex-M4F MCU for the onboard data processing, while the information on recognized faces was sent via a long-range LoRa communication module. A working prototype of the system was evaluated both for the capability of battery-less and self-sustainability, with a high accuracy of up to 97%.

TinyML-based industry systems improve the system’s operational speeds while providing tailored and real-time results. Also, as communication with cloud services is nearly eliminated, local data processing provides low power consumption and more privacy.

Despite the wide variety of TinyML applications present in both research and industrial domains, the technology is not without its challenges. These challenges, which will be elaborated upon in the subsequent section, have been the subject of considerable efforts aimed at their resolution. In the forthcoming paragraph, a succinct overview of various prospective trajectories for TinyML will be presented. These trajectories encompass potential enhancements to the technology itself and ways in which the technology could be employed to augment deployed systems and applications.

Until now, a typical ML model or NN was meant to be trained and run on powerful research workstations or in the cloud [176]. Implementing and deploying models, though, in a device that not only lacks powerful CPUs and GPUs but, on the contrary, relies on an MCU with limited memory, storage, and CPU capabilities, is a different story. The majority of the devices, as mentioned earlier, have an average clock speed of 100 MHz, with less than 1 MB of flash memory. As indicated above, optimizing a model to be suitable for deployment in such a device can be a complex process that mostly affects the model’s overall accuracy. There is a need for new and more aggressive optimization methods along with advances regarding memory and CPU speeds in the hardware sector.

Software and hardware heterogeneity can be identified as the second major issue. The diversity of hardware and algorithms makes prohibitive the development of a universal framework capable of training, optimizing, and deploying a model to several different devices with a single compilation. Since every system is different, despite the fact that two different systems may share the same components, a model trained for a specific device may not work with another one [172]. Furthermore, as stated many times, the ecosystem of TinyML requires careful software and hardware co-design. Building models universally could hinder accuracy, power consumption, and storage requirements.

The aforementioned issue continues and impacts important principles of ML, such as the development of universal datasets, models, and tools used for testing and benchmarking. Before creating ML applications, the significant first step for data scientists is to acquire the appropriate data from sensors or datasets. There is a vast amount of publicly available datasets, but, since the technology is still in an evolving stage, those datasets are not altered to reflect the energy and hardware constraints found on MCUs. The community also must create pre-trained models that require low-to-moderate changes to be deployed on different devices. Finally, an essential step of the ML procedure is testing and benchmarking. Models and algorithms must be compared and evaluated to get a better knowledge of when and how to implement models for different situations and sectors [194]. MLCommons is one of the first attempts to accelerate ML innovation by providing the correct rules and requirements for benchmarking and datasets and also provides the best practices to help or, as stated on their site, empower researchers by exchanging models experiments and applications [141].

Constrained devices are not built with a focus on security due to their limited resources. Even more complex models and data storage may still force developers to upload some of the data to an external source, server, or the cloud. Data transmission, of course, shares the same dangers found on simple IoT devices that depend on external sources for processing demanding operations such as AI and NN applications.