Self-Supervised Learning from Unlabeled IoT Data

Abstract

With a network architecture selected and before elaborating other inference challenges in a book on Edge AI, it behooves us to consider the question of training neural networks for edge AI applications. Training is a pre-requisite of all that comes next. The key training bottleneck in AI is traditionally the cost of data labeling. Unlabeled data are widely available in the IoT space, but labeling is expensive. This leads to the central question covered in this chapter: how best to exploit the widely-available unlabeled data to reduce labeling cost while improving inference quality? An inspiration for the answer comes from a key recent development in artificial intelligence—the introduction of foundation models for different categories of applications (such as ChatGPT for conversational interfaces, and CLIP for images). The hallmark of foundation models is their ability to learn in an unsupervised fashion (i.e., without an explicit need to label the data). While many approaches were proposed for training foundation models, two of the most important ones are (1) contrastive learning, and (2) masking. Briefly, contrastive learning teaches the model a notion of semantic similarity by presenting it with similar data samples (e.g., an image and its rotation) and contrasting those with randomly chosen samples. Masking forces the model to extract high-level semantic structure from data by masking parts of the input and asking the model to reproduce it from the remaining (unmasked) data. Pretraining a model using either approach allows it to leverage large amounts of unlabeled data. How do these two techniques transfer to the IoT domain and how effective are they at reducing the need for data labeling? The chapter answers these questions from recent work on contrastive learning and masked auto-encoding for the IoT domain that can be thought of as a precursor of the emergence of foundation models for IoT applications.