Recognition of Gestures over Textiles with Acoustic Signatures

We demonstrate a method capable of turning textured surfaces (in particular textile patches) into opportunistic input interfaces thanks to a machine learning model pre-trained on acoustic signals generated by scratching different fabrics. A single short audio recording is then sufficient to characterize both a gesture and the textures substrate. The sensing method does not require intervention on the fabric (no special coating, additional sensors or wires). It is passive (no acoustic or RF signal injected) and works well using regular microphones, such as those embedded in smartphones. Our prototype yields 93.86% accuracy on simultaneous texture/gesture recognition on a test matrix formed by eight textures and eight gestures as long as the microphone is close enough (e.g. under the fabric), or when the patch is attached to a solid body transmitting sound waves. Preliminary results also show that the system recognizes the manipulation of Velcro straps, zippers, or the taping or scratching of plastic cloth buttons over the air when the microphone is in personal space. This research paves the way for a fruitful collaboration between wearables researchers and fashion designers that could lead to an interaction dictionary for common textile patterns or guidelines for the design of signature-robust stitched patches not compromising aesthetic elements.