Mobile touchscreen user interface

Touchscreen technology and devices that use that technology have become widely popular over the past few years. Even so, very small amount of people knows that there are different technologies implemented in those devices that they use in everyday life. Through this seminar, we would try to present the different solutions currently available on the market. Simultaneously, we will compare between those solutions, emphasizing on differences between those technologies, and making the differences between them more obvious and understandable. Later, we will give our opinions on market trends in the future, and our vision and conclusion about this topic .

This book investigates young children’s everyday digital practices, embodied digital play, and digital media products – such as mobile applications, digital games, and software tools. The book provides a critical and collective perspective on the ways young children’s mobile media culture is currently being reshaped.

The chapters draw on research that extends from the household to social media platforms and public spaces. Moving across these interconnected sites, this book explores how young children are currently configured as consumers, users, and subjects of mobile media technologies. These arrangements of media use are analysed through a conceptual lens of digital dexterity, which locates children’s capacities to use mobile media interfaces and digital products not simply in terms of physical skills or developmental capacities, but importantly, through the design and affordances of mobile technologies and touch-based interfaces, cultures of interactive play and digital parenting, and economies of digital platforms and technology product design.

This posting is an annotated bibliography focused broadly on touchscreen and gesture user interfaces, on-line character recognition (a.k.a. dynamic character recognition, a.k.a. pen and touch computing), both hardware and software. It has been a continuing work-in-progress since the 1980s. It includes references on related technical topics I have encountered in my career: for example PDAs/highly-portable computing, cryptographic communications, signature verification, biometric authentication, and digital rights management (DRM). I am posting it as a service to those with interest in the field. It may also be of special interest to anyone investigating any of the areas of digitizer tablets, touchscreens, character recognition, touch/gesture user interfaces, multi-touch computing, passive and active tactile feedback, touch and proximity sensors, augmented reality, haptics, contextdependent intrepretation of user input, and applications including the same. It covers the time period from approximately 1887 / 1891 (first electronic tablets with "touch" input and a display), through 1914 (first electronic gesture/handwriting-recognition input and user-interface system), to the first handwriting-recognition tablet device connected to a modern electronic computer in 1957 (the "Stylator") and the more famous Rand Tablet (1961), to the present day. As with any subject, the focus has modulated over the decades, and this bibliography follows these topics both forward in time, and historically back in time. Tablets and touchscreens have evolved into a variety of pointing devices, into PDAs and smart-phones, locating and gesturing sensors with three-dimensional input with six degrees of freedom, and more. For example, there are no real lines between touch sensing for robotics, touch and contact sensing for user human input, fingerprint sensors, and touch and proximity sensing in general. Likewise, there are no real lines between haptics for touchscreens, haptics for instrumentation, and biometric feedback. Earlier work on handwriting recognition, with handwritten symbols sometimes used for command input as "gestures", has evolved to be part of a much broader range of gestures, including in-air and 3D gestures. Command user interfaces have merged with direct manipulation, and then with graphical user interfaces and virtual reality. Authenticating handwritten signatures has evolved to additional forms of dynamic biometrics. Haptic feedback has evolved from "simple" force-feedback to encompass audio, tactile, and visual signaling. Virtual reality systems seem to have waxed and waned, and waxed again. It is, indeed, a rich and complicated field, in all its aspects.

This posting of an annotated bibliography on gesture user interfaces, on-line character recognition (a.k.a.  dynamic character recognition, a.k.a. pen and touch computing), both hardware and software, has been a continuing work-in-progress since the 1980s. It includes information on related application topics, such as digital rights management (DRM), portable computing, cryptographic communications, and biometric authentication. I am posting it as a service to those with interest in the field. It may also be of special interest to anyone investigating any of the areas of digitizer tablets, touchscreens, character recognition, touch/gesture user interfaces, multi-touch computing, touch and proximity sensors, augmented reality, haptics, context-dependent intrepretation of user input, and applications including the same. It covers the time period from approximately 1891 (first electronic tablet) through 1914 (first gesture/handwriting-recognition input system) to the present day.

近年人因工程研究與教學，關於人與機器的關係，仍沿用Sanders與McCormick於1982年著作Human Fac-tors in Engineering and Design一書中提出的人機系統圖。然而，介面隨著技術的演變，從最早期的按鍵、旋鈕等型態的硬體介面，直到數位時代的軟體介面、觸控顯示螢幕，使用者已經熟悉以更複雜的手勢操作產品。而由於感測技術的導入，電腦所擔負的責任更重，可以更理解使用者、主動引導使用者。近期虛擬實境技術與市場亦帶來重大變化，此時，應該適合重新審視人機系統架構，提出一個較合乎現狀的互動圖。
傳統資通訊產品，由電腦主機、顯示器、與鍵盤滑鼠等輸入設備所構成。而近年配備觸控式螢幕的平板與智慧型手機等產品，已呈現「介面即整體、顯示即控制」的特性，可以較接近解釋介面原則中「直接操作」的特徵。而近年近期VR設備不斷地低價化、無線化，更可以獨立運算，比如All-in-one的Oculus Quest 2。其中VR頭盔已經微小化成整體、不須外接電腦、不須設置定位器，因此達成「頭盔即運算」，加上無須定位基地台的控制器，形成決策與控制、感知、資訊處理、運算合而為一。
在審視互動介面技術、蒐集市場案例之後，本研究透過研究群組討論、問卷調查，推論出：由於不同形式介面並存的現象，不易提出單一通用模型。目前本研究提出四大類型的互動模型，包含：1) 通用運算互動人機介面模型 (Gener-ic Model for Human-Computer Interaction)、2) 觸控式電子互動人機介面模型 (Display-Control Model for Human-Computer Interaction)、3) VR沉浸互動人機介面 (Immersive Model for VR interaction)、與 4) 元宇宙的人機介面圖 (Human-Computer Interaction Model in Metaverse)。本研究列出四大類互動模型的特性、並比較與優缺點整理，為呼應互動趨勢的多樣化，作為更新後的通用和沈浸式的人機互動架構圖，供學者研究與教學的參考。

關鍵字：人機互動、人機系統、虛擬實境、觸控螢幕設備、互動框架 (互動架構圖)

Direct touch is the major interaction model of modern mobile touchscreen devices. Although it is praised for its intuitiveness and input speed, its suffers from three major problems, which are inaccessibility, inextensibility, and inaccuracy, due to the limited control space of touch interaction.

In this thesis, we present three contributions, each addressing one of the three major problems:

1. For the problem of inaccessibility, which refers to the inability of interacting in the distant out-of-reach screen area when using one-handed or two-handed thumb interactions, we present BezelCursor. It is a novel one-handed thumb interaction technique that improves screen accessibility in the context of target acquisition. It combines bezel-initiated interaction and pointing gesture to control a screen cursor. Unlike the existing techniques, it requires no explicit mode-switching to invoke and can be smoothly used together with commonly adopted interaction styles such as direct touch and dragging. Our user studies show that BezelCursor requires less grip adjustments, and is more accurate or faster than the state-of-the-art techniques when using a fixed secure grip.

2. For the problem of inextensibility, which refers to the limited number of supported interaction methods, we motivate mobile interface designers to use bezel-initiated gestures for implicit mode-switching to support multiple touch interaction techniques in a limited space by presenting findings of an exploration of the design space of bezel-initiated gestures. It consists of a guessability study, which elicits end-user touch gestures on mobile devices. While previous studies give full freedom to the participants for gesture design, we apply a constraint that gestures must be initiated from the bezel, to ensure they are compatible with the mainstream gestures like dragging and pitching, and are highly detectable. We analyze the consensus among our participants, and reveal a set of user-defined bezel-initiated gestures including a number of novel designs. This serves as an important foundation in designing and customizing bezel-initiated gestures for extending existing interaction techniques.

3. For the problem of inaccuracy, which refers to the offset between user-intended touch position and the system-detected touch position, we present the idea of simultaneous co-refinement of multiple touch paths. We identify that inaccurate touch input is particularly problematic when drawing in a limited control space. It is because the canvas cannot be zoomed in too much or else the user will lose the overview of the drawing. We utilize the co-refinement idea to develop an assistive image tracing system, EZ-Sketching, to automatically refine sketch lines roughly traced over single images  based on the image features being traced. Unlike existing edge snapping methods, which aim at optimizing individual strokes, we show that a co-refinement of multiple nearby strokes results in drawings that are more accurate. Our user studies confirm that the refined sketches show closer resemblance to the traced images and are often aesthetically more pleasing.

in, The Screen Media Reader: Culture, Theory, Practice, edited by Stephen Monteiro (New York and London: Bloomsbury, 2017).
Focusing less on material interaction the situational conditions of use, I describe the
different modes of watching moving images on the smartphone. developing the categories of the ambiance-screen and communication-screen to differentiate viewing and listening strategies found in my research, i demonstrate how smartphone users employ the screen alternately to keep themselves company and interact with those around them.
While my research does not encompass all forms of smartphone use, it nevertheless provides insights regarding the portable screen's adaptability in relation to multiple forms of moving image entertainment, from the amateur clip to full-length movies

Introduction It is now commonplace for infants and toddlers to inhabit household media environments characterised by the presence of multiple and mobile touchscreen devices. Within these contemporary haptic media habitats, young children are interfacing with tablet computers and mobile phones through both guided design and situated play in ways that were restricted by past computer interfaces such as keyboards, mice, and game controllers. A key dimension of young children's mobile media engagement and play centres on their embodied relations, and how these are shaped with and through the interfaces, materiality, and mobility of haptic media.

In this study, we investigated the effect of input voltage waveform on our haptic perception of electrovibration on touch screens. Through psychophysical experiments performed with eight subjects, we first measured the detection thresholds of electrovibration stimuli generated by sinusoidal and square voltages at various fundamental frequencies. We observed that the subjects were more sensitive to stimuli generated by square wave voltage than sinusoidal one for frequencies lower than 60 Hz. Using Matlab simulations, we showed that the sensation difference of waveforms in low fundamental frequencies occurred due to the frequency-dependent electrical properties of human skin and human tactile sensitivity. To validate our simulations, we conducted a second experiment with another group of eight subjects. We first actuated the touch screen at the threshold voltages estimated in the first experiment and then measured the contact force and acceleration acting on the index fingers of the subjects moving on the screen with a constant speed. We analyzed the collected data in the frequency domain using the human vibrotactile sensitivity curve. The results suggested that Pacinian channel was the primary psychophysical channel in the detection of the electrovibration stimuli caused by all the square-wave inputs tested in this study. We also observed that the measured force and acceleration data were affected by finger speed in a complex manner suggesting that it may also affect our haptic perception accordingly.

This book investigates young children’s everyday digital practices, embodied digital play, and digital media products – such as mobile applications, digital games, and software tools. The book provides a critical and collective perspective on the ways young children’s mobile media culture is currently being reshaped. The chapters draw on research that extends from the household to social media platforms and public spaces. Moving across these interconnected sites, this book explores how young children are currently configured as consumers, users, and subjects of mobile media technologies. These arrangements of media use are analysed through a conceptual lens of digital dexterity, which locates children’s capacities to use mobile media interfaces and digital products not simply in terms of physical skills or developmental capacities, but importantly, through the design and affordances of mobile technologies and touch-based interfaces, cultures of interactive play and digital parenting...

What is the relationship between the ‘haptic’ and the ‘tactile’ when it comes to media? We might question whether there is such a thing as ‘haptic media’; in other words, is there a type of media that invite the attention of one modality rather than another, or that foster certain types of interaction over others? If we were to speak about ‘haptic media’, to what extent does it engage directly (only) with touch, and to what extent does it involve some form of enhancement of another modality? In what ways can haptic media appeal beyond the visuocentric norm of the screen, and therefore to non-normate or disabled users? Further, to what extent does the haptic in particular benefit from ‘sensory substitution’, which is most usually of touch for vision in assisted living technologies for the blind, or of sound for touch for the deaf, for example? Certain historical instances of sensory substitution systems are discussed below, including Norbert Wiener’s ‘hearing glove’ and Bach-Y-Rita’s...

In this study, we have investigated the human roughness perception of periodical textures on an electrostatic display by conducting psychophysical experiments with 10 subjects. To generate virtual textures, we used low-frequency unipolar pulse waves in different waveform (sinusoidal, square, saw-tooth, triangle), and spacing. We modulated these waves with a 3kHz high frequency sinusoidal carrier signal to minimize perceptional differences due to the electrical filtering of human finger and eliminate low-frequency distortions. The subjects were asked to rate 40 different macro textures on a Likert scale of 1-7. We also collected the normal and tangential forces acting on the fingers of subjects during the experiment. The results of our user study showed that subjects perceived the square wave as the roughest while they perceived the other waveforms equally rough. The perceived roughness followed an inverted U-shaped curve as a function of groove width, but the peak point shifted to t...

Future touch screen applications will include multiple tactile stimuli displayed simultaneously or consecutively to single finger or multiple fingers. These applications should be designed by considering human tactile masking mechanism since it is known that presenting one stimulus may interfere with the perception of the other. In this study, we investigate the effect of masking on the tactile perception of electrovibration displayed on touch screens. Through conducting psychophysical experiments with nine participants, we measured the masked thresholds of sinusoidal electrovibration bursts (125 Hz) under two masking conditions: simultaneous and pedestal. The masking signals were noise bursts, applied at five different sensation levels varying from 2 to 22 dB SL, also presented by electrovibration. For each participant, the thresholds were elevated as linear functions of masking levels for both masking types. We observed that the masking effectiveness was larger with pedestal masking than simultaneous masking. Moreover, in order to investigate the effect of tactile masking on our haptic perception of edge sharpness, we compared the perceived sharpness of edges separating two textured regions displayed with and without various types of masking stimuli. Our results suggest that sharpness perception depends on the local contrast between background and foreground stimuli, which varies as a function of masking amplitude and activation levels of frequency-dependent psychophysical channels.

What is the relationship between the ‘haptic’ and the ‘tactile’ when it comes to media? We might question whether there is such a thing as ‘haptic media’; in other words, is there a type of media that invite the attention of one modality rather than another, or that foster certain types of interaction over others? If we were to speak about ‘haptic media’, to what extent does it engage directly (only) with touch, and to what extent does it involve some form of enhancement of another modality? In what ways can haptic media appeal beyond the visuocentric norm of the screen, and therefore to non- normate or disabled users? Further, to what extent does the haptic in particular benefit from ‘sensory substitution’, which is most usually of touch for vision in assisted living technologies for the blind, or of sound for touch for the deaf, for example? Certain historical instances of sensory substitution systems are discussed below, including Norbert Wiener’s ‘hearing glove’ and Bach-Y-Rita’s tactile–visual sensory substitution (TVSS) system, to make a larger argument about the role of haptic technologies, and haptic media, for more inclusive digital interactions.

This study uses an Interactive Genetic Algorithm (IGA), a design space searching method, to determine the degree to which a user’s current mobile phone impacts their design preference, and how features in a product can change preference.  IGAs mimic evolution by iteratively converging towards a design while exploring a design space through random mutations.  20 participants, 10 current Apple iPhone owners, and 10 non-iPhone owners were asked to use a web-based IGA tool to design touchscreen and non-touchscreen phones for dialing use only.  Similar to other IGA mobile phone work (Nathan-Roberts & Liu 2010), the IGA varied screen size, button spacing, and phone radius independently.  Results showed iPhone users, and non-iPhone users did have different design preferences, but that there was a bigger difference between touchscreen phone owners (iPhone and non-iPhone touchscreen phones), and non-touchscreen phone owners.  Overall participants had significantly different preferences for touchscreen and non-touchscreen designs for all variables except for the vertical button spacing, and phone radius.  This work is part of a larger research study of aesthetic ergonomics of mobile phones, specifically looking at usability, and the capacity of users to combine multiple goals in design.  Future research needs are discussed, including further testing the effect of non-iPhone touchscreen phone ownership.

Emotion is a key aspect of user experience. To design a user interface for positive emotional experience, the affective quality of the user interface needs to be carefully considered. A major factor of affective quality in today’s user interface for digital media is interactivity, in which motion feedback plays a significant role as an element. This role of motion feedback is particularly evident in touchscreen user interfaces that have been adopted rapidly in mobile devices. This paper presents two empirical studies performed to increase our understanding of motion feedback in terms of affective quality in mobile touchscreen user interfaces. In the first study, the relationships between three general motion properties and a selected set of affective qualities are examined. The results of this study provide a guideline for the design of motion feedback in existing mobile touchscreen user interfaces. The second study explores a new dimension of interactivity that is the Weight factor of Laban’s Effort system. To experiment the Weight factor in a mobile touchscreen user interface, a pressure sensitive prototype was developed to recognize the amount of force applied by the user’s finger action. With this prototype, the effects of implementing pressure requirements on four different types of user interfaces were examined. Results show that implementing the Weight factor can significantly influence the affective quality and complement the physical feel of a user interface. The issues to consider for effective implementation are also discussed.