Touchscreen

The latency of current mobile devices' touchscreens is around 100ms and has widely been explored. Latency down to 2ms is noticeable, and latency as low as 25ms reduces users' performance. Previous work reduced touch latency by extrapolating a finger's movement using an ensemble of shallow neural networks and showed that predicting 33ms into the future increases users' performance. Unfortunately, this prediction has a high error. Predicting beyond 33ms did not increase participants' performance, and the error affected the subjective assessment. We use more recent machine learning techniques to reduce the prediction error. We train LSTM networks and multilayer perceptrons using a large data set and regularization. We show that linear extrapolation causes an 116.7% higher error and the previously proposed ensembles of shallow networks cause a 26.7% higher error compared to the LSTM networks. The trained models, the data used for testing, and the source code is available on GitHub.

Since the rise of radio and television, we have lived in an era defined increasingly by the electronic circulation of images and sounds. But the flood of new computing technologies known as haptic interfaces—which use electricity, vibration, and force feedback to stimulate the sense of touch—are promising to change this situation, offering an alternative way of mediating and experiencing reality.

In Archaeologies of Touch, David Parisi offers the first full history of these increasingly vital technologies, showing how the efforts of scientists and engineers over the past 300 years have gradually remade and redefined our sense of touch. Through lively analyses of electrical machines, videogames, sex toys, sensory substitution systems, robotics, and human-computer interfaces, Parisi shows how the materiality of touch technologies has been shaped by attempts to transform humans into more efficient processors of information.

With haptics becoming ever more central to emerging virtual reality platforms (immersive bodysuits loaded with touch stimulating actuators), wearable computers (haptic messaging systems like the Apple Watch’s Taptic Engine), and smartphones (vibrations that emulate the feel of buttons and onscreen objects), Archaeologies of Touch offers a timely and provocative engagement with the long history of touch technology that helps us confront and question the power relations of giving touch its own set of technical media.

Touchscreen technology and devices that use that technology have become widely popular over the past few years. Even so, very small amount of people knows that there are different technologies implemented in those devices that they use in everyday life. Through this seminar, we would try to present the different solutions currently available on the market. Simultaneously, we will compare between those solutions, emphasizing on differences between those technologies, and making the differences between them more obvious and understandable. Later, we will give our opinions on market trends in the future, and our vision and conclusion about this topic .

Open Access (download for free): https://meson.press/books/touchscreen-archaeology/

The touchscreen belongs to a century-long history of hands-on media practices and touchable art objects. This media-archaeological excavation examines the nature of our tactile involvement with media and invites the reader to think about the touchscreen beyond its technological implications. In six chapters, the book questions and historicizes both aspects of the touchscreen: "touch" as a media practice and "screen" as a touchable object. Taking cue from the so-called rube figure of early cinema, it looks at various instances of proper and improper screen touching and discusses the tecnicity of hand gestures, such as thumbing a flipbook, pinching a tablet, and caressing a projection. Finally, it proposes to consider the image-on-screen as a touchable object similar to the touchable surface of the screen itself, for which the notion of "image+" is introduced. Picked up from image processing software, this term refers to today’s screenic image that goes beyond its visual appeal by adding an extra, often tactile or hands-on, dimension (e.g. the image as a hyperlink, as a gateway to something else, as a form of play).

With the recent publication of the edited collection Brian Eno: Oblique
Music (Albiez & Pattie, 2016) and the distribution of Brian Eno’s
26th solo studio album Reflection in 2017 on vinyl, CD and as an
innovative software application, the time is ripe for a reconsideration of
the way in which software has been used by the musician, composer,
record producer and visual artist Brian Eno. This paper explores how
Eno has used simple but innovative ideas and processes to inform his
music over the course of his career, and considers how his work with
collaborators – specifically the musician and software designer Peter
Chilvers – has converged with the emergence of touchscreen
technologies and modes of distribution. We will demonstrate how
Apple’s App Store global distribution platform has further disseminated
Eno’s ideas of ‘generative music’ to a wider audience through he and
Chilvers’ Bloom (2008), Trope (2009/2015), Scape (2012)
and Reflection (2017a) software applications for Apple’s iPhone and
iPad, and the impact on the distribution and reception of Eno’s own
music. Echoing Eno’s own processes of appropriation, remix and
collaboration the authors wrote the paper through exchanges dictated by
the turn of a card selected from the third edition of the Oblique
Strategies deck, issued by Eno and Peter Schmidt in 1979.

This posting of an annotated bibliography on gesture user interfaces, on-line character recognition (a.k.a.  dynamic character recognition, a.k.a. pen and touch computing), both hardware and software, has been a continuing work-in-progress since the 1980s. It includes information on related application topics, such as digital rights management (DRM), portable computing, cryptographic communications, and biometric authentication. I am posting it as a service to those with interest in the field. It may also be of special interest to anyone investigating any of the areas of digitizer tablets, touchscreens, character recognition, touch/gesture user interfaces, multi-touch computing, touch and proximity sensors, augmented reality, haptics, context-dependent intrepretation of user input, and applications including the same. It covers the time period from approximately 1891 (first electronic tablet) through 1914 (first gesture/handwriting-recognition input system) to the present day.

Background: Dementia can have significant detrimental impacts on the well-being of those with the disease and their carers. A range of computer-based interventions, including touchscreen-based interventions have been researched for use with this population in the hope that they might improve psychological well-being. This article reviews touchscreen-based interventions designed to be used by people with dementia (PWD), with a specific focus in assessing their impact on well-being.

Direct touch is the major interaction model of modern mobile touchscreen devices. Although it is praised for its intuitiveness and input speed, its suffers from three major problems, which are inaccessibility, inextensibility, and inaccuracy, due to the limited control space of touch interaction.

In this thesis, we present three contributions, each addressing one of the three major problems:

1. For the problem of inaccessibility, which refers to the inability of interacting in the distant out-of-reach screen area when using one-handed or two-handed thumb interactions, we present BezelCursor. It is a novel one-handed thumb interaction technique that improves screen accessibility in the context of target acquisition. It combines bezel-initiated interaction and pointing gesture to control a screen cursor. Unlike the existing techniques, it requires no explicit mode-switching to invoke and can be smoothly used together with commonly adopted interaction styles such as direct touch and dragging. Our user studies show that BezelCursor requires less grip adjustments, and is more accurate or faster than the state-of-the-art techniques when using a fixed secure grip.

2. For the problem of inextensibility, which refers to the limited number of supported interaction methods, we motivate mobile interface designers to use bezel-initiated gestures for implicit mode-switching to support multiple touch interaction techniques in a limited space by presenting findings of an exploration of the design space of bezel-initiated gestures. It consists of a guessability study, which elicits end-user touch gestures on mobile devices. While previous studies give full freedom to the participants for gesture design, we apply a constraint that gestures must be initiated from the bezel, to ensure they are compatible with the mainstream gestures like dragging and pitching, and are highly detectable. We analyze the consensus among our participants, and reveal a set of user-defined bezel-initiated gestures including a number of novel designs. This serves as an important foundation in designing and customizing bezel-initiated gestures for extending existing interaction techniques.

3. For the problem of inaccuracy, which refers to the offset between user-intended touch position and the system-detected touch position, we present the idea of simultaneous co-refinement of multiple touch paths. We identify that inaccurate touch input is particularly problematic when drawing in a limited control space. It is because the canvas cannot be zoomed in too much or else the user will lose the overview of the drawing. We utilize the co-refinement idea to develop an assistive image tracing system, EZ-Sketching, to automatically refine sketch lines roughly traced over single images  based on the image features being traced. Unlike existing edge snapping methods, which aim at optimizing individual strokes, we show that a co-refinement of multiple nearby strokes results in drawings that are more accurate. Our user studies confirm that the refined sketches show closer resemblance to the traced images and are often aesthetically more pleasing.

In this paper, I introduce Sfard's discursive framework to examine secondary-school calculus students' communication during exploratory activities mediated by the use of touchscreen dynamic geometry environments (DGEs). Six pairs of secondary-school students participated in an open-ended task to explore calculus relationships using touchscreen-DGEs. Qualitative data capturing the students' linguistic communication (speech) and hand movements (gestures and dragging) were analysed when the students interacted with the touchscreen-DGEs used during the task. Findings suggest that new forms of communication were mobilised through the act of dragging on a touchscreen-DGE. In particular, routines for comparing, reasoning, conjecturing and verifying emerged within the use of the haptic DGE interface. In this paper, potentials of the touchscreen-DGEs in facilitating new forms of gestural thinking , as well as theoretical and methodological considerations that recognise the changing ways in which the hand and media interact, are discussed.

In this exploratory case study, we describe how teacher candidates with minimal mathematics knowledge of spherical geometry construct, discuss, manipulate and share the three-right angled triangle on a sphere using GeoGebra software. The study aims to demonstrate how mathematical understanding can begin with an object construction and manipulation in Dynamic Geometry Environment (DGE). Our ideas emerge from teacher candidates' use of the touchscreen-based computers mounted in a concept classroom following by embedding constructions in Canvas– Learning Management System (LMS), aiming to develop their understanding of spatial ability and spherical geometry. In this paper, we are not focusing on trigonometric laws, or other mathematical properties of the spherical triangle, instead we are introducing the construction of spherical triangle to not mathematics or science major teacher candidates. Some theory and techniques, including angel measurements on the sphere in a dynamic way are also learned along the way. Adapting Arzarello et al.'s (2014) theoretical framework of touch interactions and using Nemirovsky et al.'s (2013) perceptuomotor integration approach, augmented by a focus on the role of the bodily interactions in the mathematical learning with a touchscreen device, we assess and analyse how mathematical learning occurs when learners interact with touchscreen-based technologies and their peers. The result of the study showed that the mathematical construction and manipulation along with touchscreen-based interactions play a crucial role in developing spatial skills, which consequently triggers dynamic forms of mathematical communication. We also present evidence of collaborative engagement where teacher candidates engage in angle measurement of spherical triangles.

My article on the Nintendo DS 'Touching is Good' campaign, recently updated and reprinted in Figures of Touch: Sense, Technics, Body (The Academy of the Arts at the University of the Arts Helsinki, 2018).

I deepen and extend this argument a bit in the fifth chapter of Archaeologies of Touch, expanding it to include the ad campaigns for other mobile touchscreens like the iPhone.

This paper presents Finger Based Technique (FBT) prototypes, a novel interaction system for blind users, which is especially designed and developed for non-visual touch screen devices and their applications. The FBT prototypes were developed with virtual keys to be identified based on finger holding positions. Two different models namely the single digit FBT and double digit FBT were propounded. FBT technique were applied using two different phone dialer applications: a single digit virtual key for the single digit FBT model and a double digit virtual key with audio feedback enabling touch as input gesture for the later one. An evaluation with 7 blind participants showed that single digit FBT was significantly faster and more accurate than double digit FBT. In addition to that, single digit FBT was found to be much faster than iPhone VoiceOver entry speeds in performing similar tasks. Furthermore, our research also suggests 11 accessible regions for quick access or navigation in flat touch screen based smart phones for blind users. These accessible regions will serve as a usability design framework and facilitate the developers to place the widget for the blind user for dynamic interaction with the touch screen devices. As far as is known to the authors, this is a novel suggestion.

The article reports on the beginnings of a research project funded by the Australian Research Council entitled Toddlers and Tablets: exploring the risks and benefits 0-5s face online. This research project recognises that at this stage it is parents who “are the real experts in their toddlers’ use of screen technologies. Accordingly, the project’s methodological approach draws on parents, pre-schoolers and their families as communities of practice in the construction of social meaning around toddlers’ use of touch screen technology. Available online M/C Journal. 18 (5).

In this exploratory case study, we describe how teacher candidates with minimal mathematics knowledge of spherical geometry construct, discuss, manipulate and share the three-right angled triangle on a sphere using GeoGebra software. The study aims to demonstrate how mathematical understanding can begin with an object construction and manipulation in Dynamic Geometry Environment (DGE). Our ideas emerge from teacher candidates’ use of the touchscreen-based computers mounted in a concept classroom following by embedding constructions in Canvas– Learning Management System (LMS), aiming to develop their understanding of spatial ability and spherical geometry. In this paper, we are not focusing on trigonometric laws, or other mathematical properties of the spherical triangle, instead we are introducing construction of spherical triangle to not mathematics or science major teacher candidates. Some theory and techniques, including angel measurements on sphere in a dynamic way are also lear...

This paper analyses the variations in the identification power among set of four features, namely Signature Precision (SP), Finger Pressure (FP), Movement Time (MT), and Speed that were extracted from finger gesture of touch-screens. The differences across users were studied using the above features individually and combined for the purpose of user identification based on the Euclidean distance and the k-nearest neighbour classifier. The paper investigates the performance variations across users using a dataset of 50 users, and concludes that the discrimination power of different feature is heavily user dependant i.e. while some features achieve 100% identification accuracy for some users, they perform poorly for others. On small smartphone 19% and on mini-tablet 42% of the users have unique FP compared to other features. We concluded that FP could be used alone to perfectly identify users for 100% accuracy.

Developments in information and communication technologies have been greatly influential on the practices in all fields, and education is not an exception to this. To illustrate with, computers were first used in computer – assisted education in order to increase the efficiency of teaching process. Recently, computer has contributed more to the field through interactive and smart class applications that are specially designed for classroom use. The aim of this study is to develop a low – cost, portable and projection – supported touchscreen to be used in educational environments by using FPGA technology and to test its usability. For the purposes of the study, the above mentioned system was developed by using the necessary hardware and software, and later it was tested in terms of usability. This usability test was administered to teachers, who were the target end – users of this touchscreen writing / drawing system. The aim of this test was to determine "user – friendliness" , "subservientness" and "usability" of the system. Several tools were used to obtain data from the users that participated in the study. The analysis and evaluation of the data collected revealed that the system has achieved its objectives successfully.

This paper is about an investigation into the usability of touch-based user interfaces. Currently, not enough knowledge is available to guide user interface designers and developers concerning the appropriate use of touch-based technology. We adopt an empirical approach using an experiment to test the effectiveness and user satisfaction of touch-based interaction compared with equivalent mouse-based interaction. We had two abstract type tasks and one contextualised task using the two methods of interaction. We measured errors, task time and user satisfaction. The data was statistically analysed and the statistically significant results show that overall the mouse-based interaction was faster, caused fewer errors and was preferred by the participants. These results are interesting for all user interface designers and developers, where we make some design suggestions based on our empirical results. Further, we also propose further ways forward to enrich this research area of further knowledge.

With the ongoing debate on young children’s Information Communication Technology (ICT) use in early childhood education (ECE), empirical studies have reported that the increase in young children’s access to and use of touchscreen tablets, hereafter referred to as tablets, could positively and negatively impact their learning and development. According to the New Zealand ECE curriculum, Te Whāriki (Ministry of Education, 2017), children growing up in the context of a changing society could benefit from using technology. This research, which explored tablet use in New Zealand’s four major early childhood service types: education and care centres, home-based services, kindergartens, and playcentres, provides useful information on the reasons why services used and did not use tablets as well as how teachers/educators used tablets to support children’s learning.  The two phases to this sequential explanatory mixed methods study were underpinned by two research paradigms, the postpositivi...

The advent of system-wide accessibility services on mainstream touch-based smartphones has been a major point of inclusion for blind and visually impaired people. Ever since, researchers aimed to improve the accessibility of specific tasks, such text-entry and gestural interaction. However, little work aimed to understand and improve the overall accessibility of these devices in real world settings. In this paper, we present an eight-week long study with five novice blind participants where we seek to understand major concerns, expectations, challenges, barriers, and experiences with smartphones. The study included pre-adoption and weekly interviews, weekly controlled task assessments, and in-the wild system-wide usage. Our results show that mastering these devices is an arduous and long task, confirming the users' initial concerns. We report on accessibility barriers experienced throughout the study, which could not be encountered in task-based laboratorial settings. Finally, w...

Abstract: In this presentation, I will describe how the use of touchscreen-based technology can enable young children's sensory engagement in learning mathematics through introducing a novel iPad application called TouchCounts. TouchCounts connects touchscreen-based technology with finger-counting and encourages the child to use their fingers to summon objects (numbers) and associate specific gestures to numerical operations. Drawing upon Nemirovsky's perceptuomotor integration theoretical lens and other foundational aspects of Husserlian phenomenology, the study reports on a preschool child named Alex developing mathematical tool fluency and expertise. Overall, this case study demonstrates that one-on-one multimodal touch, sight and auditory feedback via a touchscreen device can serve to assist in a child's development of number sense.

Introduction Children are beginning to use digital technologies at younger and younger ages. The emerging trend of very young children (babies, toddlers and pre-schoolers) using Internet connected devices, especially touch screen tablets and smartphones, has elicited polarising opinions from early childhood experts. At present there is little actual research about the risks or benefits of tablet and smartphone use by very young children. Current usage recommendations, based on research into passive television watching which claims that screen time is detrimental, is in conflict with advice from education experts and app developers who commend interactive screen time as engaging and educational. Guidelines from the health professions typically advise strict time limits on very young children’s screen-time. Based for the most part on policy developed by the American Academy of Paediatrics, it is usually recommended that children under two have no screen time at all (Brown), and childr...

In this paper, I introduce Sfard’s discursive framework to examine secondary-school calculus students’ communication during exploratory activities mediated by the use of touchscreen dynamic geometry environments (DGEs). Six pairs of secondary-school students participated in an open-ended task to explore calculus relationships using touchscreen-DGEs. Qualitative data capturing the students’ linguistic communication (speech) and hand movements (gestures and dragging) were analysed when the students interacted with the touchscreen-DGEs used during the task. Findings suggest that new forms of communication were mobilised through the act of dragging on a touchscreen-DGE. In particular, routines for comparing, reasoning, conjecturing and verifying emerged within the use of the haptic DGE interface. In this paper, potentials of the touchscreen-DGEs in facilitating new forms of gestural thinking, as well as theoretical and methodological considerations that recognise the changing ways in which the hand and media interact, are discussed.

Adding expressive haptic feedback to mobile devices has great potential to improve their usability, particularly in multitasking situations where one’s visual attention is required. Piezoelectric actuators are emerging as one suitable technology for rendering expressive haptic feedback on mobile devices. We describe the design of redundant piezoelectric haptic augmentations of touchscreen GUI buttons, progress bars, and scroll bars, and their evaluation under varying cognitive load. Our haptically augmented progress bars and scroll bars led to significantly faster task completion, and favourable subjective reactions. We further discuss resulting insights into designing useful haptic feedback for touchscreens and highlight challenges, including means of enhancing usability, types of interactions where value is maximized, difficulty in disambiguating background from foreground signals, tradeoffs in haptic strength vs. resolution, and subtleties in evaluating these types of interaction...

The understanding and control of human skin contact against technological substrates is the key aspect behind the design of several electromechanical devices. Among these, surface haptic displays that modulate the friction between the human finger and touch surface are emerging as user interfaces. One such modulation can be achieved by applying an alternating voltage to the conducting layer of a capacitive touchscreen to control electroadhesion between its surface and the finger pad. However, the nature of the contact interactions between the fingertip and the touchscreen under electroadhesion and the effects of confined material properties, such as layering and inelastic deformation of the stratum corneum, on the friction force are not completely understood yet. Here, we use a mean field theory based on multiscale contact mechanics to investigate the effect of electroadhesion on sliding friction and the dependency of the finger–touchscreen interaction on the applied voltage and oth...

Figure 1. GazeForm concept. Left: when eye-hand coordination exists for touch modality the surface is flat. Right: when the gaze is solicited by another task the surface offers a salient tangible control allowing continuity of manipulation and freeing the gaze. ABSTRACT An increasing number of domains, including aeronautics, are adopting touchscreens. However, several drawbacks limit their operational use, in particular, eyes-free interaction is almost impossible making it difficult to perform other tasks simultaneously. We introduce GazeForm, an adaptive touch interface with shape-changing capacity that offers an adapted interaction modality according to gaze direction. When the user's eyes are focused on interaction, the surface is flat and the system acts as a touchscreen. When eyes are directed towards another area, physical knobs emerge from the surface. Compared to a touch only mode, experimental results showed that GazeForm generated a lower subjective mental workload and a higher efficiency of execution (20% faster). Furthermore, GazeForm required less visual attention and participants were able to concentrate more on a secondary monitoring task. Complementary interviews with pilots led us to explore timings and levels of control for using gaze to adapt modality.

What is the relationship between the ‘haptic’ and the ‘tactile’ when it comes to media? We might question whether there is such a thing as ‘haptic media’; in other words, is there a type of media that invite the attention of one modality rather than another, or that foster certain types of interaction over others? If we were to speak about ‘haptic media’, to what extent does it engage directly (only) with touch, and to what extent does it involve some form of enhancement of another modality? In what ways can haptic media appeal beyond the visuocentric norm of the screen, and therefore to non- normate or disabled users? Further, to what extent does the haptic in particular benefit from ‘sensory substitution’, which is most usually of touch for vision in assisted living technologies for the blind, or of sound for touch for the deaf, for example? Certain historical instances of sensory substitution systems are discussed below, including Norbert Wiener’s ‘hearing glove’ and Bach-Y-Rita’s tactile–visual sensory substitution (TVSS) system, to make a larger argument about the role of haptic technologies, and haptic media, for more inclusive digital interactions.