Keywords

1 Introduction

Interfaces can be said to be intermediaries for communication between objects and humans. Especially, the interface called Human-Compute Interaction (HCI) for the interaction between computer and user has been applied to various fields as the computer technology has been developed. As a method for interacting with a computer, a method of recognizing a user’s hand or finger contact is widely used. Hand gestures and finger touch recognition can also be done using a touch screen or sensors attached to the hands or arms separately. However, the method using camera has been studied in HCI field. Among them, interfaces using RGB cameras have been studied for a long time, and many methods of using depth camera and infrared camera have been studied. Such cameras may not be suitable depending on the user environment or the recognition performance. Thermal cameras are often used in special applications because of their high cost and size. Recently, as the price has decreased and the size has become smaller, the thermal cameras are utilized in various fields. The thermal camera can detect the temperature by sensing thermal energy radiation. With this feature, the thermal camera is used to detect the motion of user’s hands and to sense the residual heat generated by fingertip touches on the target surface. A touch interface using a thermal camera is a method for detecting a residual heat generated by a user. When a user touches a surface with a hand or a fingertip, the user’s body temperature is conducted to the contact surface, which generates residual heat on the surface. The sensed residual heat provides some information of position on the surface, movement and gestures of hands, and so on. The residual heat can be generated by breath instead of the fingertip contact. In this case, the users do not need to use their hands for interaction.

This paper deals with the study of interfaces that do not require hands using thermal cameras and breath residual heat. Most HCI interfaces for computer-user interaction are based on fingertip touch or hand movements. However, for users who cannot use their hands temporarily or permanently, there is restriction on the use of the HCI interface. As an alternative method, the interface without hands would be useful for users who have difficulty in using their hands. There is a disadvantage in that it is necessary to be close to the target surface in order to generate residual heat by breath. Also, since the temperature of residual heat by breath is usually lower than that of fingertip contact, it is difficult to distinguish the residual heat by breath from fingertip heat. A hollow rod (straw) was used to increase the residual heat by breath at a distance from the interaction surface. If a user blows the hollow rod with user’s mouth, breath will be concentrated and the residual heat temperature will increase on the surface.

In this paper, various experimental evaluations were conducted to verify the feasibility and usefulness of interface method using residual heat by breath. Experimental evaluation was carried out by comparison of residual heat generation methods and various materials surfaces. According to the experimental results, the residual heat generation method using the hollow rod (straw) is most stable in the aspect of high temperature and detection performance.

The remainder of paper is organized as follows. In Sect. 2, some related works are briefly summarized. The characteristics of residual heat by breath is described in Sect. 3, and various comparative observation on residual heat generation methods and surface materials is explained in Sect. 4. Section 5 deals with the implemented interaction system for a hand-free interaction application. Finally, some problems and future researches of proposed system are discussed in Sect. 6.

2 Related Works

Researches on interfaces should take into account the means of interaction and recognition. There is fingertip contact as a typical interaction means. Fingertip contact is a system that touches a wall or table surface with a touch screen and sensor device [1, 14]. In this paper, we have studied camera based surface touch interface.

Types of cameras include a visual camera that captures visible light, an infrared camera that captures near infrared rays, and a thermal camera that captures the energy emitted from far infrared bands. Visual cameras are dependent on the sun or illumination because they can only represent the wavelengths of visible light. Therefore, visual camera based user interaction system has been studied extensively. However, when the intensity of illumination is high or low, comparison of skin color and detection of contour of hand become difficult [2, 8]. The infrared camera is a camera that captures the near infrared rays of 0.7 μm to 1.3 μm wavelength near infrared rays and reflects the object. Compared to a visual camera, it has a low illumination and has the advantage of viewing objects in dark environments. In the case of fingertip or hand shape, it is widely used because it can be seen more clearly than visual camera due to the retro-reflective characteristics of the object. Especially, when using a projector or a screen in a tabletop interaction study, it is easy to use a projector or an LED screen because it is less influenced by illumination [6, 13, 17]. However, infrared cameras require an infrared light source. And when the infrared camera is used outdoors in the daytime, there is a disadvantage that the infrared light source is affected by the sun light.

Usually the infrared camera is a near-infrared camera and the far-infrared camera is called a thermal camera. The thermal camera senses the far-infrared energy of 3 μm–14 μm wavelength band emitted by the object based on absolute temperature 0. Thermal cameras do not need a separate light source because they capture the heat energy emitted by the object. Therefore, there is no problem in using at night and daytime and outdoors. Figure 1 shows an image captured with an infrared camera at night in an outdoor space. The initial thermal imaging camera was developed for use in military sector applications [4]. In recent years, there have been many studies on prevention of overheating in industrial sites, confirmation of cracks, detection of nighttime pedestrians [3], face recognition [9], study of human skin surface change [12], medical field [7, 15], food quality and safety profiling [5], has been utilized.

Fig. 1.
figure 1

Thermal image

Full size image

In general, the temperature of the hands or fingers is higher than the room temperature, and researches using thermal cameras have been actively conducted [10, 11, 16]. Because heat has the property of being conducted, the user can contact the surface to transmit heat. The heat transferred to the surface causes the contact surface to show a higher temperature than the ambient. We call this a residual heat (see Fig. 2). When the fingertip is brought into contact with the surface to generate residual heat, the contact position can be extracted.

Fig. 2.
figure 2

Thermal image of residual heat

Full size image

3 Hands-Free Interface Using Breath Heat

The touch interface using the thermal camera is similar to the existing touch interface [10, 11]. The difference is how to find the tip of the finger that touches surface. The touched trace is called the residual heat, which can be found using the thermal camera. The proposed hands-free interface interacts by sensing the residual heat without using a hand. This paper exploits breath to generate sufficient residual heat.

3.1 Residual Heat by Breath

The residual heat generated by fingertip touches can be stably interacted with the thermal camera without being affected by the background or illumination. However, the users who cannot use their hands cannot do this interface.

This paper studies how to interact through the breath heat. The temperature of breath is usually higher than room temperature (on average between 20 and 24℃). Therefore, it is possible to detect residual heat on the surface by blowing breath. The use of breath can generate residual heat on the interaction surface without the use of hands. However, when we look at the image of the residual heat due to the breath in Fig. 3, the temperature is low and spread widely. This is because of spreading of breath air. There are two problems when we breath out. One is that the breathing air has low temperature. The other is that the warm breath air is spread out quickly. The temperature deviation of the breath heat is large due to the problems and users difference. The more the distance between the user and the surface, the more the breath spreads. And as the breath passes through the air, the temperature of breath also decreases. In order to use the residual heat by breath, it is necessary to collecting the breath air and to be close to the surface. However, when the user is very close to the surface, it may be difficult to detect residual heat at the target surface position by optical occlusion.

Fig. 3.
figure 3

Residual heat by usual breath (left: RGB image, right: thermal image)

Full size image

3.2 Residual Heat Using Straw

In this paper, we use hollow rod (straw) to collect breath heat and to generate sufficient residual heat at the position intended by the user. If you use a straw to breath, the temperature of the residual heat increases because of the concentration of warm air. And since it is 25 cm to 30 cm away from the surface, it is easy to generate residual heat at the target position. Figure 4 shows the generating residual heat using a straw. The users who have difficulty in using their hands can generate residual heat on the target surface exactly.

Fig. 4.
figure 4

Generating residual heat by using straw (left: RGB image, right: thermal image)

Full size image

4 Experimental Analysis of Residual Heat

In order to effectively generate the intended residual heat, basic research on the generation of residual heat is required. The experiments were conducted to generate and detect residual heat by various methods in order to apply it to thermal camera and application design. In the experiments, the temperature of the residual heat was checked and compared right after generation of residual heat. The experiments were conducted indoors, where the temperature was between 20 and 24℃. The participants in the experiments were eight in total, four men and four women. The thermal camera used for the experiments is the VarioCAM hr head 420 (see Fig. 5). The image resolution is 384 × 288 pixels.

Fig. 5.
figure 5

Thermal camera

Full size image

4.1 Residual Heat Generation Methods

We conducted three ways to generate residual heat on the surface. The three methods are fingertip touch, blowing breath, and blowing breath with a hollow rod (straw). The experiments generated residual heat on the paper attached to the wall. Figure 6 shows three methods to generate residual heat on the surface.

Fig. 6.
figure 6

Generation residual heat left: RGB image, right: thermal image

Full size image

Figure 7 shows a graphical representation of the three types of residual heat peak temperatures (Participants 1). The temperatures of residual heat by fingertip contact are indicated by the blue point and are distributed between 28.17° and 29.93℃. The residual heat temperatures of blowing breath are indicated by the red points and are distributed between 27.69° and 31.64℃. The residual heat temperatures of blowing breath through the hollow rod (straw) are indicated by the green points and are distributed between 31.57° and 34.09℃. As we can see in Fig. 7, the temperatures of the residual heat of blowing breath using the hollow stick are highest among the three ways. In addition, the residual heat exists in the small area. Therefore, the blowing breath with straw is most suitable for detecting with thermal cameras and localizing its position. There are no occlusion problems since the users blow their breath using the straw distant from the surface.

Fig. 7.
figure 7

Residual heat temperature comparison graph (generation methods)

Full size image

4.2 Heat Response Characteristics of Materials

The temperature of residual heat varies differently depending on the materials. There is also a temperature difference in the residual heat because the thermal conductivity is different. It reflects the radiated thermal infrared radiation according to the characteristics of the surface [16]. Considering this point, we selected six materials that are expected to be widely used as surfaces. The types of surface materials are paper, acrylic, canvas paper, foam board, Iron plate, and MDF. Each surface material was separated into fingertip contact and breath and breath through the hollow rod to generate residual heat. Figure 8 shows the residual heat generated from each surface material.

Fig. 8.
figure 8figure 8

Thermal image of residual heat (surface material in order from above: acrylic, canvas paper, form board, iron plate, MDF) left: fingertip, right: breath using the hollow rod

Full size image

The highest temperature values of residual heat generated on the each surface material were classified as fingertip contact and breath through the hollow rod (see Fig. 9). Figure 9-(a) is the fingertip contact and Fig. 9-(b) is the breath with hollow rod. Figure 9 shows that the temperatures of paper and canvas are relatively high. In the case of paper, the most of temperatures is above 30°. In case of iron plate, it shows very low temperature. In the case of iron plat, since the conductivity is high, the fingertip or the high residual heat is immediately transferred to the periphery of the iron plat and the surrounding air. In order to maintain sufficiently high residual heat, long time touch or breath with hollow rod are appropriate. In the experiment, the residual heat generation time is 1.5 s to 2 s.

Fig. 9.
figure 9

Residual heat temperature comparison graph (surface materials)

Full size image

4.3 User Characteristics of Heat Generation

According to the previous experiments, we found that the temperature distribution of residual heat by fingertip contact was different from each user. This difference is due to the characteristics of the user’s physical conditions. People with cold hands have a low temperature of contact residual heat, and vice versa. As a result, some users may fail to generate sufficient residual heat by fingertip contact. However, the temperature inside the human body is almost the same for all users and remains constant. Therefore, the heat generated by internal body heat can generate high residual heat. Figure 10-(a) shows the residual heat temperature difference for two classes of users.

Fig. 10.
figure 10

Residual heat temperature comparison graph (user characteristics)

Full size image

The two types of user temperature distribution according to the residual heat generation method can clearly distinguish the temperature distribution difference in fingertip contact (see Fig. 10-(b)). It can be seen that the difference in temperature distribution is small in case of using hollow rod (straw). Some temperature values are high or low because the participant unconsciously blows strongly or the finger contact time and intensity are not exactly the same.

4.4 Conclusions and Discussion

We conclude the following conclusions from three experiments. First, the residual heat temperature of the breath using the hollow rod (straw) was higher than the residual heat temperature due to the fingertip contact. Second, the residual heat of fingertip may not be detectable because the temperature is not consistent depending on the characteristics of the users. However, the residual heat of the breath using the hollow rod (straw) is consistent regardless of users. Third, there is the difference of residual heat temperatures with respect to the surface materials. Since the user generates residual heat for a short time, it is advantageous for surface interaction to use paper and canvas paper as surface materials.

Participants gave some comments on finger contact. Especially, in the case of steel plate, the fingertips become colder as the fingertip contact is repeated. The canvas paper and the foam board were said to have a low sense of heterogeneity with soft touch.

5 UI Application Using Residual Heat

We proposed a method that interactive interface using residual heat through a hollow rod for users who are not able to use their hands. In the previous section, residual heat experiments were performed on various surface materials. Experiments have shown that the residual heat by using the hollow rod is suitable for surface interaction.

5.1 Surface Interaction System

In this paper, we have implemented a prototype application to test the usefulness of the proposed interaction technique. The surface-based interaction used a projection-camera system. When the user creates residual heat on the surface of the paper material attached to the wall, the projector projects another image to the residual heat generating position. The conversion from the camera coordinates and the projected coordinates is accomplished by using a four-point homogeneous transformation. The application displays an historical old map at the selected location when the user views the satellite map projected onto the projector and generates residual heat at the desired location. Figure 11 shows a demonstration of a surface based interaction application. Recognizing the residual heat of the breath, it was confirmed that the user indicating location is exactly specified and interacted. Since it is possible to indicate a precise narrow position through a hollow rod to a high residual heat, it has been confirmed that a natural interface is possible without the need of hand movement.

Fig. 11.
figure 11

Demonstration of application

Full size image

6 Conclusion and Future Work

In this paper, we have proposed a method for user interaction without the use of a hand by using a thermal camera. This method can generate the residual heat with the use of a hollow rod, and detect it with a thermal camera. Thus it can interact without using a hand. To verify that the breath from hollow rod is efficient for generating residual heat, surface materials and residual heat generation methods were examined. Experimental results have shown that the proposed method enables the user who has difficulty in using their hands to interact with the same accuracy and naturalness as by touching them with their fingers. And since thermal cameras are not affected by background and illumination changes, they can be used in various environments. We have developed an application program to demonstrate the usefulness of the proposed interaction system.

The prototype system, which is made as an application example, includes a thermal imaging camera and a projector placed behind the user, so the projector image is hidden when the user generates residual heat. If the user generates residual heat and does not moved, the thermal camera cannot detect the residual heat generated on the surface. In the future work, we will improve the table top based interaction system using the rear screen. In addition, we plan to conduct research on user interfaces to enable various manipulations and interactions.