The HoverMesh

The HoverMesh: A Deformable Structure Based on Vacuum Cells [New Advances in the Research of Tangible User Interfaces] Andrea Mazzone Christian Spagno Andreas Kunz Center for Product Development Swiss Federal Institute of Technology Zurich, Switzerland Center for Product Development Swiss Federal Institute of Technology Zurich, Switzerland Center for Product Development Swiss Federal Institute of Technology Zurich, Switzerland mazzone@imes.mavt.ethz.ch spagno@imes.mavt.ethz.ch kunz@imes.mavt.ethz.ch ABSTRACT In this paper we propose a novel attempt to develop a spatial tangible user interface (TUI) [1] based on a deformable structure, the so-called HoverMesh. It consists of a stiff cubical, whose upper wall is composed of a deformable mesh of particle filled inflatable cells. This mesh can be deformed by inflating and/or deflating the cubical while consolidating (evacuating) and/or releasing (inflating) the cells. The HoverMesh is both an input and output device and we see its major benefit in the wide interaction area. The haptic feedback modality is thus embedded as well. The first results in our early experiments sustain the concept of a mesh based on inflatable cells. Keywords Smart structures, active structures, SmartMesh, HoverMesh, haptic displays, free hand haptic device, human computer interface, tangible user interface 1. INTRODUCTION Compared to the development of computational speed or the increase in the amount of memory over the last decades [2], the development of more powerful human computer interfaces (HCI) is lagging behind. Today’s broadly used output devices are practically limited to the visual (screen) and auditory (speakers) modality, while the keyboard and the mouse, practically still the same as in the seventies, literarily are dominating the community of input interfaces. The technological improvement of these tools has not yet satisfied the demand for more advanced multi modal interface devices. Interestingly, this demand has not only been growing strongly for their employment in some dedicated and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACE'04, June 3-5, 2004, Singapore Copyright 2004 ACM 1-58113-882-2/04/0006 ...$5.00. 187 specialized work environments. As more intuitive interfaces in daily used applications, such as computer entertainment for instance, they are sought after as well. The reason for this strong demand lays in the difference between the way we interact with the real world surrounding us and the way we are forced to interact with the digital world. Since childhood, human intuitively perform actions such as manipulating, assembling and disassembling objects but these innate spatial and tactile abilities unfortunately are tied by the technical limitations of today’s HCIs. Therefore, enormous effort is being put into the research for more intuitive user interfaces that address those spatial and tactile abilities and some great steps have been done: a growing number of interfaces incorporating haptic feedback capabilities is being employed in many specialized tasks. The Phantom [5], the CyberGrasp [7] or the the Haptic Master [6] for instance, are successfully used for research purposes or as advanced user interfaces in simulators for the aviation, for military or medical applications. Even in the low end sector very few devices can be found. One example is Logitech’s WingMan for computer entertainment. One further step towards bridging the afore mentioned limitation would be the development of a spatial tangible user interface. Ullmer and Ishii define TUIs as ”devices that give physical form to digital information, employing physical artifacts as representations and controls of the computational data” [8]. Sharlin et al. focus on a subgroup of TUIs, which they call ”spatial TUIs” and which mediate interaction with shape, space and structure. [1]. For this reason we have been focusing on smart structures as we see in them one possible way to successfully realize such kind of spatial tangible user interface (according to Spillman’s group, a smart structure is ”a non-biological physical structure that has: (1) a definite purpose, (2) means and imperatives to achieve that purpose, (3) a biological pattern of functioning.” [9]). A structure capable of deforming itself (thus simulating the objects themselves or at least the sections of interest) and simultaneously being able to read the deformations applied by the user would be a great step towards a novel spatial TUI. This kind of interface device would not only benefit industrial applications, but also the computer game industry for instance. They could even initiate a new generation of games, where 3D interaction is required. In amusement parks walls of these type of structures could replace today’s projection screens enhancing the 3D effects of the projections (we will shortly explain this in section 6), increasing the effect of reality. The HoverMesh introduced in this paper is our contribution to the development of an active deformable structure offering output and input capabilities. 2. BACKGROUND OF THE HOVERMESH The SmartMesh [3, 4] was a first attempt to realize an active deformable structure, affording simultaneously output and input capabilities. The mechanical structure, consisting of nodes and linkages, can be deformed by only shortening or elongating the specific linkages. The analysis of the degrees of freedom using the Kutzbach criterium [12, 13] has shown that for each degree of freedom of the SmartMesh a linkage with one prismatic joint is present proving that the structure is controllable. This has been achieved by positioning the linkages in a determined way and by integrating mechanical constraints. The double layer of the structure not only ensures a better distribution of the arising forces and torques but also provides the capability of controlling the movement in the third dimension. Figure 1(a) shows a simulation of the deformed structure. By actuating each of its linkages, the SmartMesh can actively be deformed to represent the desired object, which can then be touched. A prototype with 4x4 nodes was built (Figure 1(b)). The linkages can be elongated or shortened manually and fixed with a screw at the desired elongation. Even though the resolution is very small, the structure already shows its inherent capabilities of deforming itself to represent surfaces, including overhanging ones. The capabilities of the SmartMesh are very promising, but it seems quite difficult to develop a structure with more than 10 by 10 nodes with today’s actuation technology. The reason is, that the number of actuators grows linearly with the number of nodes (per node 4 actuators), resulting in a strong increase of weight. This leads again to higher requirements in actuation force, while the size of the actuators remains unchanged. Smart materials, especially electro active polymers [10], promise to be a solution due to their very high energy density. Our novel concept for a spatial user interface, the so-called HoverMesh, is also based on the idea of a deformable structure similar to the SmartMesh but with a completely different working principle. The benefits of this novel concept are mostly the small number of actuators that have to be employed, the light weight structure as well as the possibility to achieve a higher resolution (with today’s actuation technology). 3. THE HOVERMESH The HoverMesh is a cubical, whose upper wall has been designed as a deformable mesh of inflatable cells. Figure 2(a) gives a schematic overview of the complete system. The cubical can be pressurized or evacuated forcing the deformable mesh to deform depending on the structural state of each single cell. The cells contain particles of polystyrene and can be evacuated individually, thus compressing the particles. The compressed particles form a rigid structure, freezing the cells in their actual shape. We will detail this principle in section 3.1. Figure 2(b) shows a cross-section and 188 (a) Possible deformation of the SmartMesh (b) Developed prototype Figure 1: The SmartMesh how the pressure acts on the deformable structure. In fact, the forces, which act due to the overpressure or vacuum in the cubical, are always directed perpendicular to the surface and therefore pushing or pulling it. The cells may change their lengths as well as their thickness. 3.1 The Cells The cells deserve a more detailed description as they constitute the fundamental element of the HoverMesh. In fact, they are responsible for two important properties of the system, in a first place as single deformable and freezable units and in a second place as a grid of cells forming the airproof deformable membrane of the cubical. The main function of the cells consists in the ability to deform and to freeze in their actual shape. This is achieved by designing them as small chambers of a light-weight tearproof membrane, which are filled with particles, more precisely beads of polystyrene. The shape of a cell is not determined in its neutral state and may have an arbitrary form only constrained by the volume it encloses. The cells can be inflated allowing the particles to move freely in the enclosed volume, but can also be evacuated, pressing the particles close to each other and therefore solidifying then the actual shape of the cell. The strength of the solidification depends (a) The schematic model of the HoverMesh Cells p Flexible tube Cubical Valve (b) Cross-section of the HoverMesh Figure 2: The HoverMesh on the vacuum and on the friction coefficient of the particles’ surface. To apply the overpressure or the vacuum to each cell flexible tubes are attached (see Figure 2(b)). The second important function of the cells consists in forming the deformable surface of the cubical by simultaneously sealing it, as it will be inflated or evacuated. In addition, they directly act as the interaction surface. This leads to important design issues, as the surface for example should be as flat as possible in any deformation state. In order to freeze not only one cell but the shape of several adjacent ones, the virtual hinges between each neighbor cells need to be solidified as well. As there is no hinge between them, it is mandatory to have broad contact areas between cells. The thickness of the cells depends on the force or torque the structure has to sustain and of course on the friction coefficient of the particles and the vacuum itself. 3.2 Working Principle - Fourier Series As already mentioned, the HoverMesh differs strongly in its working principle from the one of the SmartMesh. In fact, the form of the single cell can not actively be deformed as no actuators are integrated. However, the shape of each cell can be frozen in its actual deformation state. In order to better explain our principle for the deformation, we 189 chose a stripe of nine cells. Seen from a top view, inflating the cubical leads to a convex arc, decompressing it to a concave one. However, the arc represents a half sinus wave. Now, periodic signals, even those that have constantvalued segments like a square wave, can be expressed as a sum of harmonically related sine waves, as the Fourier series demonstrates. In other words, a function can be expressed in terms of frequencies (harmonics) it is composed of. Our concept for the deformation algorithm approaches the theory of the Fourier series. The deformation is achieved by freezing one part of the structure while inflating or deflating the other one. There, another phase shifted sinus wave with a higher frequency evolves. By recursively doing that, half sinus waves are superimposed to already frozen shapes, creating arbitrary curves. Unlike the real Fourier analysis, which simultaneously superimposes all sinus waves with different frequencies, our algorithm consecutively performs this superposition, as shown in Figure 3. Let us make an example. The goal is to achieve a deformation of the stripe to resemble a narrow square wave on the very left. Thus, imagine in the beginning all cells deflated (Figure 3(a)) and the cubical being slowly pressurized: A convex arc appears forming a half sinus wave (Figure 3(b)). Now imagine that the first three of them are being frozen (shown in gray color) and the other half is deflated while the cubical is deflated as well. This would result in a curve as shown in Figure 3(c). Continuing like this, we achieve in three more steps a shape, which already shows similarities to a square wave (Figure 3(f)). The quality of the deformation can still be improved with the following two steps: freezing the first two cells and vacuumizing the cubical results into a stretching motion towards the left end. Finally, after freezing the four cells on the right hand and inflating the cubical results in a quite good approximation of the square wave as shown in Figure 3(h). The deformation has been simulated with Matlab. If the cells could contract even more, a better approximation could be achieved by contracting cells 2, 3 and 4. Thus the quality of the approximations depends on the amount of cells, on their size and on the maximal difference in their surface area between the contracted and the expanded cell. 4. FIRST PROTOTYPES We have constructed a first prototype of a 3x3 mesh. The cells have been designed in a quadratic shape to create a chess-board similar surface. They consist of simple freezer bags and have a surface area of 11 cm x 11 cm when expanded. The different bags are interconnected between each other with double sided adhesive tape. Each bag is then filled with 6 g of polystyrene beads of 2 mm - 3 mm diameter. Each of the bags is equipped with a flexible rubber tube (see Figure 5(b)), which allows to evacuate or inflate each of them separately. The internal diameter of the tubes was chosen to be 1.5 mm in order to prevent the polyester beads from being aspirated trough the tubes. The total weight per cell including the rubber tubes and the polystyrene beads is approximately 13 g. The presented prototype consists only of the mesh structure. The cubical on which the mesh structure will be mounted has not been built so far. Therefore, the following experiments and measurements will only discuss the behavior of the mesh structure. This includes the possibility to freeze (evacuate) or release each single cells of the structure (a) t=0 (b) t=200 (c) t=301 (d) t=501 (e) t=801 (f) t=1001 (g) t=1301 (h) t=1601 Figure 3: The ”Fourier Series”; the cells in the gray area are frozen expanded state, we have mounted the mesh in a setup as shown in Figure 6. The mesh is supported on two opposite sides and is hanging freely over a distance of 25 cm. A measuring jug with a diameter of 11 cm, filled with water has been placed in the center of the mesh. We found the loading capacity to be around 450 g for this setup, which is almost 4 times the weight of the structure. The structure shows a high stiffness despite of the low vacuum pressure. In the released state, when no vacuum is applied to the cells, the structure can be freely deformed without any resistance. In Figure 7(a) and 7(b) two basic shapes, generated with the 3x3 mesh are shown. Two cells per side have been bent up to form the two vertical walls of the mesh in Figure 7(a). The remaining cell in the corner between the vertical walls has been reduced to a minimal surface in order to allow to build a corner. Figure 7(b) shows a similar shape, but with two cells per side bend downwards. Figure 4: The 3x3 mesh fully extended 4.1 and therefore to test the stiffness and deformability of the structure for different shapes. A small vacuum pump for SMD (surface mount devices) soldering is used to evacuate the cells. The pump can create a vacuum of up to 88 mbar (vacuum is defined as the pressure that is below the atmospheric pressure). However, when connected to the 3x3 structure, we measured a vacuum of 26 mbar. This loss in pressure is due to the fact that the whole system consisting of bags, tubes, and joints is not fully airproof. When the 3x3 mesh is fully extended a surface area of 33 cm x 33 cm has been measured (Figure 4). The cells are flat and the surface is quite smooth in this state. The minimum achievable surface on the other hand is approximately 18 cm x 18 cm (Figure 5(a)). The surfaces of the freezing bags are wrinkled in this state, since they do not consist of an elastic material. However, due to the vacuum, the wrinkles remain attached to the surface of the bags. In order to verify the load capacity of the 3x3 mesh in its fully 190 Discussion Compared to the SmartMesh shortly introduced in section 2 the HoverMesh features some advantages. The number of actuators can significantly be decreased since basically only two pumps are necessary - one for the cubical, one for all cells. By eliminating the high number of actuators, the overall weight of the structure can be decreased. On the other hand at least one valve per cell is necessary in order to separately control the pressure in each cell. However, valves are cheaper and much smaller than actuators. Another important advantage is that the nodes with the spherical and revolute joints employed for the SmartMesh are eliminated. The cells of the HoverMesh are glued directly one to each other, which again results in a lower weight and in a smaller number of required parts. A further advantage is, that the structure does not have to completely support its own weight, since an overpressure in the cubical contributes to it. Again, this leads to a more light-weight and flexible structure. The early prototype demonstrates successfully how different shapes can be achieved. By releasing single cells of the structure while keeping others firm (evacuated), a controlled movement can be achieved. More complex and bigger structures could simply be achieved by increasing the amount of cells of the mesh. By evacuating the cells a stiff structure is created. However, in a further step the stiffness of the structure could be considerably improved by increasing the vacuum pressure. The chosen materials are very light, resulting in a total weight of 117g for the 3x3 mesh. This will allow to manipulate the whole structure with a minimum of force. Therefore an integration of the mesh into a cubical looks to be feasible. The principle used for the HoverMesh has already been employed in other fields, such as in medicine for instance. In rehabilitation, casts based on the same principle are being adopted, allowing the patient to take it off when the injured extremity is not burdened. This example shows, that the technique may already be used for commercial applications. Another example, more closely related to human computer interfaces, but still in an early development state, is the wearable force display [11]. It is a force feedback device that completely covers the arm of the user and which is also based on particle mechanical constraint. (a) The 3x3 mesh fully contracted 5. APPLICATIONS As already mentioned in the introduction, such a type of user interface could be used for many applications. Many of them can be found in the industrial sector (e.g. CAD, architecture, various haptic feedback devices). However, we also see a great potential in its use as a novel type of computer interface for a new generation of computer games, where the spatial ability of the players counts. Another application we are looking at, is to use the HoverMesh as a deformable projection screen. The idea is to use the mesh to form a basic structure and then to use multiple projectors to project textures on it. This enables seeing three-dimensional objects without wearing special stereo glasses, allowing an audience to look simultaneously from different sides. Recognizing arbitrary surfaces and projecting on them has already been solved by different groups [14, 15]. Other applications for the HoverMesh could be found in any type of computer entertainment art, such as the projection of more dimensional data onto a 3D surface, or computer animated sculptures for instance. (b) The back side of the 3x3 mesh Figure 5: The 3x3 mesh 6. FUTURE WORK Figure 6: The 3x3 mesh fully extended, carrying 450g 191 Even though some of the components seem to work quite well, such as the solidification of the cells, a lot of work has to be done to improve all components of the HoverMesh. Concerning the cells for example, a better design needs to be achieved to result in an airproof surface. A better airproof connection to the flexible tubes has to be developed as well. The friction coefficient of the particles needs to be improved and particles with different diameters should be employed to achieve a better solidification (due to better fill up of the inter-particle cavities). The cubical with all the necessary peripheral components will be built up. It should provide means to airproofly attach the mesh of cells. The valves, which control the air flow to the cells, will be integrated into the cubical, as shown in Figure 2(b). The valves should be controllable by a computer, thus, electronic peripheral hardware needs to be implemented as well. So far, we have explained how the HoverMesh can actively be deformed. However, we also need to enable its inherent input capabilities, because for many applications it would not only be a nice add but it will be mandatory. As a matter of facts, our structure can manually be deformed to resemble a specific shape or object. Additionally it can also be frozen in that actual shape. Different approaches are possible to detect the shape of the HoverMesh. Let us specify two of them, which we believe to be the most promising ones. The first one is to have multiple cameras looking at the structure and to use vision based algorithms to detect its shape. Another possibility is to use Shape Tape [16], attached directly to the surface of the mesh. 7. CONCLUSION A novel concept for a spatial tangible user interface based on a deformable surface is proposed. A very first prototype of a deformable surface has been created consisting of a chess board of cells. Each single cell contains particles of polystyrene and can be solidified by evacuating it. This surface represents the upper wall of a cubical. By inflating or deflating the cubical a specific deformation of the surface can be achieved, depending on the sequence of solidification of the cells. The presented prototype is a first attempt to develop such a type of structure and it presents quite promising results. Such a device would provide interesting benefits in the field of machine and computer user interfaces, due to its capability to represent surfaces and its 3D interaction features. (a) The 3x3 mesh bent upwards Acknowledgements This research is being done within the CO-ME Project (Computer Aided and Image Guided Medical Interventions - web: http://co-me.ch/) and is founded by the Swiss National Science Foundation. Further, we would like to thank Marc Amman and Ramon Hofer, both students of the ETH Zurich (Swiss Federal Institute of Technology) for their contributions to the development of the HoverMesh. (b) The 3x3 mesh bent downwards 8. REFERENCES [1] E. Sharlin, B. A. Watson, Y. Kitamura, F. Kishino, and Y. Itoh, ”On Tangible User Interfaces, Humans and Spatiality,” Http://www-human.ist.osakau.ac.jp/ ehud/Publications/on.humans.pdf, Personal and Ubiquitous Computing, special issue on ”Tangible Interfaces In Perspective,” September, 2004. [2] G.Moore: Cramming more Components onto Integrated Circuits; Electronics, Volume 38, Number 8; 19.April 1965 [3] A. Mazzone, C. Spagno and A. Kunz; A Haptic Feedback Device based on an Active Mesh; VRST 2003; Osaka, Japan, pp.188-195, ISBN 1-58113-569-6, 2003. [4] A. Mazzone, C. Spagno, A. Kunz and R. Zhang, ”SmartMesh: A Smart Structure for Object Creation”; Eurohaptics 2003, Dublin, Ireland, pp. 451-455, 2003. [5] T. Massie, J. Salisbury; The PHANToM Haptic Interface: A Devide For Probing Virtual Objects; Proceedings of the ASME Winter Annual Meeting Symposium on Haptic Interfaces for Virtual 192 Figure 7: The deformed 3x3 mesh [6] [7] [8] [9] Environment and Teleoperator Systems, DSC-Vol. 55-1, New York, 1994, pp. 295-300 R. Q. Van der Linde, P. Lammertse, E. Frederiksen, B. Ruiter; The Haptic Master, a new high-performance haptic interface; Eurohaptics 2002 Conference Proceedings, p. 1 Edinburgh, July 2002, Immersion Co. The CyberGrasp; Groundbreaking haptic interface for the entire hand http://www.immersion.com/products/ 3d/interaction/cybergrasp.shtml, 2003 B. Ullmer and H. Ishii, ”Emerging Frameworks for Tangible User Interfaces,” in Human Computer Interaction in the New Millennium, J. M. Carroll, Ed. Addison-Wesley, 2001, pp. 579-601. W. B. Spillman Jr, J. S. Sirkis and P. T. Gardiner; Smart mat. and struct.: what are they?; Journal of Smart Mat. and Struct. 5, No. 3, 247-254, 1996 [10] A. Mazzone, Zhang, R.; Kunz, A.: ”Novel Actuators for Haptic Displays based on Electroactive Polymers”; VRST 2003; 1. - 3. Oktober 2003; Osaka, Japan [11] T. Mitsuda, S. Kuge, M. Wakabayashi, S. Kawamura; Wearable Force Display Using a Particle Mechanical Constraint; Presence, Vol. 11, No. 6, December 2002, 569-577, The Massachusetts Institue of Technology [12] G. A. Kramer; Solving Geometric Constraint Systems: a case study in kinematics The MIT Press, Cambridge, Massachusetts London, England 1992 ISBN 0-262-11164-0 [13] M. Hiller; Kinematik und Dynamik fuer Mechanismen, Fahrzeuge und Roboter Gerhard-Mercator-Universitaet GH, Duisburg, 1994 [14] R. Yang and G. Welch; Automatic Projector Display Surface Estimation Using Every-Day Imagery; presented by Herman Towles at the 9th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision 2001. Plzen, Czech Republic. [15] R. Raskar, M. S. Brown, R. Yang, W.-C. Chen, G. Welch, H. Towles, B. Seales, H. Fuchs; Multi-Projector Displays Using Camera-Based Registration; Proceedings of IEEE Visualization 99, San Fransisco, CA, October 24-29, 1999. [16] Y.Baillot, J. J. Eliason, G. S. Schmidt, J. Edward Swan II, D. Brown, S. Julier, M. A. Livingston, L. Rosenblum; Evaluation of the ShapeTape Tracker for Wearable, Mobile Interaction; Proceedings of the IEEE Virtual Reality 2003 pp 285-286; March 22 - 26, 2003 LA, CA, USA 193