Continuously-Adaptive Hapti Rendering Jihad El-Sana1 and Amitabh Varshney2 1 Department of Computer S ien e, Ben-Gurion University, Beer-Sheva, 84105, Israel jihad s.bgu.a .il 2 Department of Computer S ien e, University of Maryland at College Park, College Park, MD 20742, USA Hapti display with for e feedba k is often ne essary in several virtual environments. To enable hapti rendering of large datasets we introdu e Continuously-Adaptive Hapti Rendering, a novel approa h to redu e the omplexity of the rendered dataset. We onstru t a ontinuous, multiresolution hierar hy of the model during the pre-pro essing and then at run time we use high-detail representation for regions around the probe pointer and oarser representation farther away. We a hieve this by using a bell-shaped lter entered at the position of the probe pointer. Using our algorithm we are able to hapti ally render one to two orders of magnitude larger datasets than otherwise possible. Our approa h is orthogonal to the previous work done in a elerating hapti rendering and thus an be used with them. Abstra t. 1 Introdu tion Hapti displays with for e and ta tile feedba k are essential to realism in virtual environments and an be used in various appli ations su h as medi ine (virtual surgery for medi al training, mole ular do king for drug design), entertainment (video games), edu ation (studying nano, ma ro, or astronomi al s ale natural and s ien e phenomena), and virtual design and prototyping (nanomanipulation, integrating hapti s into CAD systems). Humans an sense tou h in two ways: ta tile and kinestheti . Ta tile refers to the sensation aused by stimulating the skin nerves su h as by vibration, pressure, and temperature. Kinestheti refers to the sensation from motion and for es, whi h trigger the nerve re eptors in the mus les, joints, and tendons. Computer hapti s is uli to a hapti on erned with generating and rendering hapti omputer user, just as stim- omputer graphi s deals with visual stimuli. In interfa e intera tion, the user onveys a desired motor a tion by physi- ally manipulating the interfa e, whi h in turn provides ta tual sensory feedba k to the user by appropriately stimulating her/his ta tile and kinestheti systems. Figure 1 shows the basi virtual environment. As the user manipulates the generi devi e, the hapti sensory pro ess of hapti ally rendering obje ts in a probe of the hapti system keeps tra k of the position and the orientation of the probe. When a probe ollides with an obje t, the me hanisti model al ulates the rea tion for e based on the depth of the probe into the virtual obje t. Generic Probe Information Collision Detection Contact Data Human Machine Contact Servo Loop Applied Force Inverse Kinematics Fig. 1. Touch Effects Geometry Information Forward Kinematices Force Mapping Modified Force The hapti rendering pro esses Several hapti te hniques have been developed to hapti ally render 3D obje ts whi h an have either surfa e-based or volume-based representation. Hapti intera tion in virtual environment ould be Point-based or Ray-based. In pointbased hapti intera tion only the end-point of the hapti devi e, known as Hapti Interfa e Point (HIP), intera ts with the obje ts. In ray-based hapti intera tion, the generi probe of the hapti devi e is modeled as a nite ray. The ollision is dete ted between this ray and the obje t, and the orientation of the ray is used in omputing the hapti for e feedba k. In this approa h the for e re e tion is al ulated using a linear spring law F = kx. In visual rendering several te hniques are used to enhan e the intera tivity and improve the realism of the rendered obje ts. Smooth shading and texture mapping are good examples of these te hniques. Similar algorithms are used in hapti rendering to onvey the ta tual feeling of the inspe ted obje ts. Some of these approa hes have been adapted from graphi s rendering while others have been developed ex lusively for hapti rendering. 2 Previous Work In this se tion we overview some of the work done in hapti s rendering for virtual environment appli ations. Hapti rendering has been found to be parti ularly useful in mole ular do king [3℄ and nanomanipulation [19℄. Randolph et al. [13℄ have developed an approa h for point-based hapti rendering of a surfa e by using an intermediate representation. A lo al planar approximation to the surfa e is omputed at the ollision point for ea h y le of the for e loop. The rea tion for e ve tor is omputed with respe t to the tangent plane. This approa h has one major drawba k { undesirable for e dis ontinuities may appear if the generi probe of the hapti devi e is moved over large distan es before the new tangent plane is updated. An improvement to this method has been presented by Salisbury and Tarr [17℄. Basdogan et al.[2℄ have developed a ray-based rendering approa h. The generi probe of the hapti devi e is modeled as a line segment. They update the simulated generi probe of the hapti devi e (stylus) as the user manipulates the a tual one. They dete t ollisions between the simulated stylus and the virtual obje ts in three progressively nested he ks: (a) the bounding boxes of the virtual obje ts, (b) the bounding box of appropriate triangular elements, and ( ) appropriate triangular elements. They estimate the rea tion for e by using a linear spring law model. Ruspini et al. [16℄ introdu e the notion of proxy on the hapti system as a massless sphere that moves among the obje ts in the environment. They assumed that all the obsta les in the environment ould be divided into a nite set of onvex omponents. During the update pro ess, the proxy attempts to move to the goal on guration using dire t linear motion. Gregory et al. [8℄ have developed an eÆ ient system, H-Collide, for omputing onta t(s) between the probe of the for e-feedba k devi e and obje ts in the virtual environment. Their system uses spatial de omposition, a bounding volume hierar hy, and exploits frame-to-frame oheren e to a hieve a fa tor of 3 to 20 in speed improvement. Polygonal or polyhedral des riptions are often used to represent obje ts in virtual environments. The straightforward hapti rendering of these obje ts often does not onvey the desired shape for the user. Morgenbesser and Srinivasan [15℄ have developed for e shading, in whi h the for e ve tor is interpolated over the polygonal surfa es. Hapti rendering has also been su essfully pursued for volumetri datasets [1℄ and for NURBS surfa es [4℄. The sensations of tou h have been onveyed to the human ta tile system using textures generated by { for e perturbation and displa ement mapping. For e perturbation refers to the te hnique of modifying dire tion and magnitude of the for e ve tor to generate surfa e e e ts su h as roughness [14℄. In displa ement mapping the a tual geometry of the obje t is modi ed to display the surfa e details. To improve the realism of the hapti intera tion su h as the push of a button or the turn of a swit h, fri tion e e ts have been introdu ed. Fri tion an be simulated by applying stati and dynami for es in a dire tion tangential to the normal for e. 2.1 Hapti Rendering The hapti rendering pro ess involves the following three steps: { { { Initializing the hapti devi e interfa e and transferring the dataset representation from the user data bu ers to the hapti devi e drivers or API bu ers. This step may require translating the data from the user representation to mat h the hapti API representation. Collision dete tion between the elements representing virtual obje ts and the probe of the hapti devi e. Su h dete tion be omes mu h more omplex when the probe has multiple dynami ngers. Estimating the for e that the hapti devi e needs to apply to the user's hand or nger. This for e is fed to the generi probe. We would like to redu e the overhead for the above three steps. Di erent approa hes ould be used to a hieve this goal. A simple way ould be subdivide the dataset into disjoint ells (using o tree or any other spatial subdivision) during pre-pro essing. Then at run-time the ells whi h are within some threshold distan e from the probe pointer are onsidered in the ollision dete tion and for e feedba k estimation. This approa h has two drawba ks. First, the sele ted ells may eliminate part of the for e eld that a e ts the user. For example, when hapti ally rendering a surfa e as in Figure 2 the user may sense in orre t for e when using spatial subdivision. Se ond, if the user moves the probe pointer too fast for the appli ation to update the ells, the user ould per eive rough (and in orre t) for e feedba k. Another approa h to redu e the above overhead ould be to redu e the omplexity of the dataset through simpli ation. Several di erent levels of detail ould then be onstru ted o -line. At run time, an appropriate level is sele ted for ea h obje t. However, swit hing between the di erent levels of detail at run time may lead to noti eable hanges in the for e feedba k whi h is distra ting. Also, if the obje ts being studied are very large, this method will provide only one level of detail a ross the entire obje t. Selected Region Probe Cursor Surface Fig. 2. The use of spatial subdivision may result in in orre t sense of the for e eld In this paper we introdu e Continuously-Adaptive Hapti Rendering { a novel approa h to redu e the omplexity of the rendered dataset { whi h is based on the View-Dependen e Tree introdu ed by El-Sana and Varshney [6℄. We use the same o -line onstru ted tree and at run time we use a di erent poli y to determine the various levels of detail at the di erent regions of the surfa e. 2.2 View-Dependent Rendering View-dependent simpli ations using the edge- ollapse/vertex-split primitives in lude work by Xia et al. [20℄, Hoppe [10℄, Guezie et al. [9℄, and El-Sana and Varshney [6℄. View-dependent simpli ations by Luebke and Erikson [12℄, and De Floriani et al. [5℄ do not rely on the edge- ollapse primitive. Klein et al. [11℄ have developed an illumination-dependent re nement algorithm for multiresolution meshes. S hilling and Klein [18℄ have introdu ed a re nement algorithm that is texture dependent. Gieng et al. [7℄ produ e a hierar hy of triangle meshes that an be used to blend di erent levels of detail in a smooth fashion. View-dependen e tree [6℄ is a ompa t multiresolution hierar hi al datastru ture that supports view-dependent rendering. In fa t, for a given input dataset, the view-dependen e tree onstru tion often leads to a forest (set of trees) sin e not all the nodes an be merged together to form one tree. The view-dependen e trees are able to adapt to various levels of detail. Coarse details are asso iated with nodes that are lose to the top of the tree (roots) and high details are asso iated with the nodes that are lose to the bottom of the tree (leaves). The re onstru tion of a real-time adaptive mesh requires the determination of the list of verti es of this adaptive mesh and the list of triangles that onne t these verti es. Following [6℄, we refer to these lists as the list of a tive nodes and the list of a tive triangles. 3 Our Approa h We have integrated view-dependent simpli ation with hapti rendering to allow faster and more eÆ ient for e feedba k. We refer to this as as ontinuouslyadaptive hapti rendering. Similar to graphi s rendering, Continuously-adaptive hapti rendering speeds up the overall performan e of hapti rendering by redu ing the number of triangles representing the dataset. In our approa h we do not need to send the omplete surfa e to the hapti system. Instead, we send a surfa e with high details in the region lose to the generi probe pointer and oarser representation as the region gets far from the generi probe. 3.1 Lo alizing the View-Dependen e Tree The the onstru tion of view-dependen e trees results in dependen ies between the nodes of the tree. These dependen ies are used to avoid foldovers at run time by preventing the ollapse or merge of nodes before others. Therefore, these dependen ies may restri t the re nement of nodes, might have otherwise re ned to omply with the visual delity or error metri . In order to redu e su h restri tions we redu e the dependen ies between the nodes of the tree. We an redu e the dependen ies by lo alizing the tree, whi h refers to onstru ting the tree to minimize the distan e between the nodes of the tree. We de ne the radius of a subtree as the maximum distan e between the root of the subtree and any of its hildren. We are urrently using the Eu lidean distan e metri to measure the distan e. We an lo alize a view-dependen e tree by minimizing the radius of ea h subtree. Sin e we onstru t the tree bottomup, the algorithm starts by initializing ea h subtree radius to zero (ea h subtree has only one node). Ea h ollapse operation results in a merge of two subtrees. We ollapse a node to the neighbor whi h result in the minimum radius. It is important to note that our algorithm does not guarantee optimal radius for the nal tree. In pra ti e, it results in fairly a eptable small radius. 3.2 Levels of Detail When hapti ally rendering a polygonal dataset we need to dete t ollisions between the probe and the dataset and ompute the for e that the probe supplies the user at very high rates (more that 1000 Hz). The triangles lose to the probe ontribute more to the for e feedba k and have a higher probability of ollision with the probe. The triangles far from the probe have little e e t on the for e-feedba k and have a smaller probability of ollision with the probe. In our approa h we use high-detail representation for regions near the probe and oarser representation farther away. We a hieve this by using a bell-shaped lter as in Figure 3(a). In our lter, the distan e from the hapti probe pointer di tates the level of detail of ea h region. This lter ould be seen as a mapping of distan e from the probe pointer to the swit h value (swit h value is the value of the simpli ation metri at whi h two verti es had ollapsed at the tree onstru tion time). The surfa e lose to the probe should be displayed in its highest possible resolution in order to onvey the best estimation of the for e feedba k. In addition, regions far enough from the probe an not be displayed at less than the oarsest level. We were able to a hieve further speed-up by hanging the shape of our lter from bell-shaped to multiple-frustums shape. This redu es the time to ompute the swit h value of an a tive node, whi h needs to be exeuted for ea h node at ea h frame. Figure 3(b) shows the shape of the optimized leter. This hange redu es the omputation of distan e (from the probe) and ubi fun tion (whi h we use to estimate the bell-shaped lter) to nd the maximum di eren e along any of the three axes x, y, and z. We also allow the user to hange some of the lter attributes that determine the relation between the level of detail and the distan e between the probe pointer. (a) Fig. 3. (b) Ideal verse optimized lter At run time we load the view-dependen e trees and initialize the roots as the a tive verti es. Then at ea h frame we repeat the following steps. First, we query the position and the orientation of the probe. Then we s an the list of a tive verti es. For ea h vertex we ompute the distan e from the probe position, determine the mapping to the swit h value domain, and then ompare the resulting value with the swit h value stored at the node. The node splits if the omputed value is less than the swit h value and the node satis es the impli it dependen ies for split. The node merges with its sibling if the omputed value is larger than the swit h value stored at the parent of this node and the node satis es the impli it dependen ies for merge. After ea h split, we remove the node from the a tive-nodes list and insert its two hildren into the a tive-nodes list. Then we update the adja ent triangle list to mat h the hange; and insert the PAT triangles into the adja ent list of the newly inserted nodes. The merge operation is arried out in two steps, rst we remove the two merged nodes from the a tive-nodes list, and then we insert the parent node into the a tive-nodes list. Finally, we update the adja ent-triangles list by removing the PAT triangles list and merging the two merged nodes triangles (the interested reader may refer to the details in [6℄). The resulting set of a tive triangles is sent to the hapti interfa e. 4 Further Optimizations We were able to a hieve further speedups by ne-tuning spe i se tions of our implementation. For instan e, when updating the a tive-nodes list and a tivetriangles list after ea h step we repla e pointers instead of removing and inserting them. For example after split, we repla e the node with one of its hildren (the left one) and insert the se ond hild. Similarly in merge we repla e the left hild with its parent and remove the other hild. Even though the a tive lists are lists of pointer to the a tual nodes of the tree, still their allo ation and deallo ation requires more time be ause it relies on the operating system. The hapti and graphi s bu ers are updated in an in remental fashion. Sin e the hange between onse utive frames tends to be small, this results in small hanges in the hapti and graphi s bu ers. Therefore, we repla e the verti es and triangles that do not need to be rendered in the next frame with the newly added verti es and triangles. This requires very small update time that is not noti eable by the user. Sin e the graphi s rendering and the hapti rendering run at di erent frequen ies we have de ided to maintain them through di erent pro esses (whi h run on di erent pro essors for a multi-pro essor ma hine). The display runs at low update rates of about 20 Hz, while the hapti pro ess runs at higher rates of about 1000 Hz. We also use another pro ess to maintain the a tive lists and the view-dependen e tree stru ture. At 20 Hz frequen y we query the hapti probe for its position and orientation, then update the a tive lists to re e t the hange of the probe pointer. Finally, we update the graphi s and the hapti bu ers. In this s enario, the graphi s omponent is updated at 20 Hz and runs at this rate while the hapti omponent runs at 1000 Hz and is updated at only 20 Hz. To better approximate the level of detail when the user is moving the probe fast, we use the estimated motion traje tory of the probe pointer, and the distan e it has traveled sin e the previous frame to perform look-ahead estimation of the probe's likely lo ation. 5 Results We have implemented our algorithm in C++ on an SGI ONYX2 with in nite reality. For hapti rendering we have used the PHANToM hapti devi e from SensAble Te hologies with six degrees of input and three degree of output freedom. The hapti interfa e is handled through the GHOST API library (from SensAble Te hologies). This hapti devi e fails (the servo loop breaks) when it is pushed to run at less that 1000 Hz frequen y. Dataset CAD-Obj1 CAD-Obj2 Mole ule Terrain CHR OFF CHR ON Average Average Average Average Frame rate(Hz) Quality Frame rate (Hz) Quality 2K 1500 good 1500 good 8K 600 bad 1200 good 20 K | breaks 1000 good 92 K | breaks 1000 good Table 1. Results of our approa h Triangles We have ondu ted several tests on various datasets and have re eived enouraging results. Table 1 shows some of our results. It shows results of hapti rendering with (CHR ON) and without (CHR OFF) the use of ontinuouslyadpative hapti rendering. For medium size datasets the hapti devi e works for some time then fails. When the datasets be ome larger the hapti devi e fails almost immediately be ause it was not able to run at the minimum required frequen y. This failure ould be the result of failing to nish the ollision dete tion pro ess or the failure to nish the for e eld estimation pro ess. Redu ing the dataset size using our algorithm enables su essful hapti rendering of these datasets. Figure 4 shows the system on guration we used in our testing. In our system the mouse and the hapti probe pointer is used simultaneously to hange and update the viewed position of dataset. Figure 5 shows high level of detail around the probe pointer (shown as a bright sphere in the enter). Fig. 4. Our system on guration Shaded Fig. 5. Hapti 6 Wire frame rendering of terrain dataset, the yellow sphere is the hapti probe pointer Con lusions We have presented the enables hapti hapti ontinuously-adaptive hapti rendering algorithm, whi h rendering of datasets that are beyond the apability of the systems. Our approa h is based upon dynami , frame-to-frame the geometry of the surfa e and thus an be used with any of the prior s hemes, su h as bounding volume hierar hies, to a hieve superior a rendering. Hapti urrent hanges in eleration of hapti interfa es are being used in several real-life appli ations su h as mole ular do king, nanomanipulation, virtual design and prototyping, virtual surgery, and medi al training. We anti ipate that our work highlighted in this paper will a hieve a elerated hapti s rendering for all of these appli ations. A knowledgements This work has been supported in part by the NSF grants: DMI-9800690, ACR9812572, and a DURIP award N00014970362. Jihad El-Sana has been supported in part by the Fulbright/Israeli Arab S holarship Program and the Cata osinos Fellowship for Ex ellen e in Computer S ien e. We would like to thank the reviewers for their insightful omments whi h led to several improvements in the presentation of this paper. We would also like to thank our olleagues at the Center for Visual Computing at Stony Brook for their en ouragement and suggestions related to this paper. Referen es 1. R. S. Avila and L. M. Sobierajski. sualization. A hapti intera tion method for volume vi- In Pro eedings, IEEE Visualization, pages 197{204, Los Alamitos, O tober 27{November 1 1996. IEEE. 2. C. Basdogan, C. Ho, and M. Srinivasan. A ray-based hapti rendering te hnique for displaying shape and texure of 3-d obje ts in virtual environment. Dynami Systems and Control Division, November 1997. In ASME 3. F. P. Brooks Jr., M. Ouh-Young, J. J. Batter, and P. J. Kilpatri k. Proje t GROPE | hapti displays for s ienti visualization. In Computer Graphi s (SIGGRAPH '90 Pro eedings), volume 24(4), pages 177{185, August 1990. 4. F. Da hille IX, H. Qin, A. Kaufman, and J. El-Sana. Hapti s ulpting of dynami surfa es ( olor plate S. 227). In Stephen N. Spen er, editor, Pro eedings of the Conferen e on the 1999 Symposium on intera tive 3D Graphi s, pages 103{110, New York, April 26{28 1999. ACM Press. 5. L. De Floriani, P. Magillo, and E. Puppo. EÆ ient implementation of multitriangulation. In H. Rushmeier D. Elbert and H. Hagen, editors, Pro eedings Visualization '98, pages 43{50, O tober 1998. 6. J. El-Sana and A. Varshney. Generalized view-dependent simpli ation. In Computer Graphi s Forum, volume 18, pages C83{C94. Eurographi s Asso iation and Bla kwell Publishers Ltd 1999, 1999. 7. T. Gieng, B. Hamann, K. Joy, G. S hussman, and I. Trotts. Constru ting hierar hies for triangle meshes. IEEE Transa tions on Visualization and Computer Graphi s, 4(2):145{161, 1998. 8. A. Gregory, M. Lin, S. Gotts halk, and R. Taylor. H-COLLIDE: A framework for fast and a urate ollision dete tion for hapti intera tion. Te hni al Report TR98032, Department of Computer S ien e, University of North Carolina - Chapel Hill, November 03 1998. Tue, 3 Nov 1998 17:27:33 GMT. 9. A. Guezie , G. Taubin, B. Horn, and F. Lazarus. A framework for streaming geometry in VRML. IEEE CG&A, 19(2):68{78, 1999. 10. H. Hoppe. View-dependent re nement of progressive meshes. In Pro eedings of SIGGRAPH '97 (Los Angeles, CA), pages 189 { 197. ACM Press, August 1997. 11. R. Klein, A. S hilling, and W. Straer. Illumination dependent re nement of multiresolution meshes. In Computer Graphi s Intl, pages 680{687, June 1998. 12. D. Luebke and C. Erikson. View-dependent simpli ation of arbitrary polygonal environments. In Pro eedings of SIGGRAPH '97 (Los Angeles, CA), pages 198 { 208. ACM SIGGRAPH, ACM Press, August 1997. 13. W. Mark, S. Randolph, M. Fin h, J. Van Verth, and R. Taylor II. Adding for e feedba k to graphi s systems: Issues and solutions. In Pro eedings of SIGGRAPH '96 (New Orleans, LA, August 4{9, 1996), pages 447 { 452. ACM Press, 1996. 14. M. Minsky, M. Ouh-young, O. Steele, F. P. Brooks, Jr., and M. Behensky. Feeling and seeing: Issues in for e display. In 1990 Symposium on Intera tive 3D Graphi s, pages 235{243, Mar h 1990. 15. H. Morgenbesser and M. Srinivasan. For e shading for hapti shape pre eption. In ASME Dynami Systems and Control Division, volume 58, pages 407 { 412, 1996. 16. D. Ruspini., K. Kolarov, and O. Khatib. The hapti display of omplex graphi al environment. In Pro eedings of SIGGRAPH '97 (Los Angeles, CA), pages 345 { 352. ACM SIGGRAPH, ACM Press, August 1997. 17. J. Salisbury and C. Tarr. Hapti rendering of surfa e de ned by impli it fun tions. In ASME Dynami Systems and Control Division, November 1997. 18. A. S hilling and R. Klein. Graphi s in/for digital libraries | rendering of multiresolution models with texture. Computers and Graphi s, 22(6):667{679, 1998. 19. R. Taylor, W. Robinett, V. Chi, F. Brooks, Jr., W. Wright, R. Williams, and E. Snyder. The nanomanipulator: A virtual-reality interfa e for a s anning tunnelling mi ros ope. In Pro eedings, SIGGRAPH 93, pages 127{134, 1993. 20. J. Xia, J. El-Sana, and A. Varshney. Adaptive real-time level-of-detail-based rendering for polygonal models. IEEE Transa tions on Visualization and Computer Graphi s, pages 171 { 183, June 1997.