In this research, we present an object search
framework using robot-gaze interaction that support... more In this research, we present an object search framework using robot-gaze interaction that support patients with motor paralysis conditions. A patient can give commands by gazing to the target object and a robot starts to search au- tonomously. Apart from multiple gaze interaction, our approach uses few gaze interaction to specify location clue and object clue and thus integrates the RGB-D sensing to segment unknown objects from the environment. Based on hypotheses from gaze information, we utilize multiregion Graph Cuts method along with an analysis of depth information. Furthermore, our search algorithm allows a robot to find a main observation point which is the point where user can clearly observe the target object. If a first segmentation was not satisfied by the user, the robot is able to adapt its pose to find different views of object. The approach has been implemented and tested on the humanoid robot ENON. With a few gaze guidance, the success rate of segmentation of unknown objects was achieved to be 85%. The experimental results confirm its applicability on a wide variety of objects even when the target object was occluded by another object.
We propose an efficient method for detecting potential collisions among multiple objects with arb... more We propose an efficient method for detecting potential collisions among multiple objects with arbitrary motion (translation and rotation) in 3D space. The method is useful for online monitoring and path planning in a 3D environment in which there are multiple independently-moving objects. The method consists of two main stages: 1) the coarse stage, an approximate test is performed to identify interfering objects in the entire workspace using octree representation of object shapes; and 2) the fine stage, polyhedral representation of object shapes is used to more accurately identify any object parts that might cause interference and collisions. For this purpose, specific pairs of faces belonging to any of the interfering objects found in the first stage are tested, thus performing detailed computation on a reduced amount of data. Experimental results, which demonstrate the efficiency of the proposed collision detection method, are given
In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context o... more In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context of a Reality-Virtuality (RV) continuum, encompassing a large class of "Mixed Reality" (MR) displays, which also includes Augmented Virtuality (AV). MR displays are defined by means of seven examples of existing display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish different Mixed Reality display systems from each other are presented, first by means of a table in which the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and then by means of a three dimensional taxonomic framework, comprising: Extent of World Knowledge (EWK), Reproduction Fidelity (RF) and Extent of Presence Metaphor (EPM). A principal objective of the taxonomy is to clarify terminology issues and to provide a framework for classifying research across different disciplines.
... based on the ways we interact with a real world[ 1,221. Fisher et al. built a prototype syste... more ... based on the ways we interact with a real world[ 1,221. Fisher et al. built a prototype system of virtual reality for telerobotics application[3]. Sturman et al. studied the possibilities of using hand gestures in an interactive environment ...
in which participants at diflerent sites can feel as if they are at one site and can cooperativel... more in which participants at diflerent sites can feel as if they are at one site and can cooperatively work. In the teleconferencing system, the 3 0 model of a participant is constructed b y a wire frame model mapped b y color texture and is displayed on the 3 0 screen at the receiving site.
We propose an accurate collision detection algorithm for use in virtual reality applications. The... more We propose an accurate collision detection algorithm for use in virtual reality applications. The algorithm works for three-dimensional graphical environments where multiple objects, represented as polyhedra (boundary representation), are undergoing arbitrary motion (translation and rotation). The algorithm can be used directly for both convex and concave objects and objects can be deformed (nonrigid) during motion. The algorithm works efficiently by first reducing the number of face pairs that need to be checked accurately for interference, by first localizing possible collision regions using bounding box and spatial subdivision techniques. Face pairs that remain after this pruning stage are then accurately checked for interference. The algorithm is efficient, simple to implement, and does not require any memory-intensive auxiliary data structures to be precomputed and updated. The performance of the proposed algorithm is compared directly against other existing algorithms, e.g., the separating plane algorithm, octree update method, and distance-based method. Results are given to show the efficiency of the proposed method in a general environment.
Abstract: This paper proposes a new hand gesture recogni-tion method using a 3D prediction model ... more Abstract: This paper proposes a new hand gesture recogni-tion method using a 3D prediction model used for develop-ing human-computer interfaces. It focuses on real-time hand-pose estimation which is indispensable for virtual ob-ject manipulation. In the first stage, simple but ...
We propose an accurate collision detection algorithm for use in virtual reality applications. The... more We propose an accurate collision detection algorithm for use in virtual reality applications. The algorithm works for three-dimensional graphical environments where multiple objects, represented as polyhedra (boundary representation), are undergoing arbitrary motion (translation and rotation). The algorithm can be used directly for both convex and concave objects and objects can be deformed (non-rigid) during motion. The algorithm works efficiently by first reducing the number of face pairs that need to be checked accurately for interference by first localizing possible collision regions using bounding box and spatial subdivision techniques; face pairs that remain after this pruning stage are then accurately checked for interference. The algorithm is efficient, simple to implement, and does not require any memory intensive auxiliary data structures to be precomputed and updated. Since polyhedral shape representation is one of the most common shape representation schemes, this algorithm should be useful to a wide audience. Performance results are given to show the efficiency of the proposed method
d confirmation stages for both shape andbehavioral design by providing shape and constrainingprim... more d confirmation stages for both shape andbehavioral design by providing shape and constrainingprimitives in an immersive environment.That is, users can design reactive objects bysimply deforming and assembling these primitivesin a unified and comprehensible manner.8Collaboration supportSoftware researchers have developed a numberof shared virtual environments (SVEs) to supportflexible spatial collaboration.9,10These provideadaptable control over the location and scale oftheir...
... experiment described later, it was confirmed that two-handed interaction improves the efficie... more ... experiment described later, it was confirmed that two-handed interaction improves the efficiency of human-computer interaction in a ... The collision avoidance is performed as follows. ... block move toward the corresponding 3-D cursor in X direction (of the coordinate system of the ...
This paper proposes a robust registration method with stereo cameras and an inertial sensor for a... more This paper proposes a robust registration method with stereo cameras and an inertial sensor for augmented reality. The geometric registration is one of the most important problems because virtual objects should be superimposed on the right place as if they really exist in the real world. The vision-based registration is achieved by estimating a position and an orientation of camera(s)
This paper presents a new method to generate and present arbitrarily directional binocular stereo... more This paper presents a new method to generate and present arbitrarily directional binocular stereo images from a sequence of omnidirectional images. A sequence of omnidirectional images is taken by moving an omnidirectional image sensor in a static real environment. The motion of the omnidirectional image sensor is constrained to a plane. The sensor's route and speed are known. In the proposed method, a fixed length of the sequence is buffered in a computer to generate arbitrarily directional binocular stereo images by combining captured rays. Using the method a user can look around a scene in the distance with rich 3D sensation without significant time delay. This paper describes the principle of real-time generation of binocular stereo images. In addition, we introduce a prototype telepresence system of view-dependent stereo image generation and presentation
We focus on a shared augmented environment (SAE) as an almost ideal face-to-face collaborative vi... more We focus on a shared augmented environment (SAE) as an almost ideal face-to-face collaborative virtual workspace. In an SAE, multiple users can observe both a virtual world and real partners through optical see-through head mounted displays (STHMDs). This paper describes two experiments for verifying the effectiveness of an SAE compared with a conventional shared virtual environment (SVE) and exploring improvement of them. Through the experiments, the effectiveness of an SAE compared with an SVE was confirmed. It was also confirmed that enhancement of the shared environment with computer graphics, i.e. displaying a partner's body in an SVE, drawing a line as a partner's viewing direction and emphasizing virtual objects to which a partner pay attention, improves workers' feeling and collaboration efficiency.
Describes a prototype of an augmented reality system using vision-based registration. In order to... more Describes a prototype of an augmented reality system using vision-based registration. In order to build an augmented reality system with video see-through image composition, camera parameters for generating virtual objects must be obtained at video-rate. The system estimates camera parameters from four known markers in an image sequence captured by a small CCD camera mounted on a HMD (head mounted display). Virtual objects are overlaid upon a captured image sequence in real-time
A real-time three-dimensional position and orientation tracking system is proposed for use with w... more A real-time three-dimensional position and orientation tracking system is proposed for use with wearable augmented-reality systems. The system combines infrared markers with a head-mounted stereo camera to detect the user's position, and an orientation sensor to measure the orientation of the user's head. An extended Kalman filter is employed to reduce triangulation and orientation error by integrating the signals acquired by multiple sensors. The system is evaluated through a series of experiments, and a navigation system implemented using the proposed scheme is present. The accuracy of the system is shown to be sufficient to allow annotations and virtual objects to be overlaid on real scenes via a head-mounted display without confusion.
Recently virtual reality (VR) systems have been incorporating rich information available in the r... more Recently virtual reality (VR) systems have been incorporating rich information available in the real world into VR environments in order to improve their reality. This stream has created the field of mixed reality which seamlessly integrates real and virtual worlds. This paper describes a novel approach to the construction of a mixed environment. The approach is based on capturing the dynamic real world by using a video-rate omnidirectional stereo image sensor. The mixed environment is constructed of two different types of models: (1) texture-mapped cylindrical 3-D model of dynamic real scenes and (2) 3-D computer graphics (CG) model. The cylindrical 3-D model is generated from full panoramic stereo images obtained by the omnidirectional sensor that uses a pair of hexagonal pyramidal mirrors and twelve CCD cameras. A prototype system has been developed to confirm the feasibility of the proposed method, in which panoramic binocular stereo images of the mixed environment are projected on a cylindrical immersive display depending on user's viewpoint in real time.
In this research, we present an object search
framework using robot-gaze interaction that support... more In this research, we present an object search framework using robot-gaze interaction that support patients with motor paralysis conditions. A patient can give commands by gazing to the target object and a robot starts to search au- tonomously. Apart from multiple gaze interaction, our approach uses few gaze interaction to specify location clue and object clue and thus integrates the RGB-D sensing to segment unknown objects from the environment. Based on hypotheses from gaze information, we utilize multiregion Graph Cuts method along with an analysis of depth information. Furthermore, our search algorithm allows a robot to find a main observation point which is the point where user can clearly observe the target object. If a first segmentation was not satisfied by the user, the robot is able to adapt its pose to find different views of object. The approach has been implemented and tested on the humanoid robot ENON. With a few gaze guidance, the success rate of segmentation of unknown objects was achieved to be 85%. The experimental results confirm its applicability on a wide variety of objects even when the target object was occluded by another object.
We propose an efficient method for detecting potential collisions among multiple objects with arb... more We propose an efficient method for detecting potential collisions among multiple objects with arbitrary motion (translation and rotation) in 3D space. The method is useful for online monitoring and path planning in a 3D environment in which there are multiple independently-moving objects. The method consists of two main stages: 1) the coarse stage, an approximate test is performed to identify interfering objects in the entire workspace using octree representation of object shapes; and 2) the fine stage, polyhedral representation of object shapes is used to more accurately identify any object parts that might cause interference and collisions. For this purpose, specific pairs of faces belonging to any of the interfering objects found in the first stage are tested, thus performing detailed computation on a reduced amount of data. Experimental results, which demonstrate the efficiency of the proposed collision detection method, are given
In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context o... more In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context of a Reality-Virtuality (RV) continuum, encompassing a large class of "Mixed Reality" (MR) displays, which also includes Augmented Virtuality (AV). MR displays are defined by means of seven examples of existing display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish different Mixed Reality display systems from each other are presented, first by means of a table in which the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and then by means of a three dimensional taxonomic framework, comprising: Extent of World Knowledge (EWK), Reproduction Fidelity (RF) and Extent of Presence Metaphor (EPM). A principal objective of the taxonomy is to clarify terminology issues and to provide a framework for classifying research across different disciplines.
... based on the ways we interact with a real world[ 1,221. Fisher et al. built a prototype syste... more ... based on the ways we interact with a real world[ 1,221. Fisher et al. built a prototype system of virtual reality for telerobotics application[3]. Sturman et al. studied the possibilities of using hand gestures in an interactive environment ...
in which participants at diflerent sites can feel as if they are at one site and can cooperativel... more in which participants at diflerent sites can feel as if they are at one site and can cooperatively work. In the teleconferencing system, the 3 0 model of a participant is constructed b y a wire frame model mapped b y color texture and is displayed on the 3 0 screen at the receiving site.
We propose an accurate collision detection algorithm for use in virtual reality applications. The... more We propose an accurate collision detection algorithm for use in virtual reality applications. The algorithm works for three-dimensional graphical environments where multiple objects, represented as polyhedra (boundary representation), are undergoing arbitrary motion (translation and rotation). The algorithm can be used directly for both convex and concave objects and objects can be deformed (nonrigid) during motion. The algorithm works efficiently by first reducing the number of face pairs that need to be checked accurately for interference, by first localizing possible collision regions using bounding box and spatial subdivision techniques. Face pairs that remain after this pruning stage are then accurately checked for interference. The algorithm is efficient, simple to implement, and does not require any memory-intensive auxiliary data structures to be precomputed and updated. The performance of the proposed algorithm is compared directly against other existing algorithms, e.g., the separating plane algorithm, octree update method, and distance-based method. Results are given to show the efficiency of the proposed method in a general environment.
Abstract: This paper proposes a new hand gesture recogni-tion method using a 3D prediction model ... more Abstract: This paper proposes a new hand gesture recogni-tion method using a 3D prediction model used for develop-ing human-computer interfaces. It focuses on real-time hand-pose estimation which is indispensable for virtual ob-ject manipulation. In the first stage, simple but ...
We propose an accurate collision detection algorithm for use in virtual reality applications. The... more We propose an accurate collision detection algorithm for use in virtual reality applications. The algorithm works for three-dimensional graphical environments where multiple objects, represented as polyhedra (boundary representation), are undergoing arbitrary motion (translation and rotation). The algorithm can be used directly for both convex and concave objects and objects can be deformed (non-rigid) during motion. The algorithm works efficiently by first reducing the number of face pairs that need to be checked accurately for interference by first localizing possible collision regions using bounding box and spatial subdivision techniques; face pairs that remain after this pruning stage are then accurately checked for interference. The algorithm is efficient, simple to implement, and does not require any memory intensive auxiliary data structures to be precomputed and updated. Since polyhedral shape representation is one of the most common shape representation schemes, this algorithm should be useful to a wide audience. Performance results are given to show the efficiency of the proposed method
d confirmation stages for both shape andbehavioral design by providing shape and constrainingprim... more d confirmation stages for both shape andbehavioral design by providing shape and constrainingprimitives in an immersive environment.That is, users can design reactive objects bysimply deforming and assembling these primitivesin a unified and comprehensible manner.8Collaboration supportSoftware researchers have developed a numberof shared virtual environments (SVEs) to supportflexible spatial collaboration.9,10These provideadaptable control over the location and scale oftheir...
... experiment described later, it was confirmed that two-handed interaction improves the efficie... more ... experiment described later, it was confirmed that two-handed interaction improves the efficiency of human-computer interaction in a ... The collision avoidance is performed as follows. ... block move toward the corresponding 3-D cursor in X direction (of the coordinate system of the ...
This paper proposes a robust registration method with stereo cameras and an inertial sensor for a... more This paper proposes a robust registration method with stereo cameras and an inertial sensor for augmented reality. The geometric registration is one of the most important problems because virtual objects should be superimposed on the right place as if they really exist in the real world. The vision-based registration is achieved by estimating a position and an orientation of camera(s)
This paper presents a new method to generate and present arbitrarily directional binocular stereo... more This paper presents a new method to generate and present arbitrarily directional binocular stereo images from a sequence of omnidirectional images. A sequence of omnidirectional images is taken by moving an omnidirectional image sensor in a static real environment. The motion of the omnidirectional image sensor is constrained to a plane. The sensor's route and speed are known. In the proposed method, a fixed length of the sequence is buffered in a computer to generate arbitrarily directional binocular stereo images by combining captured rays. Using the method a user can look around a scene in the distance with rich 3D sensation without significant time delay. This paper describes the principle of real-time generation of binocular stereo images. In addition, we introduce a prototype telepresence system of view-dependent stereo image generation and presentation
We focus on a shared augmented environment (SAE) as an almost ideal face-to-face collaborative vi... more We focus on a shared augmented environment (SAE) as an almost ideal face-to-face collaborative virtual workspace. In an SAE, multiple users can observe both a virtual world and real partners through optical see-through head mounted displays (STHMDs). This paper describes two experiments for verifying the effectiveness of an SAE compared with a conventional shared virtual environment (SVE) and exploring improvement of them. Through the experiments, the effectiveness of an SAE compared with an SVE was confirmed. It was also confirmed that enhancement of the shared environment with computer graphics, i.e. displaying a partner's body in an SVE, drawing a line as a partner's viewing direction and emphasizing virtual objects to which a partner pay attention, improves workers' feeling and collaboration efficiency.
Describes a prototype of an augmented reality system using vision-based registration. In order to... more Describes a prototype of an augmented reality system using vision-based registration. In order to build an augmented reality system with video see-through image composition, camera parameters for generating virtual objects must be obtained at video-rate. The system estimates camera parameters from four known markers in an image sequence captured by a small CCD camera mounted on a HMD (head mounted display). Virtual objects are overlaid upon a captured image sequence in real-time
A real-time three-dimensional position and orientation tracking system is proposed for use with w... more A real-time three-dimensional position and orientation tracking system is proposed for use with wearable augmented-reality systems. The system combines infrared markers with a head-mounted stereo camera to detect the user's position, and an orientation sensor to measure the orientation of the user's head. An extended Kalman filter is employed to reduce triangulation and orientation error by integrating the signals acquired by multiple sensors. The system is evaluated through a series of experiments, and a navigation system implemented using the proposed scheme is present. The accuracy of the system is shown to be sufficient to allow annotations and virtual objects to be overlaid on real scenes via a head-mounted display without confusion.
Recently virtual reality (VR) systems have been incorporating rich information available in the r... more Recently virtual reality (VR) systems have been incorporating rich information available in the real world into VR environments in order to improve their reality. This stream has created the field of mixed reality which seamlessly integrates real and virtual worlds. This paper describes a novel approach to the construction of a mixed environment. The approach is based on capturing the dynamic real world by using a video-rate omnidirectional stereo image sensor. The mixed environment is constructed of two different types of models: (1) texture-mapped cylindrical 3-D model of dynamic real scenes and (2) 3-D computer graphics (CG) model. The cylindrical 3-D model is generated from full panoramic stereo images obtained by the omnidirectional sensor that uses a pair of hexagonal pyramidal mirrors and twelve CCD cameras. A prototype system has been developed to confirm the feasibility of the proposed method, in which panoramic binocular stereo images of the mixed environment are projected on a cylindrical immersive display depending on user's viewpoint in real time.
Uploads
Papers by Haruo Takemura
framework using robot-gaze interaction that support patients
with motor paralysis conditions. A patient can give commands
by gazing to the target object and a robot starts to search au-
tonomously. Apart from multiple gaze interaction, our approach
uses few gaze interaction to specify location clue and object clue
and thus integrates the RGB-D sensing to segment unknown
objects from the environment. Based on hypotheses from gaze
information, we utilize multiregion Graph Cuts method along
with an analysis of depth information. Furthermore, our search
algorithm allows a robot to find a main observation point which
is the point where user can clearly observe the target object.
If a first segmentation was not satisfied by the user, the robot
is able to adapt its pose to find different views of object. The
approach has been implemented and tested on the humanoid
robot ENON. With a few gaze guidance, the success rate of
segmentation of unknown objects was achieved to be 85%. The
experimental results confirm its applicability on a wide variety
of objects even when the target object was occluded by another
object.
framework using robot-gaze interaction that support patients
with motor paralysis conditions. A patient can give commands
by gazing to the target object and a robot starts to search au-
tonomously. Apart from multiple gaze interaction, our approach
uses few gaze interaction to specify location clue and object clue
and thus integrates the RGB-D sensing to segment unknown
objects from the environment. Based on hypotheses from gaze
information, we utilize multiregion Graph Cuts method along
with an analysis of depth information. Furthermore, our search
algorithm allows a robot to find a main observation point which
is the point where user can clearly observe the target object.
If a first segmentation was not satisfied by the user, the robot
is able to adapt its pose to find different views of object. The
approach has been implemented and tested on the humanoid
robot ENON. With a few gaze guidance, the success rate of
segmentation of unknown objects was achieved to be 85%. The
experimental results confirm its applicability on a wide variety
of objects even when the target object was occluded by another
object.