Kinect

This paper presents a non-intrusive system that gives the illusion of a 3D immersive and interactive environment with 2D projectors. The user does not need to wear glasses, nor to watch a (limited) screen. The virtual world is all around him, drawn on the floor. As the user is himself immersed in the virtual world, there is no need for a proxy like an avatar; he can move inside the virtual environment freely. Moreover, the I-see-3D system allows a user to manipulate virtual objects with his own body, making interactions with the virtual world very intuitive. Giving the illusion of 3D requires to render images in such a way that the deformation of the image projected on the floor is taken into account, as well as the position of the user’s “eye” in its virtual world. The resulting projection is neither perspective nor orthographic. Nevertheless, we describe how this can be implemented with the standard OpenGL pipeline, without any shader. Our experiments demonstrate that our system i...

This paper presents a home-based Senior Fitness Test (SFT) measurement system by using an inertial sensor and a depth camera in a collaborative way. The depth camera is used to monitor the correct pose of a subject for a fitness test and any deviation from the correct pose while the inertial sensor is used to measure the number of a fitness test action performed by the subject within the time duration specified by the fitness protocol. The results indicate that this collaborative approach leads to high success rates in providing the SFT measurements under realistic conditions.

— This paper presents an approach to detect and recognize actions of interest in real-time from a continuous stream of data that are captured simultaneously from a Kinect depth camera and a wearable inertial sensor. Actions of interest are considered to appear continuously and in a random order among actions of non-interest. Skeleton depth images are first used to separate actions of interest from actions of non-interest based on pause and motion segments. Inertial signals from a wearable inertial sensor are then used to improve the recognition outcome. A dataset consisting of simultaneous depth and inertial data for the smart TV actions of interest occurring continuously and in a random order among actions of non-interest is studied and made publicly available. The results obtained indicate the effectiveness of the developed approach in coping with actions that are performed realistically in a continuous manner. Keywords— Real-time continuous action detection; action recognition from continuous data streams; simultaenous utilization of depth images and inertial signals for action recognition.

Abstract. In RoboCup Middle Size league (MSL) the main referee uses assisting technology, controlled by a second referee, to support him, in particular for conveying referee decisions for robot players with the help of a wireless communication system. In this paper a vision-based system is introduced, able to interpret dynamic and static gestures of the referee, thus eliminating the need for a second one. The referee's ges-tures are interpreted by the system and sent directly to the Referee Box, which sends the proper commands to the robots. The system is divided into four modules: a real time hand tracking and feature extraction, a SVM (Support Vector Machine) for static hand posture identification, an HMM (Hidden Markov Model) for dynamic unistroke hand gesture recognition, and a FSM (Finite State Machine) to control the various system states transitions. The experimental results showed that the system works very reliably, being able to recognize the combination of gestures a...

This article describes the design and development ofa system for remote indoor 3D monitoring using an undetermined number of Microsoft® Kinect sensors. In the proposed client-server system, the Kinect cameras can be connected to different computers, addressing this way the hardware limitation of one sensor per USB controller. The reason behind this limitation is the high bandwidth needed by the sensor, which becomes also an issue for the distributed system TCP/IP communications. Since traffic volume is too high, 3D data has to be compressed before it can be sent over the network. The solution consists in self-coding the Kinect data into RGB images and then using a standard multimedia codec to compress color maps. Information from different sources is collected into a central client computer, where point clouds are transformed to reconstruct the scene in 3D. An algorithm is proposed to conveniently merge the skeletons detected locally by each Kinect, so that monitoring of people is robust to self and inter-user occlusions. Final skeletons are labeled and trajectories of every joint can be saved for event reconstruction or further analysis.

We focus on generating consistent reconstructions of indoor spaces from a freely moving handheld RGB-D sensor, with the aim of creating virtual models that can be used for measuring and remodeling. We propose a novel 6D RGBD odometry approach that finds the relative camera pose between consecutive RGB-D frames by keypoint extraction and feature matching both on the RGB and depth image planes. Furthermore, we feed the estimated pose to the highly accurate KinectFusion algorithm, which uses a fast ICP (Iterative-Closest-Point) to fine-tune the frame-to-frame relative pose and fuse the Depth data into a global implicit surface. We evaluate our method on a publicly available RGB-D SLAM benchmark dataset by Sturm et al. The experimental results show that our proposed reconstruction method solely based on visual odometry and KinectFusion outperforms the state-of-the-art RGB-D SLAM system accuracy. Our algorithm outputs a ready-to-use polygon mesh (highly suitable for creating 3D virtual worlds) without any post-processing steps.

The Follow-me bot is primarily humanoid with multiple degrees of freedom(one you might have seen in Real steel movie by Hugh Jackman). From the get go site, it has all the earmarks of being much the same as any other humanoid robot however the true contrast comes in when we go inside its working system. Follow-me bot lives up to expectations and follow the rules guideline for remote human type robot interface. It will take client's body movement as info and move much the same as him/her. It's more like offering sight to the robot and it'll mimicries client's movement. Dissimilar to another humanoid robot, the controller in this present bot's structural planning is the client himself. Controllers can steer correspond with the machine simply depending on body developments, and along these lines control the activities of remote robot progressively, coordination, and harmonization. The outline is a synthesis of new human movement catch strategies, sagacious executor controltechnique, system transmission, different sensor combination innovation. The name 'Follow-me bot' is utilized on the grounds that much the same as our follow-me, it'll move definitely in the same manner as we seem to be. This paper quickly portrays all the portions of the humanoid robot utilizing Kinect engineering.

Hoy en día las TIC (tecnologías de la información y la comunicación) son tan
usuales en la vida cotidiana que las encontramos en casi todos los aspectos,
desde mandar un simple SMS, hasta realizar una videoconferencia por internet a
través de un dispositivo móvil. Estas tecnologías han cambiado la forma en que
las personas llevan a cabo sus actividades, que él no contar con nociones básicas
de cómo utilizar un dispositivo electrónico (computadora, celular, tableta, etc.)
podría limitar las posibilidades de desarrollo y/o interacción social de las mismas.

— In this paper we present a Computer Vision based system developed to operate a manipulator. Images, representing some pre-configured gestures, are captured using a RGB-D sensor. A Machine Learning technique, known as Support Vector Machines (SVMs), is used to process these images. Each gesture indicates a different movement to be executed by the manipulator. In this system, each image is classified in a gesture and the respective command is sent to this equipment. Results show that it was possible to capture, process, classify and send the correct command to the manipulator to execute the predetermined movements.

Resumo— Neste trabalho é apresentado um sistema para operação de manipuladores robóticos baseado em técnicas de Visão Computacional. As imagens, representativas de gestos prestabelecidos, são capturadas por meio do sensor do tipo RGB-D e processadas por uma técnica de Aprendizado de Máquina, conhecida como Máquina de Vetores de Suporte (Support Vector Machines ou SVM). Uma vez classificados, cada gesto indica um movimento diferente a ser executado pelo manipulador robótico. Os comandos são enviados para esse dispositivo via comunicação serial. Os resultados obtidos demonstram que foi possível a captura, processamento, classificação e envio das informações corretas ao manipulador para execução dos movimentos predeterminados.

This mixed-methods study investigated the ability of motion-based video games (MBVG) to motivate adoption of authentic wall/rock climbing. The study also explored participants’ perceptions of MBVG and authentic wall/rock climbing. University students (n = 24) with no previous climbing experience were randomly assigned into the game-first (GF) or wall-first (WF) group. Both groups completed an initial questionnaire and two 30minute sessions of their initial activity. Afterwards, both groups completed the same questionnaire and switched activities. After finishing the second activity, participants completed a post-assessment and twelve weeks later completed a follow-up questionnaire. Two one-hour focus groups were also conducted after the second activity. Findings indicated that the MBVG did not motivate participants to pursue future wall/rock climbing; rather, the authentic experience motivated future activity. The participants did not have accurate perceptions of the mental or physi...

The detection and classification of human movements, as a joint field of Computer Vision and Pattern Recognition, is used with an increasing rate in applications designed to describe human activity. Such applications require efficient methods and tools for the automatic analysis and classification of motion capture data, which constitute an active field of research. To facilitate the development and the benchmarking of methods for action recognition, several video collections have previously been proposed. In this paper, we present a new video database that can be used for an objective comparison and evaluation of different motion analysis and classification methods. The database contains video clips that capture the 3D motion of individuals. To be more specific, the set consists of 8374 video clips, which contain 12 different types of tennis actions performed by 55 individuals, captured by Kinect. Kinect provides the depth map of motion data and helps to extract the 3D skeletal joint connections. Performing experiments using state of the art algorithms, the database shows to be very challenging. It contains very similar to each other actions, offering the opportunity to algorithms dedicated to gaming and athletics, to be developed and tested. The database is freely available for research purposes.

A number of review or survey articles have previously appeared on human action recognition where either vision sensors or inertial sensors are used individually. Considering that each sensor modality has its own limitations, in a number of previously published papers, it has been shown that the fusion of vision and inertial sensor data improves the accuracy of recognition. This survey article provides an overview of the recent investigations where both vision and inertial sensors are used together and simultaneously to perform human action recognition more effectively. The thrust of this survey is on the utilization of depth cameras and inertial sensors as these two types of sensors are cost-effective, commercially available, and more significantly they both provide 3D human action data. An overview of the components necessary to achieve fusion of data from depth and inertial sensors is provided. In addition, a review of the publicly available datasets that include depth and inertial data which are simultaneously captured via depth and inertial sensors is presented.

Human action recognition is gaining interest from many computer vision researchers because of its wide variety of potential applications. For instance: surveillance, advanced human computer interaction, content-based video retrieval, or athletic performance analysis. In this research, we focus to recognize some human actions such as waving, punching, clapping, etc. We choose exemplar-based sequential single-layered approach using Dynamic Time Warping (DTW) because of its robustness against variation in speed or style in performing action. For improving recognition rate, we perform body part tracking using depth camera to recover human joints body part information in 3D real world coordinate system. We build our feature vector from joint orientation along time series that invariant to human body size. Dynamic Time Warping is then applied to the resulted feature vector. We examine our approach to recognize several actions and we confirm our method can work well with several experiments. Further experiment for benchmarking the result will be held in near future.

GitHub Repository: https://github.com/SeyedMuhammadHosseinMousavi/Introduction-to-Facial-Micro-Expressions-Analysis-Using-Color-and-Depth-Images-a-Matlab-Coding-Appr The book attempts to introduce a gentle introduction to the field of Facial Micro Expressions Recognition (FMER) using Color and Depth images, with the aid of MATLAB programming environment. FMER is a subset of image processing and it is a multidisciplinary topic to analysis. So, it requires familiarity with other topics of Artifactual Intelligence (AI) such as machine learning, digital image processing, psychology and more. So, it is a great opportunity to write a book which covers all of these topics for beginner to professional readers in the field of AI and even without having background of AI. Our goal is to provide a standalone introduction in the field of MFER analysis in the form of theorical descriptions for readers with no background in image processing with reproducible Matlab practical examples. Also, we describe any basic definitions for FMER analysis and MATLAB library which is used in the text, that helps final reader to apply the experiments in the real-world applications. We believe that this book is suitable for students, researchers, and professionals alike, who need to develop practical skills, along with a basic understanding of the field. We expect that, after reading this book, the reader feels comfortable with different key stages such as color and depth image processing, color and depth image representation, classification, machine learning, facial micro expressions recognition, feature extraction and dimensionality reduction. The book attempts to introduce a gentle introduction to the field of Facial Micro Expressions Recognition (FMER) using Color and Depth images, with the aid of MATLAB programming environment. FMER is a subset of image processing and it is a multidisciplinary topic to analysis. So, it requires familiarity with other topics of Artificial Intelligence (AI) such as machine learning, digital image processing, psychology and more. So, it is a great opportunity to write a book which covers all of these topics for beginner to professional readers in the field of AI and even without having background of AI. My goal is to provide a standalone introduction in the field of FMER analysis in the form of theorical descriptions for readers with no background in image processing with reproducible MATLAB practical examples. Also, the book describes any basic definitions for FMER analysis and MATLAB library which is used in the text, that helps final reader to apply the experiments in the real-world applications. This book is suitable for students, researchers, and professionals alike, who need to develop practical skills, along with a basic understanding of the field. It is expected that, after reading this book, the reader feels comfortable with different key stages such as color and depth image processing, color and depth image representation, classification, machine learning, facial micro expressions recognition, feature extraction and dimensionality reduction. This book is product of several years of researches and experiments and reflects the mindset of the authors for understanding this field as easier as possible. The author encourages the reader to contact him with any comments and suggestions for improvement.

In this study, the functionality of an autonomous mobile robot is improved by interfacing it with Microsoft Kinect for finding and tracking a previously identified human target while avoiding obstacles both while searching and tracking.  A suitable platform for fixing Microsoft Kinect and a laptop on a mobile robot is designed considering simplicity and practicality. The human identification and tracking methodology and algorithm is analysed for real life scenarios. The developed algorithm is applied to the mobile robot using robotics software and evaluated through real life scenarios achieving pleasing results. The mobile robot is able to identify humans, search for them and track them while avoiding static obstacles both while searching and tracking at indoor environments where no direct sunlight is present.

Human action recognition has a wide range of applications in-cluding biometrics, surveillance, and human computer interaction. The use of multimodal sensors for human action recognition is steadily increasing. However, there are limited publicly available datasets where depth camera and inertial sensor data are captured at the same time. This paper describes a freely available dataset, named UTD-MHAD, which consists of four temporally synchronized data modalities. These modalities include RGB videos, depth videos, skeleton positions, and inertial signals from a Kinect camera and a wearable inertial sensor for a comprehensive set of 27 human actions. Experimental results are provided to show how this database can be used to study fusion approaches that involve using both depth camera data and inertial sensor data. This public domain dataset is of benefit to multimodality research activities being conducted for human action recognition by various research groups. The dataset can be downloaded from http://www.utdallas.edu/~kehtar/UTD-MHAD.html

Cross-modal mappings of auditory stimuli reveal valuable insights into how humans make sense of sound and music. Whereas researchers have investigated cross-modal mappings of sound features varied in isolation within paradigms such as speeded classification and forced-choice matching tasks, investigations of representations of concurrently varied sound features (e.g., pitch, loudness and tempo) with overt gestures—accounting for the intrinsic link between movement and sound—are scant. To explore the role of bodily gestures in cross-modal mappings of auditory stimuli we asked 64 musically trained and untrained participants to represent pure tones—continually sounding and concurrently varied in pitch, loudness and tempo—with gestures while the sound stimuli were played. We hypothesized musical training to lead to more consistent mappings between pitch and height, loudness and distance/height, and tempo and speed of hand movement and muscular energy. Our results corroborate previously ...

With the emergence of smart TVs, set-top boxes and public information screens over the last few years, there is an increasing demand to no longer use these appliances only for passive output. These devices can also be used to do text-based web search as well as other tasks which require some form of text input. However, the design of text entry interfaces for efficient input on such appliances represents a major challenge. With current virtual keyboard solutions we only achieve an average text input rate of 5.79 words per minute (WPM) while the average typing speed on a traditional keyboard is 38 WPM. Furthermore, so-called controller-free appliances such as Samsung's Smart TV or Microsoft's Xbox Kinect result in even lower average text input rates. We present SpeeG2, a multimodal text entry solution combining speech recognition with gesture-based error correction. Four innovative prototypes for the efficient controller-free text entry have been developed and evaluated. A quantitative evaluation of our SpeeG2 text entry solution revealed that the best of our four prototypes achieves an average input rate of 21.04 WPM (without errors), outperforming current state-of-the-art solutions for controller-free text input.

A video about the system can be found at: https://www.academia.edu/11590838/SpeeG2

The better quality pdf is downloadible from here: "http://www.leoalmanac.org/wp-content/uploads/2013/07/LEAVol19No3-Mailman.pdf
Through recent artistic practices and technology of interactive systems for music, composition and improvisation have more and more blended and interconnected with each other – a sui generis situation now called comprovisation. The concept of comprovisation applies equally well to the spontaneous generation and manipulation of computer graphics, especially as such graphics are systematically coordinated audio-visually, while being subject to spontaneous manipulation by a performer.
This is explained in terms of the Fluxations and FluxNoisations Human Body Interfaces, interactive dance systems that generate music and graphics spontaneously in response to hand and body movement. They enable spontaneous expressive shaping of coherent complexity and variety. Thus a multi-layered multimodal experience arises. The aesthetic experience of this multimedia spectacle relates to experiences of various prior music and visual art of the 19th, 20th, and 21st centuries, as well as to hypothetical physical realities. Thus a kind of cybernetic phenomenology of art is pursued and enacted through an embodied cybersynthesis of art with simulated alternate reality. Call it a pragmatic speculative realism, an adventurous technoetic ‘what if.’ As compared to more traditional forms of improvisation, the opportunities for risk-taking and aesthetic exploration rise to a new level of uncertainty, as the reactions of the improviser simultaneously draw from and target both visual and aural modalities, such that the intentions toward each fuse together."

This article deals with the philosophical idea of worldmaking pursued through techné, meaning the fusion of the technical means of artistic creation, theorizing, and analysis, but specifically occurring in a feedback loop involving introspection and computation (cybernetic phenomenology). The discussion ranges over a wide variety of topics, including (1) the history of music theory from Ancient Greece to the present (including a rebuttal to Daniel Chua’s (2001) account of music history); (2) emergent properties in music arising from self-organization (explored through the Ancient Chinese abstract strategy board game go); (3) the ontology of musical qualities (properties) and categories (including their relation to visual, tactile, and olfactory qualities and categories). 
Various repertoires, artists, and philosophies are referenced. The essay analyzes aspects of Ligeti’s Violin Concerto, Carter’s String Quartet No. 5, and Lachenmann’s Kinderspiel.  Connections to the author’s own previous analytical and theoretical work are also discussed in relation to his interactive algorithmic audio-visual works (such as Fluxations and FluxNOISations). The techné of these is discussed in terms of logistics as well as aesthetic influences, including Wagner, Liszt, Stravinsky, Schoenberg, Crawford, Babbitt, Carter, Xenakis, Parker, Reich, Ligeti, Truax, Oliveros, Kandinsky, Miro, Pollock, Pollock, Hofmann, Rothko, Louis, Frankenthaler, Moholy-Nagy, Newman, Richter, and Brakhage.  The essay touches on philosophical ideas of Bergson, Whitehead, Deleuze, Harman, and Bogost, and more fully engages the philosophies of Nelson Goodman and Hannah Arendt in connection with artistic creation as relating to the theorizing and analysis of artistic works.     

Contact the author at jmailman@alumni.uchicago.edu
Download the article through JSTOR: http://www.jstor.org/stable/10.7757/persnewmusi.54.1.0005
or http://tinyurl.com/zdvkllr

With the advent of new, low-cost 3D sensing hardware such as the Kinect, and continued efforts in advanced point cloud processing, 3D perception gains more and more importance in robotics, as well as other fields. In this paper we present one of our most recent initiatives in the areas of point cloud perception: PCL (Point Cloud Library - http://pointclouds.org). PCL presents an advanced and extensive approach to the subject of 3D perception, and it's meant to provide support for all the common 3D building blocks that applications need. The library contains state-of-the art algorithms for: filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation. PCL is supported by an international community of robotics and perception researchers. We provide a brief walkthrough of PCL including its algorithmic capabilities and implementation strategies.

Human action recognition is an important yet challenging task. This paper presents a low-cost descriptor called 3D Histograms of Texture (3DHoTs) to extract discriminant features from a sequence of depth maps. 3DHoTs are derived from projecting depth frames onto three orthogonal Cartesian planes, i.e., the frontal, side and top planes, and thus compactly characterize the salient information of a specific action, on which texture features are calculated to represent the action. Besides this fast feature descriptor, a new multi-class boosting classifier (MBC) is also proposed to efficiently exploit different kinds of features in a unified framework for action classification. Compared to the existing boosting frameworks, we add a new multi-class constraint into the objective function, which helps to maintain a better margin distribution by maximizing the mean of margin whereas still minimizing the variance of margin. Experiments on the MSRAction3D, MSRGesture3D, MSRActivity3D and UTD-MHAD datasets demonstrate that the proposed system combining 3DHoTs and MBC is superior to the state-of-the-art.

Human action recognition based on skeletons has wide applications in human-computer interaction and intelligent surveillance. However, view variations and noisy data bring challenges to this task. What's more, it remains a problem to effectively represent spatio-temporal skeleton sequences. To solve these problems in one goal, this work presents an enhanced skeleton visualization method for view invariant human action recognition. Our method consists of three stages. First, a sequence-based view invariant transform is developed to eliminate the effect of view variations on spatio-temporal locations of skeleton joints. Second, the transformed skeletons are visualized as a series of color images, which implicitly encode the spatio-temporal information of skeleton joints. Furthermore, visual and motion enhancement methods are applied on color images to enhance their local patterns. Third, a convolutional neural networks-based model is adopted to extract robust and discriminative features from color images. The final action class scores are generated by decision level fusion of deep features. Extensive experiments on four challenging datasets consistently demonstrate the superiority of our method.