2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021
Learning robust policies for robotic systems operating in presence of uncertainty is a challengin... more Learning robust policies for robotic systems operating in presence of uncertainty is a challenging task. For safe navigation, in addition to the natural stochasticity of the environment and vehicle dynamics, the perception uncertainty associated with dynamic entities, e.g. pedestrians, must be accounted for during motion planning. To this end, we construct an algorithm with built-in robustness to uncertainty by directly minimizing an upper confidence bound on the expected cost of trajectories instead of employing a standard approach based on minimizing the expected cost itself. Perception uncertainty is incorporated into the policy search framework by predicting each pedestrian’s intent belief and propagating their state distribution in time using closed-loop goal-directed dynamics. We train the policy in simulation and show that it could be transferred to an agile ground vehicle for successful autonomous robot navigation in presence of pedestrians with perception uncertainty. We further show the superior performance of this policy over a policy that does not consider pedestrian intent and perception uncertainty.
Prospection is an important part of how humans come up with new task plans, but has not been expl... more Prospection is an important part of how humans come up with new task plans, but has not been explored in depth in robotics. Predicting multiple task-level is a challenging problem that involves capturing both task semantics and continuous variability over the state of the world. Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment. In this work, we propose a method for learning a model encoding just such a representation for task planning. We learn a neural net that encodes the k most likely outcomes from high level actions from a given world. Our approach creates comprehensible task plans that allow us to predict changes to the environment many time steps into the future. We demonstrate this approach via application to a stacking task in a cluttered environment, where the robot must select between different colored blocks while avoiding...
SpringerBriefs in Electrical and Computer Engineering, 2017
There have been many developments in brain-machine interfaces (BMI) for controlling upper limb mo... more There have been many developments in brain-machine interfaces (BMI) for controlling upper limb movements such as reaching and grasping. One way to expand the usefulness of BMIs in replacing motor functions for patients with spinal cord injuries and neuromuscular disorders would be to improve the dexterity of upper limb movements performed by including more control of individual finger movements. Many studies have been focusing on understanding the organization of movement control in the sensorimotor cortex of the human brain. Finding the specific mechanisms for neural control of different movements will help focus signal acquisition and processing so as to improve BMI control of complex actions. In a recently published study, we demonstrated, for the first time, online BMI control of individual finger movements using electrocorticography recordings from the hand area of sensorimotor cortex. This study expands the possibilities for combined control of arm movements and more dexterous hand and finger movements.
We describe the recent development of assistive computer vision algorithms for use with the Argus... more We describe the recent development of assistive computer vision algorithms for use with the Argus II retinal prosthesis system. While users of the prosthetic system can learn and adapt to the limited stimulation resolution, there exists great potential for computer vision algorithms to augment the experience and significantly increase the utility of the system for the user. To this end, our recent work has focused on helping with two different challenges encountered by the visually impaired: face detection and object recognition. In this paper, we describe algorithm implementations in both of these areas that make use of the retinal prosthesis for visual feedback to the user, and discuss the unique challenges faced in this domain.
Human-aware robot navigation promises a range of applications in which mobile robots bring versat... more Human-aware robot navigation promises a range of applications in which mobile robots bring versatile assistance to people in common human environments. While prior research has mostly focused on modeling pedestrians as independent, intentional individuals, people move in groups; consequently, it is imperative for mobile robots to respect human groups when navigating around people. This paper explores learning group-aware navigation policies based on dynamic group formation using deep reinforcement learning. Through simulation experiments, we show that group-aware policies, compared to baseline policies that neglect human groups, achieve greater robot navigation performance (e.g., fewer collisions), minimize violation of social norms and discomfort, and reduce the robot’s movement impact on pedestrians. Our results contribute to the development of social navigation and the integration of mobile robots into human environments.
Fast, collision-free motion through unknown environments remains a challenging problem for roboti... more Fast, collision-free motion through unknown environments remains a challenging problem for robotic systems. In these situations, the robot's ability to reason about its future motion is often severely limited by sensor field of view (FOV). By contrast, biological systems routinely make decisions by taking into consideration what might exist beyond their FOV based on prior experience. In this paper, we present an approach for predicting occupancy map representations of sensor data for future robot motions using deep neural networks. We evaluate several deep network architectures, including purely generative and adversarial models. Testing on both simulated and real environments we demonstrated performance both qualitatively and quantitatively, with SSIM similarity measure up to 0.899. We showed that it is possible to make predictions about occupied space beyond the physical robot's FOV from simulated training data. In the future, this method will allow robots to navigate thro...
Abstract Revolutionizing Prosthetics is a government-sponsored program focused on maturing the ma... more Abstract Revolutionizing Prosthetics is a government-sponsored program focused on maturing the many foundational technologies that comprise neural prosthetic systems. Targeting the needs of amputees and movement-impaired individuals, the program focused on technological advancements in areas such as advanced neural recording devices, neural decoding and encoding algorithms, and upper limb prosthetic systems. A primary objective of the program was to support technological advancement of wearable prosthetic devices at the upper limb and hand level. The Modular Prosthetic Limb (MPL) is a representation of the vision for a highly anthropometric prosthetic limb. With 26 articulating joints, 17 actuators, and hundreds of internal sensors for feedback, the vision of the MPL was to replicate the dexterity, speed, and strength of the human hand to an extent never realized in a prosthetic device. Here we describe the MPL’s evolution over the past 13 years and describe the system in entirety, focusing on the fundamental characteristics of the system from hardware to software and controls. Additionally, we briefly touch upon some clinical applications and alternative use cases for the system to date.
2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016
This paper presents a nested marsupial robotic system and its execution of a notional disaster re... more This paper presents a nested marsupial robotic system and its execution of a notional disaster response task. Human supervised autonomy is facilitated by tightly-coupled, high-level user feedback enabling command and control of a bimanual mobile manipulator carrying a quadrotor unmanned aerial vehicle that carries a miniature ground robot. Each robot performs a portion of a mock hazardous chemical spill investigation and sampling task within a shipping container. This work offers an example application for a heterogeneous team of robots that could directly support first responder activities using complementary capabilities of autonomous dexterous manipulation and mobility, autonomous planning and control, and teleoperation. The task was successfully executed during multiple live trials at the DARPA Robotics Challenge Technology Expo in June 2015. A key contribution of the work is the application of a unified algorithmic approach to autonomous planning, control, and estimation supporting vision-based manipulation and non-GPS-based ground and aerial mobility, thus reducing algorithmic complexity across this capability set. The unified algorithmic approach is described along with the robot capabilities, hardware implementations, and human interface, followed by discussion of live demonstration execution and results.
2019 International Conference on Robotics and Automation (ICRA), 2019
Efficient exploration through unknown environments remains a challenging problem for robotic syst... more Efficient exploration through unknown environments remains a challenging problem for robotic systems. In these situations, the robot’s ability to reason about its future motion is often severely limited by sensor field of view (FOV). By contrast, biological systems routinely make decisions by taking into consideration what might exist beyond their FOV based on prior experience. We present an approach for predicting occupancy map representations of sensor data for future robot motions using deep neural networks. We develop a custom loss function used to make accurate prediction while emphasizing physical boundaries. We further study extensions to our neural network architecture to account for uncertainty and ambiguity inherent in mapping and exploration. Finally, we demonstrate a combined map prediction and information-theoretic exploration strategy using the variance of the generated hypotheses as the heuristic for efficient exploration of unknown environments.
2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018
This paper presents an autonomous perching concept for multirotor aerial vehicles. The Autonomous... more This paper presents an autonomous perching concept for multirotor aerial vehicles. The Autonomous Grasping Robotic Aerial System for Perching (AGRASP)represents a novel integration of robotics perception, vision-based path planning, and biomimetically-inspired manipulation on a small, lightweight aerial robot with highly-constrained sensor and processing capacity. Computationally lightweight perception algorithms pull candidate perch structures out of a complex environment with no a priori knowledge of the operational space. The innovative manipulator design combines both active grasp and passive grip enabling it to maintain hold on the perch even with all power off. We experimentally demonstrate, for the first time, a quadrotor autonomously detecting and landing on a perch relying solely on onboard sensing and processing.
Investigative ophthalmology & visual science, Feb 1, 2018
Visual scanning by sighted individuals is done using eye and head movements. In contrast, scannin... more Visual scanning by sighted individuals is done using eye and head movements. In contrast, scanning using the Argus II is solely done by head movement, since eye movements can introduce localization errors. Here, we tested if a scanning mode utilizing eye movements increases visual stability and reduces head movements in Argus II users. Eye positions were measured in real-time and were used to shift the region of interest (ROI) that is sent to the implant within the wide field of view (FOV) of the scene camera. Participants were able to use combined eye-head scanning: shifting the camera by moving their head and shifting the ROI within the FOV by eye movement. Eight blind individuals implanted with the Argus II retinal prosthesis participated in the study. A white target appeared on a touchscreen monitor and the participants were instructed to report the location of the target by touching the monitor. We compared the spread of the responses, the time to complete the task, and the amo...
2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021
Learning robust policies for robotic systems operating in presence of uncertainty is a challengin... more Learning robust policies for robotic systems operating in presence of uncertainty is a challenging task. For safe navigation, in addition to the natural stochasticity of the environment and vehicle dynamics, the perception uncertainty associated with dynamic entities, e.g. pedestrians, must be accounted for during motion planning. To this end, we construct an algorithm with built-in robustness to uncertainty by directly minimizing an upper confidence bound on the expected cost of trajectories instead of employing a standard approach based on minimizing the expected cost itself. Perception uncertainty is incorporated into the policy search framework by predicting each pedestrian’s intent belief and propagating their state distribution in time using closed-loop goal-directed dynamics. We train the policy in simulation and show that it could be transferred to an agile ground vehicle for successful autonomous robot navigation in presence of pedestrians with perception uncertainty. We further show the superior performance of this policy over a policy that does not consider pedestrian intent and perception uncertainty.
Prospection is an important part of how humans come up with new task plans, but has not been expl... more Prospection is an important part of how humans come up with new task plans, but has not been explored in depth in robotics. Predicting multiple task-level is a challenging problem that involves capturing both task semantics and continuous variability over the state of the world. Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment. In this work, we propose a method for learning a model encoding just such a representation for task planning. We learn a neural net that encodes the k most likely outcomes from high level actions from a given world. Our approach creates comprehensible task plans that allow us to predict changes to the environment many time steps into the future. We demonstrate this approach via application to a stacking task in a cluttered environment, where the robot must select between different colored blocks while avoiding...
SpringerBriefs in Electrical and Computer Engineering, 2017
There have been many developments in brain-machine interfaces (BMI) for controlling upper limb mo... more There have been many developments in brain-machine interfaces (BMI) for controlling upper limb movements such as reaching and grasping. One way to expand the usefulness of BMIs in replacing motor functions for patients with spinal cord injuries and neuromuscular disorders would be to improve the dexterity of upper limb movements performed by including more control of individual finger movements. Many studies have been focusing on understanding the organization of movement control in the sensorimotor cortex of the human brain. Finding the specific mechanisms for neural control of different movements will help focus signal acquisition and processing so as to improve BMI control of complex actions. In a recently published study, we demonstrated, for the first time, online BMI control of individual finger movements using electrocorticography recordings from the hand area of sensorimotor cortex. This study expands the possibilities for combined control of arm movements and more dexterous hand and finger movements.
We describe the recent development of assistive computer vision algorithms for use with the Argus... more We describe the recent development of assistive computer vision algorithms for use with the Argus II retinal prosthesis system. While users of the prosthetic system can learn and adapt to the limited stimulation resolution, there exists great potential for computer vision algorithms to augment the experience and significantly increase the utility of the system for the user. To this end, our recent work has focused on helping with two different challenges encountered by the visually impaired: face detection and object recognition. In this paper, we describe algorithm implementations in both of these areas that make use of the retinal prosthesis for visual feedback to the user, and discuss the unique challenges faced in this domain.
Human-aware robot navigation promises a range of applications in which mobile robots bring versat... more Human-aware robot navigation promises a range of applications in which mobile robots bring versatile assistance to people in common human environments. While prior research has mostly focused on modeling pedestrians as independent, intentional individuals, people move in groups; consequently, it is imperative for mobile robots to respect human groups when navigating around people. This paper explores learning group-aware navigation policies based on dynamic group formation using deep reinforcement learning. Through simulation experiments, we show that group-aware policies, compared to baseline policies that neglect human groups, achieve greater robot navigation performance (e.g., fewer collisions), minimize violation of social norms and discomfort, and reduce the robot’s movement impact on pedestrians. Our results contribute to the development of social navigation and the integration of mobile robots into human environments.
Fast, collision-free motion through unknown environments remains a challenging problem for roboti... more Fast, collision-free motion through unknown environments remains a challenging problem for robotic systems. In these situations, the robot's ability to reason about its future motion is often severely limited by sensor field of view (FOV). By contrast, biological systems routinely make decisions by taking into consideration what might exist beyond their FOV based on prior experience. In this paper, we present an approach for predicting occupancy map representations of sensor data for future robot motions using deep neural networks. We evaluate several deep network architectures, including purely generative and adversarial models. Testing on both simulated and real environments we demonstrated performance both qualitatively and quantitatively, with SSIM similarity measure up to 0.899. We showed that it is possible to make predictions about occupied space beyond the physical robot's FOV from simulated training data. In the future, this method will allow robots to navigate thro...
Abstract Revolutionizing Prosthetics is a government-sponsored program focused on maturing the ma... more Abstract Revolutionizing Prosthetics is a government-sponsored program focused on maturing the many foundational technologies that comprise neural prosthetic systems. Targeting the needs of amputees and movement-impaired individuals, the program focused on technological advancements in areas such as advanced neural recording devices, neural decoding and encoding algorithms, and upper limb prosthetic systems. A primary objective of the program was to support technological advancement of wearable prosthetic devices at the upper limb and hand level. The Modular Prosthetic Limb (MPL) is a representation of the vision for a highly anthropometric prosthetic limb. With 26 articulating joints, 17 actuators, and hundreds of internal sensors for feedback, the vision of the MPL was to replicate the dexterity, speed, and strength of the human hand to an extent never realized in a prosthetic device. Here we describe the MPL’s evolution over the past 13 years and describe the system in entirety, focusing on the fundamental characteristics of the system from hardware to software and controls. Additionally, we briefly touch upon some clinical applications and alternative use cases for the system to date.
2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016
This paper presents a nested marsupial robotic system and its execution of a notional disaster re... more This paper presents a nested marsupial robotic system and its execution of a notional disaster response task. Human supervised autonomy is facilitated by tightly-coupled, high-level user feedback enabling command and control of a bimanual mobile manipulator carrying a quadrotor unmanned aerial vehicle that carries a miniature ground robot. Each robot performs a portion of a mock hazardous chemical spill investigation and sampling task within a shipping container. This work offers an example application for a heterogeneous team of robots that could directly support first responder activities using complementary capabilities of autonomous dexterous manipulation and mobility, autonomous planning and control, and teleoperation. The task was successfully executed during multiple live trials at the DARPA Robotics Challenge Technology Expo in June 2015. A key contribution of the work is the application of a unified algorithmic approach to autonomous planning, control, and estimation supporting vision-based manipulation and non-GPS-based ground and aerial mobility, thus reducing algorithmic complexity across this capability set. The unified algorithmic approach is described along with the robot capabilities, hardware implementations, and human interface, followed by discussion of live demonstration execution and results.
2019 International Conference on Robotics and Automation (ICRA), 2019
Efficient exploration through unknown environments remains a challenging problem for robotic syst... more Efficient exploration through unknown environments remains a challenging problem for robotic systems. In these situations, the robot’s ability to reason about its future motion is often severely limited by sensor field of view (FOV). By contrast, biological systems routinely make decisions by taking into consideration what might exist beyond their FOV based on prior experience. We present an approach for predicting occupancy map representations of sensor data for future robot motions using deep neural networks. We develop a custom loss function used to make accurate prediction while emphasizing physical boundaries. We further study extensions to our neural network architecture to account for uncertainty and ambiguity inherent in mapping and exploration. Finally, we demonstrate a combined map prediction and information-theoretic exploration strategy using the variance of the generated hypotheses as the heuristic for efficient exploration of unknown environments.
2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018
This paper presents an autonomous perching concept for multirotor aerial vehicles. The Autonomous... more This paper presents an autonomous perching concept for multirotor aerial vehicles. The Autonomous Grasping Robotic Aerial System for Perching (AGRASP)represents a novel integration of robotics perception, vision-based path planning, and biomimetically-inspired manipulation on a small, lightweight aerial robot with highly-constrained sensor and processing capacity. Computationally lightweight perception algorithms pull candidate perch structures out of a complex environment with no a priori knowledge of the operational space. The innovative manipulator design combines both active grasp and passive grip enabling it to maintain hold on the perch even with all power off. We experimentally demonstrate, for the first time, a quadrotor autonomously detecting and landing on a perch relying solely on onboard sensing and processing.
Investigative ophthalmology & visual science, Feb 1, 2018
Visual scanning by sighted individuals is done using eye and head movements. In contrast, scannin... more Visual scanning by sighted individuals is done using eye and head movements. In contrast, scanning using the Argus II is solely done by head movement, since eye movements can introduce localization errors. Here, we tested if a scanning mode utilizing eye movements increases visual stability and reduces head movements in Argus II users. Eye positions were measured in real-time and were used to shift the region of interest (ROI) that is sent to the implant within the wide field of view (FOV) of the scene camera. Participants were able to use combined eye-head scanning: shifting the camera by moving their head and shifting the ROI within the FOV by eye movement. Eight blind individuals implanted with the Argus II retinal prosthesis participated in the study. A white target appeared on a touchscreen monitor and the participants were instructed to report the location of the target by touching the monitor. We compared the spread of the responses, the time to complete the task, and the amo...
Uploads
Papers by Kapil Katyal