Abstract

When developing planning and control algorithms for autonomous systems, we often assume that these algorithms will be the only significant source of inputs to the system. However, in some applications there may be an additional "driver", external to the autonomy pipeline, that has a large impact on the performance and safety of the overall system. In this talk, I will discuss two projects that investigate different aspects of this problem.

The first considers the effects of an external driver that acts alongside the autonomy but without any shared objective. For example, the aerodynamic forces acting on a micro aerial vehicle flying through a strong wind field can dramatically alter the vehicle’s motion, leading to violations of safety or operational constraints. Limited onboard computation also restricts modeling or incorporation of these nonlinear dynamics into the autonomy. This leads to the idea of Experience-driven Predictive Control (EPC). EPC builds on ideas from adaptive control and model predictive control by accumulating experience on how an external driver impacts the system while simultaneously leveraging that experience to ensure that constraints are met in a computationally efficient manner.

The second project considers the case where the external driver is actually the primary operator of the system, while the autonomy aims to assist this driver to safely achieve a common objective. A prime example of this is the next generation of advanced driver assistance systems that will be able to employ techniques developed for fully autonomous driving but in the context of keeping a human driver safe. The Toyota Guardian system is TRI’s novel approach to this problem, building on ideas from a variety of domains, ranging from aircraft control to shared autonomy. I will show a few preliminary examples of how this combination of the human driver and the autonomy can achieve superhuman performance and safety.

Bio

Vishnu Desaraju is a Senior Research Scientist at the Toyota Research Institute, Ann Arbor, MI working on automated driving technologies. He received a B.S.E. in Electrical Engineering from the University of Michigan in 2008, an S.M. in Aeronautics and Astronautics from MIT in 2010, and an M.S. and Ph.D. in Robotics from Carnegie Mellon University in 2015 and 2017, respectively. He received the AIAA Guidance, Navigation, and Control Best Paper award from SciTech 2018. His research interests include developing computationally efficient motion planning and feedback control algorithms for agile autonomous systems, including autonomous cars, boats, and micro air vehicles, with a focus on mitigating the effects of uncertainty to achieve safe and reliable operation in the field.

Abstract

Learning contact-rich, robotic manipulation skills is a challenging problem due to the high-dimensionality of the state and action space as well as uncertainty from noisy sensors and inaccurate motor control. In our research, we explore what representations of raw perceptual data enable a robot to better learn and perform these skills. Specifically for manipulation robots, the sense of touch is essential yet it is non-trivial to manually design a robot controller that combines different sensing modalities that have very different characteristics. I will present our set of research works that explore the question of how to best fuse the information from vision and touch for contact-rich manipulation tasks. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of visual and haptic sensory inputs, which can then be used to improve the sample efficiency of policy learning. I present experiments on a peg insertion task where the learned policy generalises over different geometry, configurations, and clearances, while being robust to external perturbations. While this work has shown very promising results on fusing vision and touch into a learned latent representation, this representation is also not interpretable. In follow-up work, we present a multimodal fusion algorithm that exploits a differentiable filtering framework for tracking the state of manipulated objects and therefore facilitating longer horizon planning. We also propose a framework where a robot can exploit information from failed manipulation attempts to recover and re-try. And finally, we show how exploiting multiple modalities helps to compensate for corrupted sensory data in one of the modalities. I will conclude this talk with a discussion of appropriate representations for multimodal sensory data.

Bio

Jeannette Bohg is an Assistant Professor of Computer Science at Stanford University. She was a group leader at the Autonomous Motion Department (AMD) of the MPI for Intelligent Systems until September 2017. Before joining AMD in January 2012, Jeannette Bohg was a PhD student at the Division of Robotics, Perception and Learning (RPL) at KTH in Stockholm. In her thesis, she proposed novel methods towards multi-modal scene understanding for robotic grasping. She also studied at Chalmers in Gothenburg and at the Technical University in Dresden where she received her Master in Art and Technology and her Diploma in Computer Science, respectively. Her research focuses on perception and learning for autonomous robotic manipulation and grasping. She is specifically interested in developing methods that are goal-directed, real-time and multi-modal such that they can provide meaningful feedback for execution and learning. Jeannette Bohg has received several awards, most notably the 2019 IEEE International Conference on Robotics and Automation (ICRA) Best Paper Award, the 2019 IEEE Robotics and Automation Society Early Career Award and the 2017 IEEE Robotics and Automation Letters (RA-L) Best Paper Award.