Abstract

Robot grippers typically include mechanical intelligence (e.g., underactuation, compliance) or computational intelligence (e.g., fully actuated with a wide array of sensors). Next generation grippers and hands will require both intelligences to work in concert across applications with resilience and dexterity. This talk will introduce the concept of mechanical and computational intelligence co-design through example case studies that focus on the particular importance of embodied sensitivity as a feature of the co-design process. For example, the most recent work on the Smart Suction Cup, conducted largely by Dr. Jungpyo Lee, demonstrates how design decisions like the number of sensitive chambers influences the resultant robot arm controller as well as physical compliance and manufacturing feasibility and cost.

Speaker Bio

Dr. Hannah Stuart is an Associate Professor in the Department of Mechanical Engineering at the University of California at Berkeley. She received her BS in Mechanical Engineering at the George Washington University in 2011, and her MS and PhD in Mechanical Engineering at Stanford University in 2013 and 2018, respectively. Her research focuses on understanding the mechanics of physical interaction in order to better design systems for dexterous manipulation. Applications range from remote robotics to assistive orthotics. Recent awards include the NSF CAREER grant, NASA Early Career Faculty grant, Hellman Fellows Fund grant, and Johnson & Johnson Women in STEM2D grant. She is a Senior Member of IEEE.

Abstract

This talk addresses the use of reinforcement learning in two-player zero-sum Markov games with finite but large state spaces, for which the goal is to find minimax policies with “modest”' computation. We use the qualifier “modest” to mean that we seek to certify policies as optimal without exploring the full state-space of the game.

The approach followed is strongly motivated by Q-learning, which was proposed in the late 1980s to extend the single-player dynamic programming principle to model-free reinforcement learning by eliminating the need for a known transition model. Extensions of Q-learning to two-player zero-sum games appeared shortly after.  Since then, most of the work devoted to proving correctness of Q-learning relies on establishing that its iteration converges to a unique fixed-point of a Bellman-like equation, which generally requires exploring the full state-space.

We will see that, for zero-sum games, it is possible to construct provably correct optimal policies using algorithms inspired by Q-learning, without requiring convergence of the Q function over the whole state-space. In fact, the samples used to update the Q-function may not even explore the whole set of reachable states and, for certain classes of games, the fraction of explored states gets smaller and smaller as the size of the state-space increases.

Speaker Bio

João Pedro Hespanha received his Ph.D. degree in electrical engineering and applied science from Yale University, New Haven, Connecticut in 1998. From 1999 to 2001, he was Assistant Professor at the University of Southern California, Los Angeles. He moved to the University of California, Santa Barbara in 2002, where he currently holds a Distinguished Professor position with the Department of Electrical and Computer Engineering.

Dr. Hespanha is a Fellow of the International Federation of Automatic Control (IFAC) and of the IEEE. He was an IEEE distinguished lecturer from 2007 to 2013.  His current research interests include multi-agent control systems; game theory; optimization; distributed control over communication networks (also known as networked control systems); stochastic modeling in biology; and network security. Additional information about his research and publications available at https://web.ece.ucsb.edu/~hespanha/.