Tutorial: Tools for Robotic Reinforcement Learning

Hands-on RL for Robotics with EAGER and Stable-Baselines3

ICRA 2022, May 23-27, 2022 Philadelphia (PA), USA

Motivation

Reinforcement learning (RL) methods have received much attention due to impressive results in many robotic applications. While RL promises learning-based control of near-optimal behaviors in theory, successful learning can elude practitioners due to various implementation challenges. Even if the best-suited learning method was selected, learning performance can nonetheless disappoint due to badly chosen hyper-parameters or an unreliable implementation of the algorithm. Furthermore, a learning task can be made unnecessarily hard by incorrect specifications.
This full-day tutorial points-out these practical pitfalls and introduces the audience to the tools for robotic RL that will aid roboticists in successfully solving robotic learning tasks, both in simulation and the real-world.

Goals

We will cover the use of Engine Agnostic Gym Environment for Robotics (EAGER) to define and create tasks that work both in simulation and on a real robot, and then learn to use the Stable-Baselines3 (SB3) library to solve it with SOTA algorithms, following best practices.
This tutorial will cover: creating tasks in EAGER, basic usage of SB3, automatic hyperparameter optimization and managing RL experiments.

Requirements

Basic knowledge of reinforcement learning and python programming is required.

Schedule

Time Talk Comments
9:00-9:05 Introduction
9:05-10:00 Accelerating physics simulators for RL on CPU, GPU and TPU Invited Speaker: Erwin Coumans
10:00-10:30 Getting Started with Gym and Stable-Baselines3 Presenter: Antonin Raffin
10:30-11:00 Hands-on Session with SB3 with colab notebooks
11:00-11:10 Break
11:10-11:40 Getting Started with EAGER Presenter: Bas Van der Heijden
11:40-12:15 Hands-on Session with EAGER with colab notebooks
12:15-13:30 Lunch Break
13:30-14:30 safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning Invited Speaker: Angela Schoellig
14:30-15:00 Automatic Hyperparameter Optimization Presenter: Antonin Raffin
15:00-15:45 Hyperparameter Tuning with Optuna with colab notebooks
15:45-16:00 Break
16:00-16:30 EAGER: Adding new robots/custom processing Presenter: Bas Van der Heijden
16:30-17:00 RL Baselines Zoo Presenter: Antonin Raffin
17:00-18:00 EAGER Advanced usage with colab notebooks
18:00-18:10 Closing remarks

Speakers

Erwin Coumans @erwincoumans
Erwin Coumans is creator of the Bullet physics engine and member of the Google Brain team, where he is responsible for real-time physics simulation research and development, with a focus on robotics and machine learning.

Angela Schoellig @angelaschoellig
Angela Schoellig is an Associate Professor at the University of Toronto Institute for Aerospace Studies and a Faculty Member of the Vector Institute for Artificial Intelligence. She conducts research at the intersection of robotics, controls, and machine learning. Her goal is to enhance the performance, safety, and autonomy of robots by enabling them to learn from past experiments and from each other.

Organizers

Bas Van der Heijden @bheijden
Bas is a PhD candidate at TU Delft working on robotics and reinforcement learning. He is co-creator of the Engine Agnostic Gym Environment for Robotics (EAGER) toolkit.

Antonin Raffin @araffin2
Antonin Raffin is a Research Engineer in Robotics and Machine Learning at the German Aerospace Center (DLR). He was previously working on state representation learning in the ENSTA robotics lab (U2IS) where he co-created the Stable-Baselines library with Ashley Hill. His research focus is now on applying reinforcement learning directly on real robots, for which he continues to maintain the Stable-Baselines3 library.

Jens Kober
Jens Kober is an associate professor at the Cognitive Robotics department, 3mE, TU Delft, Netherlands. He worked as a postdoctoral scholar jointly at the CoR-Lab, Bielefeld University, Germany and at the Honda Research Institute Europe, Germany. He graduated in 2012 with a PhD Degree in Engineering from TU Darmstadt and the MPI for Intelligent Systems. For his research he received the annually awarded Georges Giralt PhD Award for the best PhD thesis in robotics in Europe, the 2018 IEEE RAS Early Academic Career Award, and has received an ERC Starting grant. His research interests include motor skill learning, (deep) reinforcement learning, imitation learning, interactive learning, and machine learning for control.

Acknowledgements

The tutorial is supported by the EU H2020 projects VERtical Innovation in the Domain of Robotics Enabled by Artificial intelligence Methods and OpenDR .

The tutorial is also supported by the IEEE RAS Technical Committee on Robot Learning .