Why learn directly on real robots?

Simulation is all you need?

Why learn directly on real robots?

• because you can! (software/hardware)
• simulation is safer, faster
• simulation to reality (sim2real): accurate model and randomization needed
• challenges: robot safety, sample efficiency

Knowledge guided RL

The elastic quadruped bert

• Mass: 3.1kg
• Spring stiffness: 2.75 Nm/rad
• Max motor torque: 4 Nm

RL from scratch (3 years later)

Learning to Exploit Elastic Actuators for Quadruped Locomotion

Raffin et al. "Learning to Exploit Elastic Actuators for Quadruped Locomotion" In preparation RA-L, 2023.

Central Pattern Generator (CPG)

Desired foot position

$$\begin{aligned} x_{des,i} &= \textcolor{#5f3dc4}{\Delta x_\text{len}} \cdot r_i \cos(\varphi_i)\\ z_{des,i} &= \Delta z \cdot \sin(\varphi_i) \\ \Delta z &= \begin{cases} \textcolor{#5c940d}{\Delta z_\text{clear}} &\text{if $\sin(\varphi_i) > 0$ (\textcolor{#0b7285}{swing})}\\ \textcolor{#d9480f}{\Delta z_\text{pen}} &\text{otherwise (\textcolor{#862e9c}{stance}).} \end{cases} \end{aligned}$$

Black Box Optimization (BBO)

Variables

$$\begin{aligned} \alpha = \{\textcolor{#5c940d}{\Delta z_\text{clear}}, \textcolor{#d9480f}{\Delta z_\text{pen}}, \textcolor{#5f3dc4}{\Delta x_\text{len}}, \textcolor{#0b7285}{\omega_\text{swing}}, \textcolor{#862e9c}{\omega_\text{stance}} \}. \end{aligned}$$

BBO

$$\begin{aligned} \alpha^* = \argmax {f(\alpha)}. \end{aligned}$$

Fitness function

$$\begin{aligned} f: \text{mean forward speed}. \end{aligned}$$

Result

Closing the loop with RL

desired position = open-loop foot position + closed-loop control with RL

$$\begin{aligned} x_{des,i} &= \textcolor{#a61e4d}{\Delta x_\text{len} \cdot r_i \cos(\varphi_i)} + \textcolor{#1864ab}{\pi_{x,i}(s_t)} \\ z_{des,i} &= \textcolor{#a61e4d}{\Delta z \cdot r_i \sin(\varphi_i)} + \textcolor{#1864ab}{\pi_{z,i}(s_t)} \end{aligned}$$

Video

Fast trot task

Trot comparison

Learning to Exploit Elastic Actuators

Conclusion

• learn directly on the real robot
• exploit elastic actuators
• model-free, works on multiple robots
• RL from scratch is not enough

Questions?

Backup slides