RL 101

Motivation

Outline

1. Getting started with Gymnasium
2. How to define a custom RL task?
3. Getting started with SB3

What is Gymnasium? (1/2)

                                import gymnasium as gym

                                # Create the environment
                                env = gym.make("CartPole-v1", render_mode="human")
                                # Reset env and get first observation
                                obs, _ = env.reset()

                                # Step in the env with random actions
                                for _ in range(100):

                                    action = env.action_space.sample()
                                    # Retrieve new observation, reward, terminations signals
                                    # and additional infos
                                    obs, reward, terminated, truncated, info = env.step(action)

                                    # End of an episode
                                    if terminated or truncated:
                                        obs, _ = env.reset()

                            

Live Demo

1. Getting started with Gymnasium
2. How to define a custom RL task?
3. Getting started with SB3

RL in Practice: Tips and Tricks

Full video: RL in practice YT playlist

Today: only about how to define custom task

Defining a custom task

• observation space
• action space
• reward function
• termination conditions

RL 102

Choosing the observation space

• start simple
• enough information to solve the task
• do not break Markov assumption
• normalize!

CartPole Observation Space

                            high = np.array(
                                [
                                    self.x_threshold * 2,
                                   np.inf,
                                    self.theta_threshold_radians * 2,
                                   np.inf,
                                ],
                                dtype=np.float32,
                            )

                            self.observation_space = gym.spaces.Box(low=-high, high=high, dtype=np.float32)
                        

Choosing the Action space

• start simple
• discrete / continuous
• complexity vs final performance

CartPole Action Space

                            """
                            Actions:
                            Type: Discrete(2)

                            Num   Action
                            0     Push cart to the left
                            1     Push cart to the right
                            """
                            self.action_space = gym.spaces.Discrete(2)
                        

Choosing the reward function

• start simple
• reward shaping
• primary / secondary reward
• normalize!

CartPole Reward

                            if not terminated:
                                reward = 1.0
                        

Termination conditions?

• early stopping
• special treatment needed for timeouts
• should not change the task (reward hacking)

CartPole Termination

                            terminated = bool(
                                x < -self.x_threshold
                                or x > self.x_threshold
                                or theta < -self.theta_threshold_radians
                                or theta > self.theta_threshold_radians
                            )

                            # in the registration:
                            register(
                                id="CartPole-v1",
                                entry_point="gymnasium.envs.classic_control.cartpole:CartPoleEnv",
                                max_episode_steps=500,  # truncation
                                reward_threshold=475.0,
                            )
                        

Questions?

1. Getting started with Gymnasium
2. How to define a custom RL task?
3. Getting started with SB3

Getting Started with SB3

                            import gymnasium as gym
                            from stable_baselines3 import SAC

                            # Train an agent using Soft Actor-Critic on Pendulum-v1
                            env = gym.make("Pendulum-v1")
                            model = SAC("MlpPolicy", env, verbose=1)

                            # Train the model
                            model.learn(total_timesteps=20_000)

                            # Save the model
                            model.save("sac_pendulum")

                            # Load the trained model
                            model = SAC.load("sac_pendulum")
                            # Start a new episode
                            obs, _ = env.reset()
                            # What action to take in state `obs`?
                            action, _ = model.predict(obs, deterministic=True)
                        

Reproducible Reliable RL: SB3 + RL Zoo

Recap

Questions?

Colab Notebook

https://github.com/araffin/tools-for-robotic-rl-icra2022/

Backup Slides

Plotting

                            python -m rl_zoo3.cli all_plots -a sac -e HalfCheetah Ant -f logs/ -o sac_results
                            python -m rl_zoo3.cli plot_from_file -i sac_results.pkl -latex -l SAC --rliable