GitHub Repository

https://github.com/araffin/rlss23-dqn-tutorial

Blog post
https://araffin.github.io/post/rl102/

Outline

1. RL 101 refresher
2. Tabular Q-Learning
3. RL as a regression problem (Fitted Q Iteration)
4. From FQI to Deep Q-Network (DQN)

RL 101

Value Functions

How good is it to be in this state?

Source: Freek Stulp - Master AIC

Action-Value Function: Q-Value

What if we have no model?

Solution: $Q_\pi(s, a)$ instead of $V_\pi(s)$

\[\begin{aligned} \pi(s) = \argmax_{a \in A} Q_\pi(s, a) \end{aligned} \]

Tabular Q-Learning: Discrete States

Tabular Q-Learning: Update rule

Bellman equation for optimal value function:

\[\begin{aligned} \textcolor{#1864ab}{Q^*(s_t, a_t)} = \mathop{\mathbb{E}}[\textcolor{#a61e4d}{r(s_t, a_t) + \gamma \max_{a'} Q^*(s_{t+1},a')}]. \end{aligned} \]

Q-learning update rule

Questions?

Tabular Q-Learning: Limitations

• Discrete states
• No generalization (lookup table)
• Discrete actions

How to go beyond tabular Q-Learning?

Q-Learning Regression

One small detail...

$\textcolor{#1864ab}{Q_\theta(s_t, a_t)}$ depends on $\textcolor{#a61e4d}{Q_\theta(s_{t+1}, a')}$...

What can we do about it?

Iterate!     Use $Q^{\textcolor{green}{n}}_\theta(s_t, a_t)$

CartPole Env

https://gymnasium.farama.org/environments/classic_control/cart_pole/

Gym/Gymnasium API

                             import gymnasium as gym

                             # Create the environment
                             env = gym.make("CartPole-v1", render_mode="human")
                             # Reset env and get first observation
                             obs, _ = env.reset()

                             # Step in the env with random actions and display the env
                             for _ in range(100):
                                 env.render() # Display the env

                                 action = env.action_space.sample()
                                 # Retrieve new observation, reward,
                                 # termination signal, truncation signal
                                 # and additional infos
                                 next_obs, reward, terminated, truncated, info = env.step(action)
                                 # Update current observation
                                 obs = next_obs
                                 # End of an episode
                                 if terminated or truncated:
                                     obs, _ = env.reset()

                         

FQI in practice (1st notebook)

https://github.com/araffin/rlss23-dqn-tutorial

FQI Limitations

• Offline RL
• Loop over all possible actions $A$ to get next best action $\textcolor{#a61e4d}{a'}$: \[\begin{aligned} \max_{\textcolor{#a61e4d}{a' \in A}} Q_\theta(s_{t+1}, \textcolor{#a61e4d}{a'}) \end{aligned} \]
• Instability (target depends on $Q^{n-1}_\theta(s_{t+1}, a')$)

From FQI to DQN

• Offline RL → Online RL
• Loop over actions → One forward pass to get all $Q_\theta(s, a)$
• Instability → Target Network $Q_{\textcolor{green}{\theta'}}(s, a)$

Questions?

Deep Q-Network (DQN)

Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." (2013).

Replay Buffer

Replay Buffer Sampling

Q-network

The training loop

Collecting Experience

                                # Retrieve q values for the current observation
                                q_values = q_model(current_obs)

                                # Follow greedy-policy:
                                # take the action with the highest q_value
                                action = np.argmax(q_values)

                                # Do one step in the env
                                next_obs, reward, terminated, _, _ = env.step(action)

                                # Store transition in the replay buffer
                                replay_buffer.store(obs, action, reward, terminated, next_obs)
                            

Exploration / Exploitation

Epsilon-greedy Exploration

                             # Flip a biased coin
                             take_random_action = np.random.rand() < exploration_rate

                             if take_random_action:
                                 # Random action
                                 action = action_space.sample()
                             else:
                                 # Greedy action
                                 action = np.argmax(q_values)
                         

Exploration Schedule

Target Q-Network

DQN Overview

Questions?

PyTorch API

Backup Slides

Scikit-Learn API

                                import numpy as np
                                from sklearn.linear_model import LinearRegression
                                # Generate some data (noisy linear function)
                                x = np.linspace(0, 5, num=50).reshape(50, 1)
                                y = 2 * x + 10 + 0.1 * np.random.rand()
                                # Fit a linear model using least squares
                                model = LinearRegression().fit(x, y)
                                y_predict = model.predict(x)
                                # Retrieve the optimized parameters
                                slope, bias = model.coef_, model.intercept_