GitHub Repository

https://github.com/araffin/rlss23-dqn-tutorial

Outline

1. RL 101 refresher
2. Tabular Q-Learning
3. RL as a regression problem (Fitted Q Iteration)
4. From FQI to Deep Q-Network (DQN)

RL 101 (1/2)

RL 101 (2/2)

• Agent: the "boat", our main character
• State: Where are we? (position, speed, ...)
• Action: What can we do? (steer left, right, ...)
• Reward: How good are we doing?
• Policy: The "captain", defines the agent's behavior

Value Functions

How good is it to be in this state?

Source: Freek Stulp - Master AIC

Action-Value Function: Q-Value

What if we have no model?

Solution: $Q_\pi(s, a)$ instead of $V_\pi(s)$

\[\begin{aligned} \pi(s) = \argmax_{a \in A} Q_\pi(s, a) \end{aligned} \]

Tabular Q-Learning: Discrete States

Tabular Q-Learning: Update rule

Questions?

Tabular Q-Learning: Limitations

• Discrete states
• No generalization (lookup table)
• Discrete actions

How to go beyond tabular Q-Learning?

Q-Value Estimator

Q-Learning Regression (1/2)

\[\begin{aligned} \textcolor{#1864ab}{Q_{\textcolor{black}{\theta}}(s_t, a_t)} = \textcolor{#a61e4d}{r_t + \gamma \cdot \max_{a' \in A}(Q_{\textcolor{black}{\theta}}(s_{t+1}, a'))} \end{aligned} \]

\[\begin{aligned} \textcolor{#1864ab}{f_{\textcolor{black}{\theta}}(x)} = \textcolor{#a61e4d}{y} \end{aligned} \]

$\theta$: parameters of the estimator

One small detail...

$\textcolor{#1864ab}{Q_\theta(s_t, a_t)}$ depends on $\textcolor{#a61e4d}{Q_\theta(s_{t+1}, a')}$...

What can we do about it?

Iterate!     Use $Q^{\textcolor{green}{n}}_\theta(s_t, a_t)$

CartPole Env

https://gymnasium.farama.org/environments/classic_control/cart_pole/

Gym/Gymnasium API

							 import gymnasium as gym

							 # Create the environment
							 env = gym.make("CartPole-v1", render_mode="human")
							 # Reset env and get first observation
							 obs, _ = env.reset()

							 # Step in the env with random actions and display the env
							 for _ in range(100):
							     env.render() # Display the env

							     action = env.action_space.sample()
							     # Retrieve new observation, reward,
							     # termination signal, truncation signal
							     # and additional infos
							     next_obs, reward, terminated, truncated, info = env.step(action)
							     # Update current observation
							     obs = next_obs
							     # End of an episode
							     if terminated or truncated:
							         obs, _ = env.reset()

 						 

FQI in practice (1st notebook)

https://github.com/araffin/rlss23-dqn-tutorial

FQI Limitations

• Offline RL
• Loop over all possible actions $A$ to get next best action $\textcolor{#a61e4d}{a'}$: \[\begin{aligned} \max_{\textcolor{#a61e4d}{a' \in A}} Q_\theta(s_{t+1}, \textcolor{#a61e4d}{a'}) \end{aligned} \]
• Instability (target depends on $Q^{n-1}_\theta(s_{t+1}, a')$)

From FQI to DQN

• Offline RL → Online RL
• Loop over actions → One forward pass to get all $Q_\theta(s, a)$
• Instability → Target Network $Q_{\textcolor{green}{\theta'}}(s, a)$

Questions?

Deep Q-Network (DQN)

Replay Buffer

Replay Buffer Sampling

Q-network

The training loop

Collecting Experience

								# Retrieve q values for the current observation
								q_values = q_model(current_obs)

								# Follow greedy-policy:
								# take the action with the highest q_value
								action = np.argmax(q_values)

								# Do one step in the env
								next_obs, reward, terminated, _, _ = env.step(action)

								# Store transition in the replay buffer
								replay_buffer.store(obs, action, reward, terminated, next_obs)
							

Exploration / Exploitation

Epsilon-greedy Exploration

							 # Flip a biased coin
							 take_random_action = np.random.rand() < exploration_rate

							 if take_random_action:
							     # Random action
							     action = action_space.sample()
							 else:
							     # Greedy action
							     action = np.argmax(q_values)
						 

Exploration Schedule

Target Q-Network

DQN Overview

Questions?

PyTorch API

Backup Slides

Scikit-Learn API

								import numpy as np
								from sklearn.linear_model import LinearRegression
								# Generate some data (noisy linear function)
								x = np.linspace(0, 5, num=50).reshape(50, 1)
								y = 2 * x + 10 + 0.1 * np.random.rand()
								# Fit a linear model using least squares
								model = LinearRegression().fit(x, y)
								y_predict = model.predict(x)
								# Retrieve the optimized parameters
								slope, bias = model.coef_, model.intercept_