CS 285, Lecture 4, Intro to RL

CS 285 Deep Reinforcement Learning, Sergey Levine, Lecture 4 notes.

Cornell insight

Imitation Learning different from supervised learning Motivation: implicitly program robots

Problem: feedback compounds errors feedback drives covariate shift btw train and test distr training distr: states visited by human test distr: states visited by learner

Solution:
interactively query expert on states learner visits, s.t. training distr is similar to test distr
seminal algorithm:
DAGGER (Dataset Aggregation)
4 steps:

1. collect human demonstration, train initial policy $\pi$
2. rollout $\pi(\cdot)$ to collect states
3. ask human for correct action
4. aggregate data & train $\pi(\cdot)$, go back to step 2

---

CS285

• imitation learning via behavioral cloning is not guaranteed to work
  • different from supervised learning
  • reason: iid assumption does not hold
• theory explains why (the quadratic term in # error expectation)
• solutions:
  • be smart about how to collect (and augment) data
  • powerful models that make few mistakes
  • use multi-task learning
  • change the algorithm (DAgger)

---

Following is notes for CS285, lecture 4.

Markov Decision Process

state: $\mathbf{s}_t$, observation: $\mathbf{o}_t$, action: $\mathbf{a}_t$
policy: $\pi_\theta (\mathbf{a}_t | \mathbf{o}_t)$, policy (fully observed): $\pi(\mathbf{a}_t | \mathbf{s}_t)$
observations are partially observed, while states are fully observed and satisfy the Markov property ($\mathbf{s}_{t+1}$ is independent of $\mathbf{s}_{t-1}$ given the current state $\mathbf{s}_t$ and action $\mathbf{a}_t$)
reward function: $r (\mathbf{s}, \mathbf{a})$, which tells us which state-action pairs are better
Markov Decision Process (MDP): $\mathbf{s}$, $\mathbf{a}$, $r (\mathbf{s}, \mathbf{a})$, $p(\mathbf{s}^\prime | \mathbf{s}, \mathbf{a})$

$\mathcal{M} = \{\mathcal{S}, \mathcal{A}, \mathcal{T}, r\}$

state space (discrete or continuous): $\mathcal{S}$
action space (discrete or continuous): $\mathcal{A}$
transition operator: $\mathcal{T}$, a tensor
let $\mu_{t,j} = p(s_t = j)$, $\xi_{t,k} = p(a_t = k)$, $\mathcal{T}_{i,j,k} = p(s_{t+1} = i | s_t = j, a_t = k)$
then $\mu_{t+1, i} = \sum_{j, k} \mathcal{T}_{i,j,k} \mu_{t,j} \xi_{t,k}$
reward function: $r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

Partially Observed Markov Decision Process

make decisions based on partially observed observations, without access to true states

$\mathcal{M} = \{\mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \mathcal{\varepsilon}, r\}$

additionally, observations (discrete or continuous): $\mathcal{O}$
emission probability: $\varepsilon = p(o_t | s_t)$

Goal of RL

let $\tau = \mathbf{s}_1, \mathbf{a}_1, \ldots, \mathbf{s}_T, \mathbf{a}_T$ , then

$p_\theta (\tau) = p(\mathbf{s}_1) \prod_{t=1}^T \pi_\theta (\mathbf{a}_t | \mathbf{s}_t) p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$

$\theta^\star = \arg \max_\theta E_{\tau \sim p_\theta (\tau)} \left[ \sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right]$

if we define augmented state $(\mathbf{s}, \mathbf{a})$, then

$p((\mathbf{s}_{t+1}, \mathbf{a}_{t+1}) | (\mathbf{s}_t, \mathbf{a}_t)) = p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t) \pi(\mathbf{a}_{t+1} | \mathbf{s}_{t+1})$

Finite horizon case: state-action marginal

state-action marginal: $p_\theta (\mathbf{s}_t, \mathbf{a}_t)$

$\begin{aligned} \theta^\star &= \arg \max_\theta E_{\tau \sim p_\theta (\tau)} \left[ \sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right] \\ &= \arg \max_\theta \sum_{t = 1}^T E_{(\mathbf{s}_t, \mathbf{a}_t) \sim p_\theta (\mathbf{s}_t, \mathbf{a}_t)} [r(\mathbf{s}_t, \mathbf{a}_t)] \end{aligned}$

Infinite horizon case: stationary distribution

Let $\mathcal{T}$ be the state-action transition operator, then:

$\begin{aligned} \begin{bmatrix} \mathbf{s}_{t+1} \\ \mathbf{a}_{t+1} \end{bmatrix} &= \mathcal{T} \begin{bmatrix} \mathbf{s}_{t} \\ \mathbf{a}_{t} \end{bmatrix} \\\\ \begin{bmatrix} \mathbf{s}_{t+k} \\ \mathbf{a}_{t+k} \end{bmatrix} &= \mathcal{T}^k \begin{bmatrix} \mathbf{s}_{t} \\ \mathbf{a}_{t} \end{bmatrix} \end{aligned}$

when $T \rightarrow \infty$, we generally have 2 ways:

1. because $T$ is a constant, this way does not change the max. but when $T \rightarrow \infty$, the return is well-defined (discuss now)
2. discount factor (will discuss later)

under ergodicity and aperiodicity, $p(\mathbf{s}_t, \mathbf{a}_t)$ stationary distribution (same before and after the transition) exists:

$\mu = \mathcal{T} \mu$

$\mu$ is eigenvector of $\mathcal{T}$ with eigenvalue $1$, and always exists under ergodicity and aperiodicity assumptions:

$(T - \mathbf{I})\mu = 0$

ergodicity: every state can be reached from every other state with non-zero probability (if starting in one part of MDP may never reach another part, then starting state matters, and stationary distribution does not exist)
aperiodicity: Markov chain is not periodic

when $T \rightarrow \infty$, have initially finite number of terms not in stationary distribution, and infinite number of terms under stationary distribution $\mu = p_\theta (\mathbf{s}, \mathbf{a})$. The goal of RL in infinite horizon as $T \rightarrow \infty$ becomes:

$\theta^\star = \arg \max_\theta \frac{1}{T} \sum_{t = 1}^T E_{(\mathbf{s}_t, \mathbf{a}_t) \sim p_\theta (\mathbf{s}_t, \mathbf{a}_t)} [r(\mathbf{s}_t, \mathbf{a}_t)] \rightarrow E_{(\mathbf{s}, \mathbf{a}) \sim p_\theta (\mathbf{s}, \mathbf{a})} [r(\mathbf{s}, \mathbf{a})]$

the average reward is expected reward under stationary distribution.

For example, $r(x)$ is $1$ if not fall, and -1 if fall off the cliff, and not smooth. Policy $\pi_\theta (\mathbf{a} = \text{fall}) = \theta$. The expectation is:

$E_{\pi_\theta} [r(x)] = 1- \theta - \theta$

even if the function (reward) is discontinuous, the expectation is usually continuous.

In RL, we almost always care about expectations. Even though the reward itself may be sparse, discontinuous, and non-differentiable, the policy is typically differentiable with respect to its parameters, and the expected reward is often differentiable enough to optimize with gradient-based methods.

Anatomy of a RL algorithm

the following 3 steps in a cycle:

• generate samples run the policy in the env, get trajectories can be prohibitively expensive in real world, but trivially cheap in simulator
• fit a model / estimate the return measure how good the trajectories are can just sum up the rewards into return (fast)

$J(\theta) = E_\theta \left[ \sum_t r_t \right] \approx \frac{1}{N} \sum_{i=1}^N \sum_t{r_t^i}$

in model-based RL, learn $f_\theta$ such that $\mathbf{s}_{t+1} \approx f_\theta (\mathbf{s}_t, \mathbf{a}_t)$ (can be slow)

• improve the policy

modify policy to make the better trajectories have a higher probability (1 gradient step, cheap)

$\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

in model-based RL, backprop through $f_\theta$ and $r$, train $\pi_\theta (\mathbf{s}_t) = \mathbf{a}_t$ (backprop, can be expensive)

Value Functions

The expectation can be decoupled:

$E_{\tau \sim p_\theta(\tau)} \left[ \sum_{t=1}^{T} r(s_t, a_t) \right] = E_{s_1 \sim p(s_1)} \left[ E_{a_1 \sim \pi(a_1 | s_1)} \left[ r(s_1, a_1) + E_{s_2 \sim p(s_2 | s_1, a_1)} \left[ E_{a_2 \sim \pi(a_2 | s_2)} \left[ r(s_2, a_2) + \ldots | s_2 \right] | s_1, a_1 \right] \right] \right]$

define:

$Q(s_1, a_1) = r(s_1, a_1) + E_{s_2 \sim p(s_2 | s_1, a_1)} \left[ E_{a_2 \sim \pi(a_2 | s_2)} \left[ r(s_2, a_2) + \ldots | s_2 \right] | s_1, a_1 \right]$

then the RL objective is:

$E_{\tau \sim p_\theta(\tau)} \left[ \sum_{t=1}^{T} r(s_t, a_t) \right] = E_{s_1 \sim p(s_1)} \left[ E_{a_1 \sim \pi(a_1 | s_1)} \left[ Q(s_1, a_1) | s_1 \right] \right]$

it's easy to modify $\pi_\theta (\mathbf{a}_1 | \mathbf{s}_1)$ if $Q(\mathbf{s}_1, \mathbf{a}_1)$ is known. e.g., $\pi(\mathbf{a}_1 | \mathbf{s}_1) = 1$ if $\mathbf{a}_1 = \arg \max_{\mathbf{a}_1} Q(\mathbf{s}_1, \mathbf{a}_1)$ (always choose the action giving max Q)

Q-function definition

total reward from taking action $a_{t}$ in state $s_{t}$

$Q^\pi (\mathbf{s}_t, \mathbf{a}_t) = \sum_{t^\prime = t}^T E_{\pi_{\theta}}[r(\mathbf{s}_{t^\prime}, \mathbf{a}_{t^\prime}) | \mathbf{s}_t, \mathbf{a}_t]$

expected sum of future rewards conditioned on starting at $(\mathbf{s}_t, \mathbf{a}_t)$

Value function definition

total reward from $\mathbf{s}_t$, conditioned only on state, rather than state and action

$V^\pi (\mathbf{s}_t) = \sum_{t^\prime = t}^T E_{\pi_{\theta}}[r(\mathbf{s}_{t^\prime}, \mathbf{a}_{t^\prime}) | \mathbf{s}_t]$

also can be written as the expected value over actions of the Q function:

$V^\pi (\mathbf{s}_t) = E_{\mathbf{a}_t \sim \pi(\mathbf{a}_t | \mathbf{s}_t)} [Q^\pi (\mathbf{s}_t, \mathbf{a}_t)]$

the expectation of value function at state $\mathbf{s}_1$ is the RL objective:

$J(\theta) = E_{\mathbf{s}_1 \sim p(\mathbf{s}_1)} [V^\pi (\mathbf{s}_1)]$

Using Q-function and value function

2 important ideas:

idea 1: if we have policy $\pi$, and we know $Q^\pi (\mathbf{s}, \mathbf{a})$, then can improve $\pi$:
set $\pi^\prime(\mathbf{a} | \mathbf{s}) = 1$ if $\mathbf{a} = \arg \max_\mathbf{a} Q^\pi(\mathbf{s}, \mathbf{a})$
this policy is at least as good as $\pi$
used in policy iteration and Q learning

idea 2: compute gradient to increase probability of good actions $\mathbf{a}$:
if $Q^\pi (\mathbf{s}, \mathbf{a}) > V^\pi (\mathbf{s})$, then $\mathbf{a}$ is better than average (recall: $V^\pi (\mathbf{s}) = E_{\mathbf{a} \sim \pi(\mathbf{a} | \mathbf{s})} [Q^\pi (\mathbf{s}, \mathbf{a})]$)
can modify $\pi (\mathbf{a} | \mathbf{s})$ to increase probability of $\mathbf{a}$

will reoccur in model-free RL algos

Types of RL algorithms

• policy gradients: directly differentiate the objective, then gradient descent
• value-based: estimate $V$ or $Q$ of the optimal policy (no explicit policy). value function represented by a function approximator like NN
• actor-critic: estimate $V$ or $Q$ of the current policy, improve policy with policy gradient
• model-based RL: estimate the transition model, and then:
  • use it for planning (no explicit policy)
  • use it to improve a policy

TODO: diff btw value-based & actor-critic?

Model-based RL algorithms

in 2nd step: fit a model / estimate the return
learn a model for $p(\mathbf{s}_{t+1} | \mathbf{s}_{t}, \mathbf{a}_{t})$
if a deterministic model, just predict $\mathbf{s}_{t+1}$

in 3rd step: improve the policy
a few options:

1. use the model to plan (no policy)

• in continuous action spaces: trajectory optimization / optimal control, essentially backprop to optimize over actions
• in discrete action spaces: discrete planning, e.g., Monte Carlo tree search

2. backprop gradients into the policy

• requires some tricks to work

3. use the model to learn a value function, then improve policy

• V or Q learned using dynamic programming
• use a model to generate simulated experience for model-free learner

Value function based algorithms

in 2nd step: fit a model / estimate the return
fit $V(\mathbf{s})$ or $Q(\mathbf{s}, \mathbf{a})$

in 3rd step: improve the policy
set $\pi(\mathbf{s}) = \arg \max_\mathbf{a} Q(\mathbf{s}, \mathbf{a})$

Direct policy gradients

in 2nd step: fit a model / estimate the return
evaluate returns by adding rewards obtained during the rollout (sample policy 1 step at a time):
$R_\tau = \sum_t r(\mathbf{s}_t, \mathbf{a}_t)$

in 3rd step: improve the policy
using gradient ascent
$\theta \leftarrow \theta + \alpha \nabla_\theta E [\sum_t r(\mathbf{s}_t, \mathbf{a}_t)]$

Actor-critic

in 2nd step: fit a model / estimate the return
like value-based methods
fit $V(\mathbf{s})$ or $Q(\mathbf{s}, \mathbf{a})$

in 3rd step: improve the policy
like policy gradient methods, utilize $V$ or $Q$ to obtain a more accurate gradient
$\theta \leftarrow \theta + \alpha \nabla_\theta E [Q(\mathbf{s}_t, \mathbf{a}_t)]$

Why so many RL algorithms

• different tradeoffs
  • sample efficiency
  • stability & ease of use (hyperparams can be difficult to select)
• different assumptions
  • stochastic vs deterministic
  • continuous vs discrete state & action
  • episodic vs infinite horizon
• different things are easy/hard in different settings
  • easier to represent the policy
  • easier to represent the model

Comparison: sample efficiency

sample efficiency: how many samples needed to obtain a good policy
off-policy: able to improve policy without generating new samples from that policy
on-policy: each time the policy is changed (even a little bit), new samples must be collected

Why would we use a less efficient algorithm? Because wall clock time is not the same as efficiency!

Comparison: stability and ease of use

• Does it converge?
• And if it converges, to what?
• And does it converge every time?

supervised learning: almost always gradient descent
RL: often not gradient descent

• Q-learning: fixed point iteration, guarantee convergence under simplified tabular discrete state assumptions (often not held in practice). convergence of many RL algos is an open problem
• model-based RL: getting a better model does not mean a better expected reward
• policy gradient: is gradient descent, but often the least efficient
• value function fitting
  • at best, minimize error of fit (Bellman error). eval value function predicting error accurately, but not the same as expected reward
  • at worst, does not optimize anything. many popular deep RL value fitting algorithms are not guaranteed to converge to anything in the nonlinear case
• model-based RL
  • model minimizes error of fit (will converge), but no guarantee that better model = better policy
• policy gradient
  • actually performs gradient descent (ascent) on the true objective

Comparison: common assumptions

1. full observability

• generally assumed by value function fitting methods
• can be mitigated by adding recurrence

2. episodic learning: make a trial, reset, make another trial

• often assumed by pure policy gradient methods
• assumed by some model-based RL methods
• value-based methods work better under this assumption

3. continuity or smoothness

• assumed by some continuous value function learning methods
• often assumed by some model-based RL methods

Examples of specific algorithms

• Value function fitting methods
  • Q-learning, DQN
  • Temporal difference learning
  • Fitted value iteration
• Policy gradient methods
  • REINFORCE
  • Natural policy gradient
  • Trust region policy optimization
• Actor-critic algorithms
  • Asynchronous advantage actor-critic (A3C)
  • Soft actor-critic (SAC)
• Model-based RL algorithms
  • Dyna
  • Guided policy search