CS 285, Lecture 6, Actor-Critic Algorithms

CS 285 Deep Reinforcement Learning, Sergey Levine, Lecture 6, Actor-Critic Algorithms notes.

Recap: Policy Gradients

REINFORCE algorithm:

1. Sample $\{\tau^i\}$ from $\pi_{\theta}(\mathbf{a}_t | \mathbf{s}_t)$ (run the policy)
2. $\nabla_{\theta}J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N} \sum_{t=1}^T \nabla_{\theta}\log\pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{Q}_{i,t}^\pi$, where the "reward to go" is $\hat{Q}^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) = \sum_{t^\prime=t}^T r(\mathbf{s}_{i,t^\prime}, \mathbf{a}_{i,t^\prime})$
3. $\theta \leftarrow \theta + \alpha \nabla_{\theta} J(\theta)$

Improving the Policy Gradient

The policy gradient estimate relies on a sample estimate of the return:

$\nabla_{\theta}J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^T\nabla_{\theta}\log\pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{Q}_{i, t}$

The "reward to go" $\hat{Q}_{i,t} \approx \sum_{t^\prime=t}^T r(\mathbf{s}_{i,t^\prime}, \mathbf{a}_{i,t^\prime})$ is an estimate of expected return if we take action $\mathbf{a}_{i,t}$ in state $\mathbf{s}_{i,t}$.

A single sample estimate has high var, so generate many samples and calc the exp of reward to go with a lower var.

Can we get a better estimate? The ideal is the true expected reward-to-go, $Q(\mathbf{s}_t, \mathbf{a}_t)$:

$Q(\mathbf{s}_t, \mathbf{a}_t) = \sum_{t^\prime=t}^T E_{\pi_{\theta}}[r(\mathbf{s}_{t^\prime},\mathbf{a}_{t^\prime}) | \mathbf{s}_t, \mathbf{a}_t]$

If we can replace $\hat{Q}$ with true Q function, the var of policy gradient would be much lower:

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^T \nabla_{\theta}\log\pi_{\theta}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})Q(\mathbf{s}_{i,t},\mathbf{a}_{i,t})$

Subtract the Baseline

Previously, we subtract the average reward as baseline. Now, if we have true expected reward-to-go:

$Q(\mathbf{s}_t,\mathbf{a}_t) = \sum_{t^\prime=t}^T E_{\pi_{\theta}}[r(\mathbf{s}_{t^\prime}, \mathbf{a}_{t^\prime}) | \mathbf{s}_t, \mathbf{a}_t]$

Then rewrite policy gradient w/ true $Q$ to subtract baseline:

$\begin{aligned} \nabla_\theta J(\theta) &\approx \frac{1}{N} \sum_{i=1}^N \nabla_\theta \log \pi_\theta(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) (Q(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) - b_t) \\ b_t &= \frac{1}{N} \sum_i Q(\mathbf{s}_{i,t},\mathbf{a}_{i,t}) \end{aligned}$

Baseline depending on actions would lead to bias, so instead depend on state, using average reward over all actions for the state $\mathbf{s}_{i,t}$:

$V(\mathbf{s}_t) = E_{\mathbf{a}_t \sim \pi_{\theta}(\mathbf{a}_t | \mathbf{s}_t)}[Q(\mathbf{s}_t, \mathbf{a}_t)]$

Choose value function as the baseline:

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^T \nabla_{\theta}\log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) A(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})$

where the advantage function $A(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})$ is defined as:

$A(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) = Q(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) - V(\mathbf{s}_{i,t})$

If take the actions better than avg in the state, then increase the prob. If take the actions worse than avg in the state, then decrease the prob.

State & state-action value function (Q function)

Total expected reward starting from $\mathbf{s}_t$, taking action $\mathbf{a}_t$, and following policy $\pi$:

$Q^\pi(\mathbf{s}_t, \mathbf{a}_t) = \sum_{t^\prime=t}^T E_{\pi_{\theta}}[r(\mathbf{s}_{t^\prime}, \mathbf{a}_{t^\prime}) | \mathbf{s}_t, \mathbf{a}_t]$

We add the superscript $\pi$ to emphasize $Q$ function depends on $\pi$. Every policy has a different $Q$ function.

The value function is the expected value of $Q$ values over all actions in state $\mathbf{s}_t$ under policy $\pi$. Or, the total reward starting from $\mathbf{s}_t$ and following policy $\pi$:

$V^\pi(\mathbf{s}_t) = E_{\mathbf{a}_t \sim \pi_{\theta}(\mathbf{a}_t | \mathbf{s}_t)}[Q^\pi(\mathbf{s}_t, \mathbf{a}_t)]$

The advantage function represents how much better the action $\mathbf{a}_t$ is compared to the avg performance of policy $\pi$ in state $\mathbf{s}_t$:

$A^\pi(\mathbf{s}_t, \mathbf{a}_t) = Q^\pi(\mathbf{s}_t, \mathbf{a}_t) - V^\pi(\mathbf{s}_t)$

Conventionally, policy gradient uses a Monte Carlo estimate of the advantage, is calculated by using 1 sample in the remainder of current traj, summing up rewards in the traj, and subtracting a baseline.

$\nabla_{\theta}J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^T \nabla_{\theta}\log\pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \left(\sum_{t^\prime=1}^T r(\mathbf{s}_{i,t^\prime}, \mathbf{a}_{i,t^\prime}) - b\right)$

This is an unbiased single sample estimate with high var, but we now replace it with lower var approximate advantage function.

$\nabla_{\theta}J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N}\sum_{t=1}^T \nabla_{\theta}\log\pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \textcolor{red}{A^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})}$

In practice, we don't have the correct value of advantage, but can only estimate it. The better estimate of advanatage, the lower var is. If the advantage is incorrect, policy gradients can be biased. Usually, we trade large var reduction for slight increase in bias from value function.

Now, in the 2nd step (green box), when we "fit a model to estimate return", we fit $Q^\pi, V^\pi$, or $A^\pi$.

Policy Evaluation

The policy is held fixed while we estimate the return it induces from each state. This step is called policy evaluation because it evaluates the current policy rather than improving it.

Value function fitting

Among $Q^\pi, V^\pi$, or $A^\pi$, which should we fit to?

Because $\mathbf{s}_t$ and $\mathbf{a}_t$ are not RVs, can rewrite the $Q^\pi$ as current reward (exact, not random) $+$ expected value of future rewards:

$\begin{aligned} Q^\pi(\mathbf{s}_t, \mathbf{a}_t) &= r(\mathbf{s}_t, \mathbf{a}_t) + \sum_{t'=t+1}^T E_{\pi_\theta} [r(\mathbf{s}_{t'}, \mathbf{a}_{t'}) | \mathbf{s}_t, \mathbf{a}_t] \\ &= r(\mathbf{s}_t, \mathbf{a}_t) + E_{\mathbf{s}_{t+1} \sim p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)} [V^\pi(\mathbf{s}_{t+1})] \end{aligned}$

Assuming for $\mathbf{s}_{t+1}$, the actual state in current traj is representative of the average state. For the next time step, approximate the distr of states with a single-sample estimator:

$Q^\pi(\mathbf{s}_t, \mathbf{a}_t) \approx r(\mathbf{s}_t, \mathbf{a}_t) + V^\pi(\mathbf{s}_{t+1})$

The advantage function is approximated as current reward $+$ next value $-$ current value:

$A^\pi(\mathbf{s}_t, \mathbf{a}_t) \approx r(\mathbf{s}_t, \mathbf{a}_t) + V^\pi(\mathbf{s}_{t+1}) - V^\pi(\mathbf{s}_t)$

We do the approximation b/c $V$ only depends on state, whereas $Q$ and $A$ depend on both state and action. Function approximator is easier to learn due to fewer samples.

So, fit $V^\pi(\mathbf{s})$.

Monte Carlo Policy Evaluation

Policy evaluation: calc $V^\pi(\mathbf{s})$, the value of policy at every state.

$V^\pi (\mathbf{s}_t) = \sum_{t'=t}^T E_{\pi_\theta} [r(\mathbf{s}_{t'}, \mathbf{a}_{t'}) | \mathbf{s}_t]$

The RL objective can be written as:

$J(\theta) = E_{\mathbf{s}_1 \sim p(\mathbf{s}_1)} [V^\pi (\mathbf{s}_1)]$

To perform policy evaluation, use Monte Carlo policy evaluation. Like policy gradients, run policy many times, sum rewards obtained along the generated trajectories, use it as unbiased but high-var single sample estimate of the policy's total reward.

single sample estimate:

$V^\pi (\mathbf{s}_t) \approx \sum_{t'=t}^T r(\mathbf{s}_{t'}, \mathbf{a}_{t'})$

Ideally, want to sum over all possible trajectories. Unfortunately, need to reset to $\mathbf{s}_t$ in model-free setting and run multiple trials starting from that state (can't do this unless in a simulator). But usually we only assume we can run multiple trials from the initial-state distribution.

multi-sample estimate (but can’t reset the world):

$V^\pi (\mathbf{s}_t) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t'=t}^T r(\mathbf{s}_{i, t'}, \mathbf{a}_{i, t'})$

Monte Carlo evaluation with function approximation

Function approximator can generalize: it understands that similar states visited in different trajectories take similar values:

training data: $\{(\mathbf{s}_t, y_{i,t})\}$, where

$y_{i,t} = \sum_{t'=t}^T r(\mathbf{s}_{i,t'}, \mathbf{a}_{i,t'})$

supervised regression (over all samples, minimize the MSE btw value function's prediction and the single-sample Monte-Carlo estimate of value at the state):

$\mathcal{L}(\phi) = \frac{1}{2} \sum_i \left\lVert \hat{V}_\phi^\pi(\mathbf{s}_i) - y_i \right\rVert^2$

Bootstrapped estimate

ideal target can be approximated by current reward $+$ next value:

$\begin{aligned} y_{i,t} &= \sum_{t'=t}^T E_{\pi_\theta} \left[r(\mathbf{s}_{t'}, \mathbf{a}_{t'}) | \mathbf{s}_{i,t}\right] \\ &= r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \sum_{t'=t+1}^T E_{\pi_\theta} \left[r(\mathbf{s}_{t'}, \mathbf{a}_{t'}) | \mathbf{s}_{i,t+1}\right] \\ &\approx r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + V^\pi(\mathbf{s}_{i,t+1}) \end{aligned}$

Using previously fitted value function, get the bootstrapped estimate:

$y_{i,t} \approx r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1})$

training data becomes: $\{(\mathbf{s}_{i,t}, y_{i,t})\}$

Then do supervised regression like before.

Compared with single sample estimator, bootstrapped estimate has lower var, but higher bias b/c $\hat{V}_\phi^\pi$ may be incorrect.

Estimating $V^\pi$: Monte Carlo vs. Bootstrap

Monte Carlo (supervise with roll-out's summed rewards):

$y_{i,t} = \sum_{t'=t}^T r(\mathbf{s}_{i,t'}, \mathbf{a}_{i,t'})$

Bootstrapped (supervise using reward $+$ next value estimate):

$y_{i,t} = r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1})$

$\begin{aligned} \hat{V}_{MC}^\pi(\textcolor{blue}{\mathbf{s}}) \approx 0 &\quad \hat{V}_{MC}^\pi(\textcolor{magenta}{\mathbf{s}}) \approx -1 \\ \hat{V}_{TD}^\pi(\textcolor{blue}{\mathbf{s}}) \approx 0 &\quad \hat{V}_{TD}^\pi(\textcolor{magenta}{\mathbf{s}}) \approx 0 \end{aligned}$

Actor-Critic algorithm

actor: policy
critic: value function

Aside: discount factor

Bootstrap rule in infinite horizon

if $T \to \infty$, $\hat{V}_\phi^\pi(\mathbf{s}_i)$ can get infinitely large.

better to get rewards sooner than later w/ disount factor $\gamma \in [0, 1]$ ($0.99$ works well):

$y_{i,t} \approx r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1})$

Mathematically, say there is an MDP of 4 states with transition probability $p(\mathbf{s}^\prime | \mathbf{s}, \mathbf{a})$:

By modifying the MDP to introduce the death state (never leaves once enter) w/ $0$ reward and prob $1 - \gamma$, the MDP becomes:

with critic, the policy gradient uses the bootstrapped advantage (TD error):

$\begin{aligned} \nabla_{\theta} J(\theta) &\approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{A}^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) \\ \hat{A}^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) &= r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1}) - \hat{V}_\phi^\pi(\mathbf{s}_{i,t}) \end{aligned}$

Monte Carlo policy gradients:

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \left( \sum_{t^\prime=t}^T \gamma^{t^\prime-t} r(\mathbf{s}_{i,t^\prime}, \mathbf{a}_{i,t^\prime}) \right)$

Batch actor-critic algorithm (with discount)

In episodic batch setting, we collect a batch of trajectories, each trajectory runs to the end, use the batch to eval gradient and update policy.

1. sample $\{\mathbf{s}_{i,1}, \mathbf{a}_{i,1}, \cdots, \mathbf{s}_{i,T}, \mathbf{a}_{i,T}\}$ from $\pi_\theta(\mathbf{a}|\mathbf{s})$
2. fit $\hat{V}_\phi^\pi(\mathbf{s})$ to sampled reward sums
3. evaluate $\hat{A}^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) = r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \textcolor{red}{\gamma} \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1}) - \hat{V}_\phi^\pi(\mathbf{s}_{i,t}) \quad \forall i,t$
4. evaluate $\nabla_\theta J(\theta) \approx \sum_{i,t} \nabla_\theta \log \pi_\theta(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{A}^\pi(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})$
5. update $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

need to train 2 NN's: actor $\pi_\theta(\mathbf{a} | \mathbf{s})$ and critic $\hat{V}_\phi^\pi(\mathbf{s})$.

$y_{i,t}$ can be Monte Carlo or Bootstrapped. With discount factor $\gamma$, the bootstrapped estimate is

$y_{i,t} = r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1})$

The MSE loss is same as before:

$\mathcal{L}(\phi) = \frac{1}{2} \sum_i \left\| \hat{V}_\phi^\pi(\mathbf{s}_i) - y_i \right\|^2$

Online actor-critic algorithm (with discount)

The update equations are the same as in batch actor-critic; the difference is the update schedule. Batch actor-critic freezes the policy while collecting many transitions, averages their gradients, and then updates. Online actor-critic updates the actor and critic immediately after each transition, so the policy can change during an episode.

1. take action $\mathbf{a} \sim \pi_\theta(\mathbf{a} | \mathbf{s})$, get $(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime, r)$
2. update $\hat{V}_\phi^\pi$ using target $r + \gamma \hat{V}_\phi^\pi(\mathbf{s}^\prime)$
3. evaluate $\hat{A}^\pi(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma \hat{V}_\phi^\pi(\mathbf{s}^\prime) - \hat{V}_\phi^\pi(\mathbf{s})$
4. evaluate $\nabla_\theta J(\theta) \approx \nabla_\theta \log \pi_\theta(\mathbf{a} | \mathbf{s})\, \hat{A}^\pi(\mathbf{s}, \mathbf{a})$
5. update $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

collect transitions for bootstrapped estimate of $V^\pi$.

Architecture Design

2 networks to learn value and policy:
1 maps from $\mathbf{s}$ to $\hat{V}_\phi^\pi(\mathbf{s})$
1 maps from $\mathbf{s}$ to $\pi_\theta(\mathbf{a} | \mathbf{s})$

+: simple & stable
-: no shared features btw actor & critic

or shared network design:
map from $\mathbf{s}$ to 2 heads: $\hat{V}_\phi^\pi(\mathbf{s})$ and $\pi_\theta(\mathbf{a} | \mathbf{s})$

+: can be more efficient due to shared internal representations
-: harder to train, unstable b/c of different gradients, need more hyperparameter tuning

Online actor-critic in practice: w/ multi-threading

synchronized parallel actor-critic
batch size = # of worker threads, e.g., 32
step 1: multiple simulators use diff random seeds, choose an action, generate diff transitions
step 2 & 4: update using data from all threads
step 5: update 1 policy

faster: asynchronous parallel actor-critic
each thread runs at diff speed
as soon as get 32 transitions, update policy
problem: the transitions may be collected by diff actors
this creates policy lag, a small off-policy mismatch: workers may collect transitions using stale policy parameters, while the updates are applied to the latest policy

in practice, the async method leads to perf gain that outweighs bias incurred from using slightly older actors.

Eligibility trace & n-step return

for critic training, the 1-step bootstrapped target is:

$y_{i,t} = r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_{i,t+1})$

+: lower variance than Monte Carlo target
-: higher bias if value is wrong (always is)

Monte-Carlo target:

$y_{i,t} = \sum_{t'=t}^{\infty} \gamma^{t'-t} r(\mathbf{s}_{i,t'}, \mathbf{a}_{i,t'})$

+: no bias
-: higher variance (b/c single-sample estimate)

want to combine these 2, to control bias-variance tradeoff
higher var when far in the future
lower var when closer to the present

estimating $\hat{V}_\phi^\pi(\mathbf{s}_{i,t})$ using n-step return: sum up rewards of a single-sample estimator until some time step $n$, then cut off before var gets too big, and replace the remainder with a value function estimate:

$y_{i,t} = \sum_{t'=t}^{t+n-1} \gamma^{t'-t} r(\mathbf{s}_{i,t'}, \mathbf{a}_{i,t'}) + \gamma^n \hat{V}_\phi^\pi(\mathbf{s}_{i,t+n})$

the larger $n$ is, the lower the bias (smaller 2nd term), the higher the var (larger 1st term)
sweet spot is an intermediate value btw $1$ and $\infty$
less variance than MC, lower bias than 1-step bootstrap

Generalized advantage estimation

While $n$-step return helps, we still face the question: which $n$ should we choose? Ideally, we want a method that combines all possible $n$-step estimators to get the best of both worlds (low bias and low variance).

Core Definitions

Generalized Advantage Estimation (GAE) is defined by three core mathematical components:

1. The $n$-step Advantage ($\hat{A}_n^\pi$):
   The difference between our $n$-step sampled return and our current baseline:

   $\hat{A}_n^\pi(\mathbf{s}_t, \mathbf{a}_t) = \left( \sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n \hat{V}_\phi^\pi(\mathbf{s}_{t+n}) \right) - \hat{V}_\phi^\pi(\mathbf{s}_t)$
   • Intuition: It balances "reality" (actual rewards collected for $n$ steps) with our "belief" (the Critic’s estimate of the future from step $n$ onwards).

2. Weighted Combination of Estimators:
   Instead of picking one $n$, take a weighted average of all $n$-step advantages. To ensure the weights sum to 1 while favoring shorter (lower variance) horizons, use an exponential decay: $w_n = (1 - \lambda)\lambda^{n-1}$

   $\hat{A}_{GAE}^\pi(\mathbf{s}_t, \mathbf{a}_t) = (1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1} \hat{A}_n^\pi(\mathbf{s}_t, \mathbf{a}_t)$
   • Intuition: This is a "blending" strategy. $\lambda$ acts as a secondary discount factor that decides how much we trust our multi-step reward samples versus our single-step TD estimates.

3. The TD error (temporal difference error) $\delta_t$:
   the one-step "surprise" — the difference between the bootstrapped target and the current value estimate:

   $\delta_t = r(\mathbf{s}_t, \mathbf{a}_t) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_{t+1}) - \hat{V}_\phi^\pi(\mathbf{s}_t)$

   This is exactly the 1-step advantage estimate $\hat{A}^\pi(\mathbf{s}_t, \mathbf{a}_t)$ used in the actor-critic algorithms above.

The Telescoping Derivation

The $n$-step advantage can be rewritten as a discounted sum of TD errors. When we expand the sum $\sum_{k=0}^{n-1} \gamma^k \delta_{t+k}$, the intermediate value function terms cancel out due to the telescoping property:

$\begin{aligned} \sum_{k=0}^{n-1} \gamma^k \delta_{t+k} = \quad & (\color{red}{r_t} + \color{blue}{\gamma V(\mathbf{s}_{t+1})} - \color{red}{V(\mathbf{s}_t)}) \\ + \quad & (\color{red}{\gamma r_{t+1}} + \color{green}{\gamma^2 V(\mathbf{s}_{t+2})} - \color{blue}{\gamma V(\mathbf{s}_{t+1})}) \\ + \quad & (\color{red}{\gamma^2 r_{t+2}} + \dots - \color{green}{\gamma^2 V(\mathbf{s}_{t+2})}) \\ \dots \quad & \\ + \quad & (\color{red}{\gamma^{n-1} r_{t+n-1}} + \color{red}{\gamma^n V(\mathbf{s}_{t+n})} - \dots) \end{aligned}$

• The $\color{blue}{\text{blue terms}}$ ($+\gamma V_{t+1}$ and $-\gamma V_{t+1}$) cancel.
• The $\color{green}{\text{green terms}}$ ($+\gamma^2 V_{t+2}$ and $-\gamma^2 V_{t+2}$) cancel.
• The $\color{red}{\text{red terms}}$ are left behind, which exactly match the definition of the $n$-step advantage:

$\hat{A}_n^\pi(\mathbf{s}_t, \mathbf{a}_t) = \sum_{k=0}^{n-1} \gamma^k \delta_{t+k}$

Final Form and Implementation

Substituting the telescoping sum into the weighted average definition simplifies the entire estimator into a single discounted sum of TD errors:

$\hat{A}_{GAE}^\pi(\mathbf{s}_t, \mathbf{a}_t) = \sum_{t'=t}^{\infty} (\gamma\lambda)^{t'-t} \delta_{t'}$

In practice, this is implemented using a recursive structure that "mixes" rewards and value estimates at every step:

$\hat{A}_{GAE}^{\pi}(\mathbf{s}_{t}, \mathbf{a}_{t}) = r_t - V(\mathbf{s}_t) + \gamma \Big[ (1-\lambda)V(\mathbf{s}_{t+1}) + \lambda \big( r_{t+1} + \gamma [ (1-\lambda)V(\mathbf{s}_{t+2}) + \dots ] \big) \Big]$

Or, implement this recursively (backwards from the end of an episode):

$\hat{A}_{GAE}^{\pi}(\mathbf{s}_{t}, \mathbf{a}_{t}) = \delta_t + (\gamma \lambda) \hat{A}_{GAE}^{\pi}(\mathbf{s}_{t+1}, \mathbf{a}_{t+1})$

This recursive view shows the specific choice at each step:

• $(1-\lambda)V(\mathbf{s}_{t+1})$: Trust the Critic's immediate prediction (Lower variance).
• $\lambda(r_{t+1} + \dots)$: Trust the actual reward and the subsequent trajectory (Lower bias).

$\lambda$ controls the bias-variance tradeoff: smaller $\lambda$ cuts earlier and reduces variance, while a larger $\lambda$ moves closer to an unbiased Monte Carlo estimate.

Review so far

Algorithms

• Policy gradients: observe what is good vs. bad, then do more of good stuff
• Actor-critic: learn to estimate what is good vs. bad, then do more of the good stuff
  i.e. get a better policy gradient by using neural network to estimate value function

How to estimate value (policy evaluation):

• Supervised learning directly on observed sum of future rewards
• Supervised learning on current reward + value estimate of next state
• Hybrid: Supervised learning on sum of next $n$ rewards + value of state after that

Off-policy actor critic

replay buffer stores all previously seen transitions.
implementation: FIFO ring buffer

1. take action $\mathbf{a} \sim \pi_\theta(\mathbf{a}|\mathbf{s})$, get $(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime, r)$, store in $\mathcal{R}$
2. sample a batch $\{\mathbf{s}_i, \mathbf{a}_i, r_i, \mathbf{s}_i^\prime\}$ from buffer $\mathcal{R}$
3. update $\hat{V}_\phi^\pi$ using targets $y_i = r_i + \gamma \hat{V}_\phi^\pi(\mathbf{s}_i^\prime)$ to each $\mathbf{s}_i$
4. evaluate $\hat{A}^\pi(\mathbf{s}_i, \mathbf{a}_i) = r(\mathbf{s}_i, \mathbf{a}_i) + \gamma \hat{V}_\phi^\pi(\mathbf{s}_i^\prime) - \hat{V}_\phi^\pi(\mathbf{s}_i)$
5. evaluate $\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_i \nabla_\theta \log \pi_\theta(\mathbf{a}_i | \mathbf{s}_i) \, \hat{A}^\pi(\mathbf{s}_i, \mathbf{a}_i)$
6. update $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

use older transitions from the same but old actor
collect transitions, store in replay buffer, sample a batch w/ size $N$ (e.g., 32 transitions) from the replay buffer
transitions $\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime$ in replay buffer did not come from latest policy $\pi_\theta$ $\to$ need to modify the classical online value-based actor-critic algorithm

2 problems:

1. In step 3, the actions in the transitions from replay buffer were taken by older actors, which would not give value of the latest actor. Formally, $\mathbf{a}_i$ did not come from the latest policy $\pi_\theta$, but from an older policy. Thus, $\mathbf{s}_i^\prime$ was not the results of taking an action with the latest actor.
2. In step 5, because $\mathbf{a}_i$ did not come from the latest $\pi_\theta$, cannot compute off-policy policy gradient this way. Previously, use importance sampling.

Problem 1 $\to$ fix the value function

value function: expected return if start in state $\mathbf{s}_t$ and follow the policy $\pi$
$Q$ function: expected return if start in state $\mathbf{s}_t$, take action $\mathbf{a}_t$, then follow policy $\pi$

Keep the stored action $\mathbf{a}_i$ paired with the reward $r_i$ and next state $\mathbf{s}_i^\prime$ it produced. Learn $Q^\pi(\mathbf{s}_i, \mathbf{a}_i)$ rather than $V^\pi(\mathbf{s}_i)$: take the stored action first, then follow the current policy $\pi$.

So step 3 should be: update $\textcolor{red}{\hat{Q}_\phi^\pi}$ using targets $y_i = r_i + \gamma \textcolor{red}{\hat{V}_\phi^\pi}(\mathbf{s}_i^\prime)$ for each $\mathbf{s}_i, \mathbf{a}_i$
then regress against MSE: $L(\phi) = \frac{1}{N} \sum_i \| \hat{Q}_\phi^\pi(\mathbf{s}_i, \mathbf{a}_i) - y_i \|^2$

We want to learn $Q$, but how do we get the target value?
Because $V^\pi(\mathbf{s}) = \sum_{t'=t}^T E_{\pi_\theta} [r(\mathbf{s}_{t'}, \mathbf{a}_{t'}) | \mathbf{s}_t] = E_{\mathbf{a} \sim \pi(\mathbf{a}|\mathbf{s})} [Q^\pi(\mathbf{s}, \mathbf{a})]$, we can replace $V$ with $Q$ and use the target value

$y_i = r_i + \gamma \textcolor{red}{\hat{Q}_\phi^\pi}(\mathbf{s}_i^\prime, \mathbf{a}_i^{\prime,\pi})$

where $\mathbf{a}_i^{\prime,\pi} \sim \pi_\theta(\mathbf{a} | \mathbf{s}_i^\prime)$ is a fresh action sampled from the current policy at the replay-buffer next state, not an action stored in the replay buffer.

Problem 2 $\to$ fix the policy update

For the actor update, do not replace the action stored in the replay buffer. Instead, sample a separate fresh action $\mathbf{a}_i^\pi \sim \pi_\theta(\mathbf{a} | \mathbf{s}_i)$ from the current policy at the replay-buffer state $\mathbf{s}_i$. Step 5 becomes:

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_i \nabla_\theta \log \pi_\theta(\textcolor{red}{\mathbf{a}_i^\pi} | \mathbf{s}_i) \hat{A}^\pi(\mathbf{s}_i, \textcolor{red}{\mathbf{a}_i^\pi})$

Instead of using advantage $\hat{A}^\pi$, use $\hat{Q}^\pi$ directly in the equation.

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_i \nabla_\theta \log \pi_\theta(\mathbf{a}_i^\pi | \mathbf{s}_i) \textcolor{red}{\hat{Q}^\pi}(\mathbf{s}_i, \mathbf{a}_i^\pi)$

Higher var b/c there is no baseline, but OK b/c sampling $\mathbf{a}_i^\pi$ does not require the simulator; we only need extra forward passes through the policy network. In practice this trades lower bias from the action mismatch for higher variance, and the larger replay buffer batch size can offset some of that variance. Also, after switching to $\hat{Q}^\pi$, we can drop step 4.

Another small problem

$\mathbf{s}_i$ did not come from state marginal of latest policy $p_\theta (\mathbf{s})$, but from an old policy. Unfortunately, we can't do anything and this is a source of bias. It's acceptable b/c we want the optimal policy on $p_\theta (\mathbf{s})$, but we get optimal policy on a broader distr. The replay buffer contains many samples from the latest policy, and also many samples from older policies. The states from latest policy are not missed, but the trained policy is also good at other not so probable states.

Some implementation details

In step 4, do reparameterization trick to better estimate the gradient (for Gaussian policy)

there are better way to fit Q function (more on this in next two lectures)
e.g., Q learning with deterministic actor