CS 285, Lecture 5, Policy Gradients

CS 285 Deep Reinforcement Learning, Sergey Levine, Lecture 5, Policy Gradients notes.

The goal of RL

Let $\tau = \mathbf{s}_1, \mathbf{a}_1, \ldots, \mathbf{s}_T, \mathbf{a}_T$, then

$p_\theta(\tau) = p_\theta(\mathbf{s}_1, \mathbf{a}_1, \ldots, \mathbf{s}_T, \mathbf{a}_T) = p(s_1) \prod_{t=1}^{T} \pi_\theta(\mathbf{a}_t | \mathbf{s}_t) p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$

The RL objective is the expectation under the trajectory distribution:

$\theta^\star = \arg \max_\theta E_{r \sim p_\theta (\tau)} \left[ \sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right]$

can push sum out of expectation

finite horizon case:

$\theta^\star = \arg \max_\theta \sum_{t = 1}^T E_{(\mathbf{s}_t, \mathbf{a}_t) \sim p_\theta (\mathbf{s}_t, \mathbf{a}_t)} \left[r(\mathbf{s}_t, \mathbf{a}_t) \right]$

infinite horizon case:

$\theta^\star = \arg \max_\theta E_{(\mathbf{s}, \mathbf{a}) \sim p_\theta (\mathbf{s}, \mathbf{a})} [r(\mathbf{s}, \mathbf{a})]$

this lecture focuses on finite horizon version.

In model-free RL, do not assume we know transition probability $p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$ and initial state probability $p(\mathbf{s}_1)$ . Instead, we assume can interact with the real world, and samples from these distributions.

Evaluate the objective

$\theta^\star = \arg \max_\theta E_{\tau \sim p_\theta (\tau)}\left[\sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right]$

Let $J(\theta) = \arg \max_\theta E_{\tau \sim p_\theta(\tau)} \left[\sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right]$
run our policy N times and collect N sampled trajectories ($i, t$ means time step $t$ in the $i$-th sample):

$J(\theta) = E_{\tau \sim p_\theta(\tau)} \left[\sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right] \approx \frac{1}{N}\sum_i\sum_t r(\mathbf{s}_{i, t}, \mathbf{a}_{i, t})$

where $\sum_i$ is the sum over samples from $\pi_\theta$.

Direct policy differentiation

Let $r(\tau) = \sum_{t=1}^T r(\mathbf{s}_t, \mathbf{a}_t)$

$J(\theta) = E_{\tau \sim p_\theta(\tau)} \left[r(\tau) \right] = \int p_\theta(\tau) r(\tau) d\tau$

use a convenient identity:

$p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) = p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} = \nabla_\theta p_\theta(\tau)$

we want to optimize $J$ using its gradient:

$\nabla_\theta J(\theta) = \int \nabla_\theta p_\theta(\tau) r(\tau) d\tau = \int p_\theta(\tau) \nabla_\theta\log p_\theta(\tau) r(\tau) d\tau = E_{\tau \sim p_\theta(\tau)} \left[\nabla_\theta \log p_\theta(\tau)r(\tau) \right]$

recall that

$p_\theta(\tau) = p_\theta(\mathbf{s}_1, \mathbf{a}_1, \ldots, \mathbf{s}_T, \mathbf{a}_T) = p(s_1) \prod_{t=1}^{T} \pi_\theta(\mathbf{a}_t | \mathbf{s}_t) p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$

take $\log$ from both sides,

$\log p_\theta(\tau) = \log p(\mathbf{s}_1) + \sum_{t=1}^T \log \pi_\theta(\mathbf{a}_t | \mathbf{s}_t) + \log p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$

As $\log p(\mathbf{s}_1)$ and $\log p(\mathbf{s}_{t+1} | \mathbf{s}_t, \mathbf{a}_t)$ are independent of $\theta$,

$\begin{aligned} \nabla \log p_\theta(\tau) &= \nabla_\theta \left(\sum_{t=1}^T \log \pi_\theta (\mathbf{a}_t | \mathbf{s}_t) \right) \\ \nabla_\theta J(\theta) &= E_{\tau \sim p_\theta(\tau)} \left[\nabla_\theta \log p_\theta(\tau)r(\tau) \right] = E_{\tau \sim p_\theta(\tau)} \left[\nabla_\theta \left(\sum_{t=1}^T \log \pi_\theta (\mathbf{a}_t | \mathbf{s}_t) \right) r(\tau) \right] = E_{\tau \sim p_\theta(\tau)} \left[ \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t | s_t) \right) \left( \sum_{t=1}^T r(s_t, a_t) \right) \right] \end{aligned}$

Now everything inside the expectation is known, as we know policy $\pi$ and can evaluate reward for samples. The unknown terms, like initial prob and transition prob, only occur in the $\log p_\theta(\tau)$.

Evaluate policy gradient

recall:

$J(\theta) = E_{\tau \sim p_\theta(\tau)} \left[\sum_t r(\mathbf{s}_t, \mathbf{a}_t) \right] \approx \frac{1}{N}\sum_i\sum_t r(\mathbf{s}_{i, t}, \mathbf{a}_{i, t})$

and we just got:

$\nabla_\theta J(\theta) = E_{\tau \sim p_\theta(\tau)} \left[ \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t | s_t) \right) \left( \sum_{t=1}^T r(s_t, a_t) \right) \right]$

Run the policy, sum up the rewards to determine if a trajectory is good or bad, then multiply by the sum of $\nabla_\theta \log \pi_\theta$'s:

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_i, t | s_i, t) \right) \left( \sum_{t=1}^T r(s_i, t, a_i, t) \right)$

$\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

REINFORCE algorithm (a loop):

1. sample $\{r^i\}$ from $\pi_\theta(a_t | s_t)$ (run the policy)
2. sum up rewards along each sample trajectory, then calc policy gradient: $\nabla_\theta J(\theta) \approx \sum_i \left( \sum_t \nabla_\theta \log \pi_\theta(a_t^i | s_t^i) \right) \left( \sum_t r(s_t^i, a_t^i) \right)$
3. 1 step of gradient descent/ascent: $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

Problem: won't work well in practice.

Understanding Policy Gradient

Comparison to maximum likelihood

policy gradient runs own policy, can give good or bad actions:

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \left( \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | s_{i,t}) \right) \left( \sum_{t=1}^{T} r(s_{i,t}, \mathbf{a}_{i,t}) \right)$

maximum likelihood:

$\nabla_{\theta} J_{\text{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \left( \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | s_{i,t}) \right)$

ML gradient increases prob of all actions, whereas policy gradient may increase or decrease. High reward trajectory increases log probability, and low reward trajectory decreases log probability.

Example: Gaussian policies

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta (\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \right) \left( \sum_{t=1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) \right)$

example: $\pi_\theta (\mathbf{a}_t | \mathbf{s}_t) = \mathcal{N}(f_{\textrm{NN}} (\mathbf{s}_t); \Sigma)$. Neural network predicts the mean, and covariance could be learned or fixed. Train the neural network $f$:

$\log \pi_\theta (\mathbf{a}_t | \mathbf{s}_t) = -\frac{1}{2} \| f(\mathbf{s}_t) - \mathbf{a}_t \|_\Sigma^2 + \text{const}$

$\nabla_\theta \log \pi_\theta (\mathbf{a}_t | \mathbf{s}_t) = -\frac{1}{2} \Sigma^{-1} (f(\mathbf{s}_t) - \mathbf{a}_t) \frac{df}{d\theta}$

iteration 2000: humanoid can walk!

Intuition of Policy Gradients

Gradient of policy gradients in full form:

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta (\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \right) \left( \sum_{t=1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) \right)$

Simply put, let $\log \pi_\theta (\tau_i) = \sum_{t=1}^T \nabla_\theta \log_\theta \pi_\theta (\mathbf{a}_{i,t} | \mathbf{s}_{i,t})$, then:

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \nabla_\theta \log \pi_\theta (\tau_i) r(\tau_i)$

In comparison, the gradient for maximum likelihood is:

$\nabla_\theta J_{\text{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^N \nabla_\theta \log \pi_\theta (\tau_i)$

Intuitively, good trajectory is made more likely and bad trajectory is made less likely. This simply formalizes the notion of "trial and error".

Partial observability

So far, we have not actually used Markov property for states. For partial observations, can use policy gradient in partially observed MDPs without modification.

$\nabla_{\theta}J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \left( \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{o}_{i,t}) \right) \left( \sum_{t=1}^{T} r(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) \right)$

Problem: policy gradient has high variance

For finite sample size, adding constant to sample rewards could yield high variance and influence policy gradients in different degrees.

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log \pi_{\theta}(\tau) r(\tau)$

Example: if the two "good" samples have $r(\tau) = 0$, their gradients don't matter.

Reduce Variance

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta (\mathbf{a}_{i,t} | s_{i,t}) \right) \left( \sum_{t=1}^T r(s_{i,t}, \mathbf{a}_{i,t}) \right)$

Causality (always true): policy at time $t^\prime$ cannot affect reward at time $t$ if $t < t^\prime$.
past rewards are independent of present decision.
not the same as Markov property (not always true): future state is independent of past state given the present.

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t = 1}^T \nabla_\theta \log \pi_\theta (\mathbf{a}_{i, t} | s_{i, t}) \left( \sum_{t^\prime = t}^T r(s_{i, t^\prime}, \mathbf{a}_{i, t^\prime}) \right)$

After removing past rewards, the estimator stays unbiased, but with lower variance (smaller number of terms).

This quantity is sometimes called "reward to go" (without rewards in the past):

$\hat{Q}_{i, t} = \sum_{t^\prime = t}^T r(s_{i, t^\prime}, \mathbf{a}_{i, t^\prime})$

where the $\,\hat{}\,$ means single sample estimator, and $Q$ means state-value function.

Baselines

increase the rewards for the trajectory better than average, and decrease if worse than average.

$\begin{aligned} \nabla_\theta J(\theta) &\approx \frac{1}{N} \sum_{i=1}^N \nabla_\theta \log p_\theta(\tau)[r(\tau) - b] \\ b &= \frac{1}{N} \sum_{i=1}^N r(\tau) \end{aligned}$

Why are we allowed to do that?
For any $b$, subtracting $b$ would make the estimator unbiased.

$E[\nabla_\theta \log p_\theta(\tau)b] = \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau)b \, d\tau = \int \nabla_\theta p_\theta(\tau)b \, d\tau = b \nabla_\theta \int p_\theta(\tau)d\tau = b \nabla_\theta 1 = 0$

The 2nd equal sign comes from a convenient identity:

$p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) = \nabla_\theta p_\theta(\tau)$

Subtracting a baseline is unbiased in expectation and reduces the variance.
Average reward is not the best baseline, but it's pretty good.

Find the best baseline

The variance:

$\begin{aligned} &\mathrm{Var}[x] = E[x^2] - E[x]^2 \\ &\nabla_\theta J(\theta) = E_{\tau \sim p_\theta(\tau)}[\nabla_\theta \log p_\theta(\tau)(r(\tau)-b)] \\ &\textrm{Var}[\nabla_\theta J(\theta)] = \mathbb{E}_{\tau \sim p_\theta(\tau)} \left[ \left( \nabla_\theta \log p_\theta(\tau)(r(\tau)-b) \right)^2 \right] - \underbrace{ \mathbb{E}_{\tau \sim p_\theta(\tau)} \left[ \nabla_\theta \log p_\theta(\tau)(r(\tau)-b) \right]^2 }_{ \substack{ \text{this is } \mathbb{E}_{\tau \sim p_\theta(\tau)} [\nabla_\theta \log p_\theta(\tau) r(\tau)] \\ \text{(baselines are unbiased in expectation)} }} \end{aligned}$

Let $g(\tau) = \nabla_\theta \log p_\theta(\tau)$, and cross out $E[g(\tau)^2 r(\tau)^2]$ as not dependent on $b$:

$\begin{aligned} \frac{d\textrm{Var}}{db} &= \frac{d}{db} E\left[ g(\tau)^2 (r(\tau)-b)^2 \right] = \frac{d}{db} \left( {\color{red}\cancel{\color{black}E[g(\tau)^2 r(\tau)^2]}} - 2 E[g(\tau)^2 r(\tau)] + b^2 E[g(\tau)^2] \right) \\ &= -2E[g(\tau)^2 r(\tau)] + 2bE[g(\tau)^2] = 0 \end{aligned}$

So, optimal baseline: expected reward weighted by param gradient magnitude.

$b = \frac{E[g(\tau)^2(r(\tau)]}{E[g(\tau)^2]}$

Baseline depends on the gradient of the param.

In practice, we don't calc the optimal baseline for every dim, and just use the expected reward.

Summary

In summary, to eliminate the high var of policy gradient, we can exploit causality as future does not affect the past, subtract baseline as it renders the estimator unbiased. We could use optimal baseline, but usually just use the expected reward as baseline.

Problem: Policy gradient is on-policy

Why is policy gradient on-policy?

$\begin{aligned} \theta^* &= \arg\max_{\theta} J(\theta) \\ J(\theta) &= E_{\tau \sim p_\theta(\tau)}[r(\tau)] \\ \nabla_\theta J(\theta) &= E_{\color{red}{\tau \sim p_\theta(\tau)}}[\nabla_\theta \log p_\theta(\tau)\, r(\tau)] \end{aligned}$

As 1st step of REINFORCE algo, the red part means policy gradients have to sample $\{\tau^i\}$ from $\pi_\theta(\mathbf{a}_t | \mathbf{s}_t)$, even though NN changes only a little bit with each gradient step. But on-policy learning can be extremely inefficient!

on-policy: each update step requires fresh samples from own current policy.

When generating samples is very cheap, policy gradient can be a great choice, b/c it's simple to implement and work fairly well.

Importance sampling

a general technique to evaluate an expectation from one distribution when only given samples from another distribution.

importance sampling:

$\begin{aligned} E_{x \sim p(x)}[f(x)] &= \int p(x) f(x) dx \\ &= \int \frac{q(x)}{q(x)} p(x) f(x) dx \\ &= \int q(x) \frac{p(x)}{q(x)} f(x) dx \\ &= E_{x \sim q(x)} \left[ \frac{p(x)}{q(x)} f(x) \right] \end{aligned}$

$\begin{aligned} \theta^* &= \arg\max_{\theta} J(\theta) \\ J(\theta) &= \mathbb{E}_{\tau \sim p_\theta(\tau)}[\,r(\tau)\,] \end{aligned}$

For off policy, we have samples from some $\bar{p}(\tau)$ instead of $p_\theta(\tau)$:

$\begin{aligned} J(\theta) &= \mathbb{E}_{\tau \sim \bar{p}(\tau)} \left[ {\color{red}{\frac{p_\theta(\tau)}{\bar{p}(\tau)}}} \; r(\tau) \right] \\ p_\theta(\tau) &= p(\mathbf{s}_1) \prod_{t=1}^T \pi_\theta(\mathbf{a}_t | \mathbf{s}_t)\; p(\mathbf{s}_{t+1} | \mathbf{s}_t,\mathbf{a}_t) \\ \frac{p_\theta(\tau)}{\bar{p}(\tau)} &= \frac{ p(\mathbf{s}_1) \prod_{t=1}^T \pi_\theta(\mathbf{a}_t | \mathbf{s}_t)\; p(\mathbf{s}_{t+1} | \mathbf{s}_t,\mathbf{a}_t) }{ p(\mathbf{s}_1) \prod_{t=1}^T \bar{\pi}(\mathbf{a}_t | \mathbf{s}_t)\; p(\mathbf{s}_{t+1} | \mathbf{s}_t,\mathbf{a}_t) } \\ &= \frac{ {\color{red}\cancel{\color{black}p(\mathbf{s}_1)}}\; \prod_{t=1}^T \pi_\theta(\mathbf{a}_t | \mathbf{s}_t)\; {\color{red}\cancel{\color{black}p(\mathbf{s}_{t+1} | \mathbf{s}_t,\mathbf{a}_t)}} }{ {\color{red}\cancel{\color{black}p(\mathbf{s}_1)}}\; \prod_{t=1}^T \bar{\pi}(\mathbf{a}_t | \mathbf{s}_t)\; {\color{red}\cancel{\color{black}p(\mathbf{s}_{t+1} | \mathbf{s}_t,\mathbf{a}_t)}} } \\ &= \frac{ \prod_{t=1}^T \pi_\theta(\mathbf{a}_t | \mathbf{s}_t) }{ \prod_{t=1}^T \bar{\pi}(\mathbf{a}_t | \mathbf{s}_t) } \end{aligned}$

the unknown initial prob and transition prob cancel out. Luckily, we know the policy prob and can evaluate the importance weight.

We have samples from a diff policy $p_\theta(\tau)$, and want to estimate the value of some new parameters $\theta^\prime$:

$J(\theta^\prime) = E_{\tau \sim p_\theta(\tau)} \left[ \frac{\color{red}p_{\theta^\prime}(\tau)}{p_\theta(\tau)} r(\tau) \right]$

where $p_{\theta^\prime}(\tau)$ is the only bit that depends on $\theta^\prime$.

Use the convenient identity $p_\theta(\tau) \nabla_{\theta} \log p_\theta(\tau) = \nabla_{\theta} p_\theta(\tau)$,

$\nabla_{\theta^\prime} J(\theta^\prime) = E_{\tau \sim p_\theta(\tau)} \left[ \frac{\nabla_{\theta^\prime} p_{\theta^\prime}(\tau)}{p_\theta(\tau)} \; r(\tau) \right] = E_{\tau \sim p_\theta(\tau)} \left[ \frac{p_{\theta^\prime}(\tau)}{p_\theta(\tau)} \; \nabla_{\theta^\prime} \log p_{\theta^\prime}(\tau)\; r(\tau) \right]$

If estimate locally at $\theta = \theta^\prime$:

$\nabla_{\theta} J(\theta) = E_{\tau \sim p_\theta(\tau)} [\nabla_{\theta} \log p_\theta(\tau) r(\tau)]$

This accords with our prev deduction of policy gradient.

The off-policy policy gradient

$\theta^\star = \arg\max_\theta J(\theta) \qquad\qquad J(\theta) = E_{\tau \sim p_\theta(\tau)}[r(\tau)]$

When $\theta \neq \theta^\prime$,

$\nabla_{\theta^\prime} J(\theta^\prime) = E_{\tau\sim p_\theta(\tau)} \left[ \frac{p_{\theta^\prime}(\tau)}{p_\theta(\tau)} \nabla_{\theta^\prime} \log p_{\theta^\prime}(\tau)\, r(\tau) \right]$

Since $\frac{p_{\theta^\prime}(\tau)}{p_{\theta}(\tau)}=\frac{\prod_{t=1}^{T}\pi_{\theta^\prime}(\mathbf{a}_{t} | \mathbf{s}_{t})}{\prod_{t=1}^{T}\pi_{\theta}(\mathbf{a}_{t} | \mathbf{s}_{t})}$, we can expand:

$= E_{\tau\sim p_\theta(\tau)} \left[ \left( \prod_{t=1}^T \frac{\pi_{\theta^\prime}(a_t | s_t)}{\pi_{\theta}(a_t | s_t)} \right) \left( \sum_{t=1}^T \nabla_{\theta^\prime}\log\pi_{\theta^\prime}(a_t | s_t) \right) \left( \sum_{t=1}^T r(s_t,a_t) \right) \right]$

By causality (future actions don't affect current weights):

$= E_{\tau\sim p_\theta(\tau)} \left[ \sum_{t=1}^T \nabla_{\theta^\prime}\log\pi_{\theta^\prime}(a_t | s_t) \left( \prod_{t''=1}^{t} \frac{\pi_{\theta^\prime}(a_{t''} | s_{t''})}{\pi_\theta(a_{t''} | s_{t''})} \right) \left( \sum_{t^\prime=t}^T r(s_{t^\prime},a_{t^\prime}) \left( {\color{red}\cancel{\color{black} \prod_{t''=t}^{t} \frac{\pi_{\theta^\prime}(a_{t''} | s_{t''})} {\pi_\theta(a_{t''} | s_{t''})} }} \right) \right) \right]$

The crossing can be done b/c if we ignore this, we get a policy iteration algorithm (more on this in a later lecture).

A first-order approximation for IS (preview)

$\nabla_{\theta^\prime} J(\theta^\prime) = E_{\tau \sim p_\theta(\tau)} \left[ \sum_{t=1}^T \nabla_{\theta^\prime} \log \pi_{\theta^\prime}(\mathbf{a}_t | \mathbf{s}_t) \underbrace{\left( \prod_{t''=1}^t \frac{\pi_{\theta^\prime}(\mathbf{a}_{t''} | \mathbf{s}_{t''})}{\pi_{\theta}(\mathbf{a}_{t''} | \mathbf{s}_{t''})} \right)}_{\text{exponential in T}} \left( \sum_{t^\prime=t}^T r(\mathbf{s}_{t^\prime}, \mathbf{a}_{t^\prime}) \right) \right]$

To avoid the exponential and high variance, let's write the objective a bit differently.

on-policy policy gradient, sample $(\mathbf{s}_{i,t}, \mathbf{a}_{i,t}) \sim \pi_\theta(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})$ by rolling out policy:

$\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \nabla_\theta \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \hat{Q}_{i,t}$

off-policy policy gradient, sample $(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})$ from state-action marginal at $t$:

$\begin{aligned} \nabla_{\theta^\prime} J(\theta^\prime) &\approx \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \frac{\pi_{\theta^\prime}(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})}{\pi_{\theta}(\mathbf{s}_{i,t}, \mathbf{a}_{i,t})} \nabla_{\theta^\prime} \log \pi_{\theta^\prime}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \hat{Q}_{i,t} \\ &= \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \underbrace{\color{red}{\cancel{\color{black}{\frac{\pi_{\theta^\prime}(\mathbf{s}_{i,t})}{\pi_{\theta}(\mathbf{s}_{i,t})}}}}}_{\text{ignore this part}} \frac{\pi_{\theta^\prime}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})}{\pi_{\theta}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})} \nabla_{\theta^\prime} \log \pi_{\theta^\prime}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \hat{Q}_{i,t} \end{aligned}$

ignore state marginal prob when $\theta^\prime$ is not too different from $\theta$, but keep state-action conditional prob.

TODO: We'll see why this is reasonable later in the course.

Implementing Policy Gradients

Policy gradient with automatic differentiation

$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{Q}_{i,t}$

Have to calc policy gradient explicitly for every param in the NN, but pretty inefficient.

How can we compute policy gradients with automatic differentiation? We need a graph such that its gradient is the policy gradient!

maximum likelihood:

$\nabla_{\theta} J_{\text{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \quad J_{\text{ML}}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t})$

Just implement "pseudo-loss" as a weighted maximum likelihood:

$\tilde{J}(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T} \log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t}) \hat{Q}_{i,t}$

$\log \pi_{\theta}(\mathbf{a}_{i,t} | \mathbf{s}_{i,t})$ would be cross entropy for discrete distribution or squared error for Gaussian.

Pseudocode example (with discrete actions)

Maximum likelihood:

# Given:
# actions - (N*T) x Da tensor of actions
# states - (N*T) x Ds tensor of states
# Build the graph:
logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logits
negative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)
loss = tf.reduce_mean(negative_likelihoods)
gradients = loss.gradients(loss, variables)

Policy gradient:

# Given:
# actions - (N*T) x Da tensor of actions
# states - (N*T) x Ds tensor of states
# q_values - (N*T) x 1 tensor of estimated state-action values
# Build the graph:
logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logits
negative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)
weighted_negative_likelihoods = tf.multiply(negative_likelihoods, q_values)
loss = tf.reduce_mean(weighted_negative_likelihoods)
gradients = loss.gradients(loss, variables)

q_values refer to $\hat{Q}_{i,t}$.

Policy gradient in practice

• Remember that policy gradient has high variance (soln: causality, baseline)
• B/c gradients are quite noisy, need larger batches (at least 1k)
• Tweaking learning rates is very hard
  • Adaptive step size rules like ADAM can be an OK starting point
  • will talk about policy gradient-specific learning rate adjustment methods

Review

• Policy gradient is on-policy
• Can derive off-policy variant
  • Use importance sampling
  • Exponential scaling in T
  • Can ignore state portion (approximation)
• Can implement with automatic differentiation - need to know what to backpropagate
• Practical considerations: batch size, learning rates, optimizers

Advanced Policy Gradients

Numerical issue

Policy gradient has numerical issue especially in continuous action space.

$\begin{aligned} r(\mathbf{s}_t, \mathbf{a}_t) &= -\mathbf{s}_t^2 - \mathbf{a}_t^2 \\ \log \pi_\theta(\mathbf{a}_t|\mathbf{s}_t) &= -\frac{1}{2\sigma^2}(k\mathbf{s}_t - \mathbf{a}_t)^2 + \text{const} \quad \theta = (k, \sigma) \end{aligned}$

Near Gaussian mode, arrow does not point towards the optimal. As $\sigma$ approaches the optimum, its gradient gets larger. After gradient is normalized, the $\sigma$ portion dominates whereas the $k$ portion is too small. So if using 1st order methods, will spend a long time to reach optimum due to poor conditioning.

Essentially the same problem as this:

Covariant/natural policy gradient

Different params affect the policy in different degrees. Some params change the policy a lot (want smaller lr), some don't change the policy much (want larger lr).

$\theta \leftarrow \theta + \alpha \nabla_{\theta} J(\theta) \qquad \pi_{\theta}(\mathbf{a}_t|\mathbf{s}_t)$

constrained optimization for 1st order gradient descent, where the constraint controls how far we can go:

$\theta^\prime \leftarrow \arg \max_{\theta^\prime} (\theta^\prime - \theta)^T \nabla_{\theta} J(\theta) \text{ s.t. } ||\theta^\prime - \theta||^2 \le \epsilon$

$\epsilon$ ball is in $\theta$ space, but want to reparametrize s.t. steps are of equal size in policy space rather than param space.

rescale the gradient:

$\theta^\prime \leftarrow \arg \max_{\theta^\prime} (\theta^\prime - \theta)^T \nabla_{\theta} J(\theta) \text{ s.t. } D(\pi_{\theta^\prime}, \pi_{\theta}) \le \epsilon$

using parameterization-independent divergence measure, usually KL divergence:

$D_{\text{KL}}(\pi_{\theta}||\pi_{\theta^\prime}) = E_{\pi_{\theta}}[\log \pi_{\theta} - \log \pi_{\theta^\prime}]$

The KL divergence can be approximated as:

$D_{\text{KL}}(\pi_{\theta^\prime}||\pi_{\theta}) \approx (\theta^\prime - \theta)^T \mathbf{F} (\theta^\prime - \theta)$

where $F$ is the Fisher-information matrix, and can estimate with samples:

$\mathbf{F} = E_{\pi_{\theta}}[\nabla_{\theta} \log \pi_{\theta}(\mathbf{a}|\mathbf{s}) \nabla_{\theta} \log \pi_{\theta}(\mathbf{a}|\mathbf{s})^T]$

Now the params are updated as:

$\theta \leftarrow \theta + \alpha \mathbf{F}^{-1} \nabla_\theta J(\theta)$

where $\alpha$ is the Lagrange multiplier, and we add $mathbf{F}^{-1}$ to param update.

natural gradient: pick $\alpha$
TRPO (trust region policy optimization): pick $\epsilon$, then derive $\alpha$
using conjugate gradient, can solve for optimal $\alpha$ while solving $\mathbf{F}^{-1} \nabla_\theta J(\theta)$.

Next time

• Introduce value functions and Q-function
• Later in the class: more on natural gradient and automatic step size adjustment

Policy gradients suggested readings

• Classic papers
  [x] Williams (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning: introduces REINFORCE algorithm
  [] Baxter & Bartlett (2001). Infinite-horizon policy-gradient estimation: temporally decomposed policy gradient (not the first paper on this! see actor-critic section later)
  [] Peters & Schaal (2008). Reinforcement learning of motor skills with policy gradients: very accessible overview of optimal baselines and natural gradient
• Deep reinforcement learning policy gradient papers
  [] Levine & Koltun (2013). Guided policy search: deep RL with importance sampled policy gradient (unrelated to later discussion of guided policy search)
  [] Schulman, L., Moritz, Jordan, Abbeel (2015). Trust region policy optimization: deep RL with natural policy gradient and adaptive step size
  [] Schulman, Wolski, Dhariwal, Radford, Klimov (2017). Proximal policy optimization algorithms: deep RL with importance sampled policy gradient