CS 285, Lecture 7, Value Function Methods

CS 285 Deep Reinforcement Learning, Sergey Levine, Lecture 7, Value Function Methods notes.

This lecture asks whether we can omit policy gradients entirely. Instead of learning an explicit policy network, learn how good states and state-action pairs are, then choose the action with the highest value.

From Actor-Critic to Value-Based Methods

Recall the batch-mode actor-critic algorithm from Lecture 6:

1. Generate trajectories with the current policy $\pi$.
2. Fit a value function $V^\pi(\mathbf{s})$ to the collected data.
3. Estimate an advantage for every sampled state-action pair:

$\hat{A}^\pi(\mathbf{s}_t, \mathbf{a}_t) = r(\mathbf{s}_t, \mathbf{a}_t) + \gamma V^\pi(\mathbf{s}_{t+1}) - V^\pi(\mathbf{s}_t).$

4. Use the advantages in a policy-gradient update of the actor.

The actor-critic recipe still has the usual RL loop: collect samples, fit a value function, then improve the policy. The question for this lecture is: can the value function itself tell us how to act, so that the last step does not need a policy-gradient update or a separate policy network?

The key intuition is simple: a value function tells us which future states are better. If we choose actions that lead toward better states, an explicit actor may be unnecessary.

Greedy Improvement with the Advantage

For a policy $\pi$, the advantage is

$A^\pi(\mathbf{s}, \mathbf{a}) = Q^\pi(\mathbf{s}, \mathbf{a}) - V^\pi(\mathbf{s}).$

It measures how much better action $\mathbf{a}$ is than the average action selected by $\pi$ in state $\mathbf{s}$. Therefore,

$\arg\max_{\mathbf{a}} A^\pi(\mathbf{s}, \mathbf{a})$

is the best first action from $\mathbf{s}$ if we follow $\pi$ afterward. It must be at least as good as sampling an action from the current policy, because it is the best action under the same continuation policy.

This is true even if the current policy is poor or random. We can construct an improved policy $\pi^\prime$ that assigns probability one to the greedy action:

$\pi^\prime(\mathbf{a} \mid \mathbf{s}) = \begin{cases} 1 & \text{if } \mathbf{a} = \arg\max_{\mathbf{a}^\prime} A^\pi(\mathbf{s}, \mathbf{a}^\prime), \\ 0 & \text{otherwise.} \end{cases}$

The new policy is represented implicitly by an $\arg\max$, not by another neural network. After improving the policy, estimate values for $\pi^\prime$ and repeat.

Policy Iteration

This alternating procedure is called policy iteration:

1. Policy evaluation: estimate $A^\pi$ (or equivalently $V^\pi$ or $Q^\pi$) for the current policy.
2. Policy improvement: set $\pi \leftarrow \pi^\prime$, where $\pi^\prime$ is greedy with respect to that estimate.

For discrete actions, the improvement step is easy: evaluate each action and take the maximum. Continuous action spaces require optimization over actions and are treated later in the course. The main problem in this lecture is policy evaluation: how can we calculate the value function well enough to improve the policy?

Using the Bellman relation,

$A^\pi(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})}[V^\pi(\mathbf{s}^\prime)] - V^\pi(\mathbf{s}),$

so estimating the advantage reduces to estimating $V^\pi$.

Tabular Dynamic Programming

To make policy evaluation concrete, the lecture first considers an unusually favorable setting:

• the transition distribution $p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})$ is known;
• states and actions are small and discrete;
• every value can be stored in a table.

For example, in a grid world with $16$ states and four actions (left, right, up, down), $V^\pi$ is a table with $16$ entries. The transition model can be represented by a $16 \times 16 \times 4$ tensor. This is the tabular setting: no neural network is needed because the whole state-action space can be enumerated.

Policy Evaluation by Bellman Backups

For a fixed policy $\pi$, the Bellman backup is

$V^\pi(\mathbf{s}) \leftarrow E_{\mathbf{a} \sim \pi(\mathbf{a} \mid \mathbf{s})} \left[ r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})} [V^\pi(\mathbf{s}^\prime)] \right].$

In a tabular MDP with known dynamics, both expectations can be computed exactly by summing over all possible actions and next states, weighted by their probabilities. Repeatedly applying this update to every table entry converges to the true $V^\pi$.

Once policy iteration makes the policy greedy, it is deterministic. If $\pi(\mathbf{s})$ denotes its single selected action, the backup simplifies to

$V^\pi(\mathbf{s}) \leftarrow r(\mathbf{s}, \pi(\mathbf{s})) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \pi(\mathbf{s}))} [V^\pi(\mathbf{s}^\prime)].$

At convergence, these Bellman equations form a system of linear equations for the value table. It can be solved directly with a linear-system solver, or iteratively with the Bellman backups above.

From Policy Iteration to Value Iteration

Policy improvement can use $Q^\pi$ instead of $A^\pi$, because

$A^\pi(\mathbf{s}, \mathbf{a}) = Q^\pi(\mathbf{s}, \mathbf{a}) - V^\pi(\mathbf{s}).$

When taking $\arg\max$ over $\mathbf{a}$, the term $-V^\pi(\mathbf{s})$ is constant with respect to the action. Thus,

$\arg\max_{\mathbf{a}} A^\pi(\mathbf{s}, \mathbf{a}) = \arg\max_{\mathbf{a}} Q^\pi(\mathbf{s}, \mathbf{a}).$

Think of $Q(\mathbf{s}, \mathbf{a})$ as a table: each row is a state and each column is an action. The $\arg\max$ in a row gives the greedy policy action. Plugging that action back into the row gives the policy's value, which is simply the maximum entry in that row:

$V(\mathbf{s}) = \max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a}).$

For a fixed policy, the usual identity is still

$V^\pi(\mathbf{s}) = E_{\mathbf{a} \sim \pi(\mathbf{a} \mid \mathbf{s})} [Q^\pi(\mathbf{s}, \mathbf{a})].$

Policy evaluation averages Q-values under $\pi$. Value iteration instead performs greedy policy improvement at every backup, so it uses $V_{k+1}(\mathbf{s}) = \max_{\mathbf{a}}Q_k(\mathbf{s}, \mathbf{a})$ and converges to $V^\star(\mathbf{s}) = \max_{\mathbf{a}}Q^\star(\mathbf{s}, \mathbf{a})$. For a deterministic greedy policy, the expectation places all probability on the maximizing action and reduces to the same maximum.

This lets us skip explicitly constructing the policy. The resulting procedure is value iteration:

1. Form state-action values from the current value estimate:

$Q(\mathbf{s}, \mathbf{a}) \leftarrow r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})}[V(\mathbf{s}^\prime)].$

2. Set each state value to the best action value:

$V(\mathbf{s}) \leftarrow \max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a}).$

The maximization is policy improvement in implicit form. It avoids maintaining a separate policy representation: the greedy policy can always be recovered as $\arg\max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a})$.

Q-Value Bellman Optimality Backup

Substituting $V(\mathbf{s}^\prime) = \max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime)$ into value iteration eliminates $V$ as a separate object:

$Q(\mathbf{s}, \mathbf{a}) \leftarrow r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})} \left[\max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime)\right].$

This is the Bellman optimality backup for Q-values. It is the bridge to Q-learning, which removes the remaining requirement that we know the transition probabilities while continuing to represent the policy implicitly through the learned Q-function.

Function Approximation

Value iteration above relies on two restrictive assumptions: the value table can enumerate every state, and the transition dynamics are known. The lecture removes them in that order, first replacing the table with a function approximator, then replacing exact transition expectations with sampled transitions.

Tabular value functions only work when the state space is small enough to enumerate. An image with $200 \times 200$ pixels and three color channels already has

$255^{3 \times 200 \times 200}$

possible observations. A continuous state space has infinitely many states. More generally, the size of a table grows exponentially with the state dimension; this is the curse of dimensionality.

We therefore replace the table with a function approximator such as a neural network:

$V_\phi: \mathcal{S} \rightarrow \mathbb{R}.$

Fitted Value Iteration

Suppose we have sampled states $\{\mathbf{s}_i\}$. Fitted value iteration first calculates a target for every state using the current value function:

$y_i = \max_{\mathbf{a}} \left[ r(\mathbf{s}_i, \mathbf{a}) + \gamma E_{\mathbf{s}_i^\prime \sim p(\mathbf{s}_i^\prime \mid \mathbf{s}_i, \mathbf{a})} [V_\phi(\mathbf{s}_i^\prime)] \right].$

Then regress the value function onto those targets:

$\phi^\prime = \arg\min_{\phi^\prime} \frac{1}{2}\sum_i \left(V_{\phi^\prime}(\mathbf{s}_i) - y_i\right)^2.$

The two steps are alternated: perform a Bellman backup to construct targets, then fit the function approximator to the targets.

However, fitted value iteration still requires the transition dynamics. Computing each target needs the expectation over next states and the ability to try every action from exactly the same state. In model-free RL, we can usually execute a policy from the initial-state distribution, but cannot reset the real world to an arbitrary state and test every action.

Model-Free Fitted Q Iteration

Function approximation solves the representation problem, but fitted value iteration still needs known dynamics. To remove that assumption, return to policy evaluation and change what is approximated. A state-value recurrence for a deterministic policy is

$V^\pi(\mathbf{s}) = r(\mathbf{s}, \pi(\mathbf{s})) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \pi(\mathbf{s}))} [V^\pi(\mathbf{s}^\prime)].$

The action inside the transition distribution changes whenever the policy changes. By contrast, a Q-function recurrence conditions on an action already present in the data:

$Q^\pi(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})} [Q^\pi(\mathbf{s}^\prime, \pi(\mathbf{s}^\prime))].$

Given a sampled tuple $(\mathbf{s}, \mathbf{a}, r, \mathbf{s}^\prime)$, the transition to $\mathbf{s}^\prime$ is fixed and independent of the policy. Policy changes only affect the next action inside the Q-function. This small change lets us learn from stored transitions without knowing $p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})$.

Let the data set be

$\mathcal{D} = \{(\mathbf{s}_i, \mathbf{a}_i, r_i, \mathbf{s}_i^\prime)\}_{i=1}^N.$

The fitted Q-iteration target is

$y_i = r_i + \gamma \max_{\mathbf{a}_i^\prime} Q_\phi(\mathbf{s}_i^\prime, \mathbf{a}_i^\prime).$

Using the sampled next state replaces the expectation with a single-sample estimate. Fit a new Q-function by regression:

$\phi^\prime = \arg\min_{\phi^\prime} \frac{1}{2}\sum_i \left(Q_{\phi^\prime}(\mathbf{s}_i, \mathbf{a}_i) - y_i\right)^2.$

The general fitted Q-iteration algorithm is:

1. Collect transitions $(\mathbf{s}_i, \mathbf{a}_i, r_i, \mathbf{s}_i^\prime)$ using some behavior policy.
2. Calculate target values $y_i$ with the current Q-function.
3. Take one or more gradient steps to fit $Q_{\phi^\prime}(\mathbf{s}_i, \mathbf{a}_i)$ to $y_i$.
4. Alternate target calculation and fitting for $K$ iterations, then optionally collect more data.

For discrete actions, the Q-network may take both state and action as inputs and return one scalar. A common equivalent architecture takes only the state as input and returns one Q-value for every discrete action.

Fitted Q iteration needs only the transitions that were actually observed. The data need not come from the latest greedy policy, so old samples can be retained and reused. This makes it an off-policy algorithm, unlike the on-policy actor-critic method from Lecture 6.

Why Off-Policy Data Works

The greedy policy appears inside the target as

$\max_{\mathbf{a}^\prime} Q_\phi(\mathbf{s}^\prime, \mathbf{a}^\prime) = Q_\phi\left(\mathbf{s}^\prime, \arg\max_{\mathbf{a}^\prime} Q_\phi(\mathbf{s}^\prime, \mathbf{a}^\prime)\right).$

The Q-function evaluates the new action without requiring the environment to execute that action again. For a fixed observed $(\mathbf{s}, \mathbf{a})$, the next-state distribution does not change when the policy changes. As long as the stored transitions cover the relevant parts of the state-action space, the same data can support many Q-function updates.

The regression residual

$Q_\phi(\mathbf{s}_i, \mathbf{a}_i) - y_i$

is the Bellman error for the sample. In the tabular case the targets can be written directly into the table, and repeated updates converge to $Q^\star$. With nonlinear function approximation, the fit may not drive the error to zero, and the convergence guarantee is lost.

Practical Q-Learning

The generic fitted Q-iteration procedure has several free choices: how much data to collect, which behavior policy to use, how many fitting steps to take, and how many target-and-fit rounds to run before gathering more data. Particular choices recover familiar algorithms. The simplest online version processes one transition at a time.

Online Update

A special case of fitted Q iteration processes one transition at a time:

1. Take one action $\mathbf{a}_i$ and observe $(\mathbf{s}_i, \mathbf{a}_i, r_i, \mathbf{s}_i^\prime)$.
2. Calculate

$y_i = r_i + \gamma \max_{\mathbf{a}^\prime} Q_\phi(\mathbf{s}_i^\prime, \mathbf{a}^\prime).$

3. Take one gradient step on

$\mathcal{L}_i(\phi) = \frac{1}{2} \left(Q_\phi(\mathbf{s}_i, \mathbf{a}_i) - y_i\right)^2.$

The corresponding semi-gradient update is

$\phi \leftarrow \phi - \alpha \nabla_\phi Q_\phi(\mathbf{s}_i, \mathbf{a}_i) \left(Q_\phi(\mathbf{s}_i, \mathbf{a}_i) - y_i\right).$

The term $Q_\phi(\mathbf{s}_i, \mathbf{a}_i) - y_i$ is the temporal-difference (TD) error. In a tabular representation, the update is

$Q(\mathbf{s}_i, \mathbf{a}_i) \leftarrow Q(\mathbf{s}_i, \mathbf{a}_i) + \alpha \left[ r_i + \gamma \max_{\mathbf{a}^\prime}Q(\mathbf{s}_i^\prime, \mathbf{a}^\prime) - Q(\mathbf{s}_i, \mathbf{a}_i) \right].$

This is online Q-learning, also called Watkins' Q-learning. Its learned policy is greedy with respect to $Q$, but the behavior policy used to gather data does not have to be greedy.

Exploration

Always executing $\arg\max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a})$ during training is risky. The greedy policy is deterministic, while the initial Q-function is arbitrary and inaccurate. It may repeatedly select the same poor action and never discover better alternatives.

$\varepsilon$-Greedy Exploration

With $|\mathcal{A}|$ discrete actions, an $\varepsilon$-greedy behavior policy takes the greedy action most of the time and explores otherwise:

$\pi(\mathbf{a} \mid \mathbf{s}) = \begin{cases} 1 - \varepsilon & \text{if } \mathbf{a} = \arg\max_{\mathbf{a}^\prime}Q(\mathbf{s}, \mathbf{a}^\prime), \\ \dfrac{\varepsilon}{|\mathcal{A}| - 1} & \text{otherwise.} \end{cases}$

A common schedule starts with a larger $\varepsilon$ while the Q-function is unreliable, then decreases it as learning progresses.

Boltzmann Exploration

Another option samples actions according to exponentiated Q-values:

$\pi(\mathbf{a} \mid \mathbf{s}) = \frac{\exp(Q_\phi(\mathbf{s}, \mathbf{a}) / \tau)} {\sum_{\mathbf{a}^\prime}\exp(Q_\phi(\mathbf{s}, \mathbf{a}^\prime) / \tau)}.$

This is called Boltzmann or softmax exploration. Similarly valued actions receive similar probabilities, while an action already known to be very poor is selected rarely. The temperature $\tau$ controls how random the behavior is.

Convergence Theory

The lecture closes by separating what is guaranteed in the tabular case from what can fail after introducing function approximation.

The Bellman Operator

For tabular value iteration, define the Bellman optimality operator $\mathcal{B}$:

$(\mathcal{B}V)(\mathbf{s}) = \max_{\mathbf{a}} \left[ r(\mathbf{s}, \mathbf{a}) + \gamma E_{\mathbf{s}^\prime \sim p(\mathbf{s}^\prime \mid \mathbf{s}, \mathbf{a})} [V(\mathbf{s}^\prime)] \right].$

Value iteration is simply

$V \leftarrow \mathcal{B}V.$

The optimal value function is the unique fixed point:

$V^\star = \mathcal{B}V^\star.$

The Bellman operator is a $\gamma$-contraction under the infinity norm:

$\|\mathcal{B}V - \mathcal{B}\bar{V}\|_\infty \leq \gamma \|V - \bar{V}\|_\infty.$

Setting $\bar{V}=V^\star$ gives

$\|\mathcal{B}V - V^\star\|_\infty \leq \gamma \|V - V^\star\|_\infty.$

Because $\gamma < 1$, every Bellman update moves the largest entry-wise error closer to zero. Therefore tabular value iteration converges to $V^\star$, from which the optimal policy is recovered by the greedy $\arg\max$.

Why Function Approximation Can Diverge

Let $\Omega$ be the set of value functions representable by a chosen function approximator. A Bellman backup $\mathcal{B}V$ may lie outside $\Omega$, so supervised fitting projects it back into the hypothesis class. Denote this projection by $\Pi$:

$\Pi V = \arg\min_{V^\prime \in \Omega} \|V^\prime - V\|_2^2.$

Fitted value iteration is

$V \leftarrow \Pi\mathcal{B}V.$

The Bellman backup is a contraction in the infinity norm, while least-squares projection is associated with the $L_2$ norm. Even though each operation has a contraction property in its own norm, their composition $\Pi\mathcal{B}$ need not be a contraction under any one norm. A backup may move toward $V^\star$, then the projection can move the estimate farther away. Consequently:

• tabular value iteration converges;
• fitted value iteration with function approximation is not guaranteed to converge;
• the same issue applies to fitted Q iteration and online Q-learning with neural networks;
• bootstrapped actor-critic policy evaluation has the same general problem.

Q-learning resembles gradient descent on a squared Bellman error, but it is not ordinary gradient descent on a fixed objective. Its target

$y_i = r_i + \gamma \max_{\mathbf{a}^\prime}Q_\phi(\mathbf{s}_i^\prime, \mathbf{a}^\prime)$

also depends on the current parameters $\phi$, while the usual update does not differentiate through the target. It is therefore a semi-gradient method rather than the true gradient of a fixed loss. Differentiating through the target leads to residual-gradient methods, which have convergence guarantees in some settings but poor numerical behavior in practice.

Takeaways

• Value-based RL can improve a policy by choosing the action with the largest estimated value rather than differentiating through a policy network.
• Policy iteration alternates between evaluating the current policy and greedily improving it.
• In small, known tabular MDPs, exact dynamic programming evaluates a policy through repeated Bellman backups.
• Value iteration combines evaluation and improvement by taking a maximum over actions.
• Fitted Q iteration learns from sampled transitions, does not require known dynamics, and can reuse off-policy data.
• Online Q-learning performs one TD update per transition; exploration policies prevent a poor initial Q-function from getting stuck.
• The tabular Bellman operator is a contraction and converges to the optimal value function.
• Combining Bellman backups with nonlinear function approximation is not guaranteed to converge, because backup and projection contract under different norms.