DQN, DDPG, SAC, and Modern RL Applications

From DQN to continuous action spaces: the deterministic policy gradient, TD3's 3 fixes, SAC's maximum-entropy formulation, the squashed Gaussian policy, and where these ideas stand in modern robot learning.

Notation. Critic params $\theta$, actor params $\phi$, target networks $\bar\theta, \bar\phi$. Note this is the opposite of the lecture notes convention, where $\theta$ is the actor and $\phi$ the critic — the DQN/DDPG/TD3/SAC papers all use the convention below, so it's easier to read the code against it.

Background: Q-learning & DQN

What does $Q^\pi$ mean? For a fixed policy $\pi$, $Q^\pi(\mathbf{s}, \mathbf{a})$ is the expected discounted return if we start in state $\mathbf{s}$, take action $\mathbf{a}$ first, and follow $\pi$ from the next step onward:

$Q^\pi(\mathbf{s}, \mathbf{a}) = \mathbb{E}_{\pi,p} \left[ \left. \sum_{t=0}^{\infty} \gamma^t r_t \,\right|\, \mathbf{s}_0 = \mathbf{s},\; \mathbf{a}_0 = \mathbf{a} \right]$

Here $r_t$ is the reward after taking $\mathbf{a}_t$, and $\mathbf{a}_t \sim \pi(\cdot \mid \mathbf{s}_t)$ for $t \geq 1$; the expectation includes both policy randomness and environment transitions $p$. The superscript $\pi$ labels which policy we follow afterward, not a power. The first action $\mathbf{a}$ need not be the one $\pi$ would choose.

$Q^\pi$ vs. $Q$. Plain $Q$ is a generic action-value function; in the update rules below it denotes the current learned estimate. Evaluating a fixed $\pi$ aims for $Q \to Q^\pi$, while Q-learning aims for $Q \to Q^*$, the optimal action-value function. The current estimate need not equal either one yet.

Goal: learn the optimal action-value function $Q^*(\mathbf{s}, \mathbf{a})$, the expected discounted return starting from $\mathbf{s}$, taking $\mathbf{a}$, then acting optimally.

Bellman optimality equation:

$Q^*(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma \mathbb{E}_{\mathbf{s}^\prime \sim p(\cdot | \mathbf{s}, \mathbf{a})} \left[ \max_{\mathbf{a}^\prime} Q^*(\mathbf{s}^\prime, \mathbf{a}^\prime) \right]$

Tabular Q-learning update: evaluation & improvement

Given a transition $(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime)$ and reward $r$, separate the update into 2 ideas (assume $\mathbf{s}^\prime$ is nonterminal for now).

1. Policy evaluation — how good is the current policy? Hold a policy $\pi$ fixed and use its action probabilities at the next state to form a TD target:

$\begin{aligned} y^\pi &= r + \gamma \sum_{\mathbf{a}^\prime} \pi(\mathbf{a}^\prime \mid \mathbf{s}^\prime) Q(\mathbf{s}^\prime, \mathbf{a}^\prime) \\ Q(\mathbf{s}, \mathbf{a}) &\leftarrow Q(\mathbf{s}, \mathbf{a}) + \alpha \left[ y^\pi - Q(\mathbf{s}, \mathbf{a}) \right] \end{aligned}$

This changes the value estimate, not the policy. Repeated evaluation of a fixed $\pi$ aims to recover $Q^\pi$; one TD update is only a partial evaluation step.

2. Policy improvement — which action should we choose? Hold the current Q-table fixed and define a greedy policy (break ties arbitrarily):

$\pi_{\text{new}}(\mathbf{s}) \in \arg\max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a})$

Why $\in$ instead of $=$? Here $\arg\max$ means the set of all maximizing actions, while $\pi_{\text{new}}(\mathbf{s})$ is one chosen action from that set. For example, if left and right both have Q-value $5$ and stay has Q-value $2$, then $\arg\max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a}) = \{\text{left}, \text{right}\}$; choosing $\pi_{\text{new}}(\mathbf{s}) = \text{left}$ satisfies the $\in$ relation. Writing $\pi_{\text{new}}(\mathbf{s}) = \arg\max_{\mathbf{a}} Q(\mathbf{s}, \mathbf{a})$ is common shorthand when $\arg\max$ is understood to return a single action using a tie-breaking rule.

We write this greedy policy as a deterministic action map $\pi_{\text{new}}(\mathbf{s})$. In the evaluation formula above, $\pi(\mathbf{a} \mid \mathbf{s})$ instead denotes an action probability; a deterministic policy assigns probability $1$ to its chosen action and $0$ to the others.

This changes the policy, not the value estimate. With an exact $Q^\pi$, the policy improvement theorem guarantees that this greedy policy is at least as good as $\pi$.

Putting them together gives Q-learning. At each update, use the greedy policy $\pi_Q$ implied by the current table as the target policy. Its evaluation target becomes

$y^{\pi_Q} = r + \gamma Q(\mathbf{s}^\prime, \pi_Q(\mathbf{s}^\prime)) = r + \gamma \max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime)$

so the combined update is

$Q(\mathbf{s}, \mathbf{a}) \leftarrow Q(\mathbf{s}, \mathbf{a}) + \alpha \left[ \underbrace{r + \gamma \max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime)}_{\text{target}} - Q(\mathbf{s}, \mathbf{a}) \right]$

The $\max$ supplies the greedy improvement, and moving $Q$ toward the resulting target supplies a TD evaluation step. Q-learning interleaves these through Bellman optimality backups; it does not evaluate each fixed policy to convergence before improving it. It is also off-policy: the behavior policy that collected the transition can be exploratory (e.g., $\epsilon$-greedy), while the target policy is greedy.

DDPG makes these roles explicit in separate networks: the critic evaluates the actor's policy, and the actor improves its actions using the critic.

From a Q-table to a Q-network

When state spaces are large (e.g., images), we can't store a table, so approximate $Q$ w/ a NN $Q_\theta(\mathbf{s}, \mathbf{a})$ and regress on the squared TD error:

$\mathcal{L}(\theta) = \mathbb{E}_{(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime) \sim \mathcal{D}} \left[ \left( Q_\theta(\mathbf{s}, \mathbf{a}) - \left( r + \gamma \max_{\mathbf{a}^\prime} Q_\theta(\mathbf{s}^\prime, \mathbf{a}^\prime) \right) \right)^2 \right]$

Naive deep Q-learning is unstable, from 2 sources:

1. correlated samples: sequential transitions $(\mathbf{s}_t, \mathbf{s}_{t+1}, \mathbf{s}_{t+2}, \dots)$ are highly correlated
2. moving target: the target $r + \gamma \max_{\mathbf{a}^\prime} Q_\theta(\mathbf{s}^\prime, \mathbf{a}^\prime)$ shifts as we update $\theta$

DQN (Mnih et al., 2015) fixes these w/ 2 tricks.

Trick 1: experience replay

Store transitions in a replay buffer $\mathcal{D}$ (fixed size, FIFO) and sample random mini-batches for each gradient step.

+: breaks correlation btw consecutive samples
+: reuses data efficiently (each transition used multiple times)
+: reduces non-stationarity of the training data

Trick 2: target networks

Use a separate target network $Q_{\bar\theta}$ that updates slowly, so the regression label stops moving under us:

$\mathcal{L}(\theta) = \mathbb{E}_{(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime) \sim \mathcal{D}} \left[ \left( Q_\theta(\mathbf{s}, \mathbf{a}) - \underbrace{\left( r + \gamma \max_{\mathbf{a}^\prime} Q_{\bar\theta}(\mathbf{s}^\prime, \mathbf{a}^\prime) \right)}_{\text{computed w/ frozen } \bar\theta} \right)^2 \right]$

2 ways to update the target network:

method	update rule	typical value
hard update	$\bar\theta \leftarrow \theta$ every $N$ steps	$N = 10000$
soft update (Polyak)	$\bar\theta \leftarrow \tau \theta + (1 - \tau) \bar\theta$ each step	$\tau = 0.005$

The soft update is what DDPG/TD3/SAC all use: instead of a discontinuous jump every $N$ steps, the target drifts smoothly toward the online net. This matters more in the continuous-action setting, where the actor is chasing the critic's gradient and a sudden target jump can knock the policy off a good region.

Overestimation bias & double Q-learning

The $\max$ operator makes Q-learning overestimate Q-values. Formally, $\max$ and $\mathbb{E}$ don't commute:

$\mathbb{E} \left[ \max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime) \right] \geq \max_{\mathbf{a}^\prime} \mathbb{E} \left[ Q(\mathbf{s}^\prime, \mathbf{a}^\prime) \right]$

Using the same network to both select and evaluate the best action amplifies noise: whichever action happens to have a positive error gets picked, so the target inherits that error. Worse, the inflated target becomes the next regression label, so the bias compounds through bootstrapping.

Double DQN (van Hasselt et al., 2016) decouples the two: the online network selects, the target network evaluates.

$y = r + \gamma \, Q_{\bar\theta} \Big( \mathbf{s}^\prime, \underbrace{\arg\max_{\mathbf{a}^\prime} Q_\theta(\mathbf{s}^\prime, \mathbf{a}^\prime)}_{\text{selected by online network}} \Big)$

Exploration: $\epsilon$-greedy

$\mathbf{a} = \begin{cases} \text{random action} & \text{w/ prob } \epsilon \\ \arg\max_\mathbf{a} Q_\theta(\mathbf{s}, \mathbf{a}) & \text{w/ prob } 1 - \epsilon \end{cases}$

Common schedule: decay $\epsilon$ from $1.0$ to $0.01$ over the first $N$ env steps.

Full DQN algorithm

1. init replay buffer $\mathcal{D}$, Q-network $Q_\theta$, target network $Q_{\bar\theta} \leftarrow Q_\theta$
2. for each step:
   1. select action $\mathbf{a}$ using $\epsilon$-greedy w.r.t. $Q_\theta$
   2. execute $\mathbf{a}$, observe $r, \mathbf{s}^\prime$; store $(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime, r)$ in $\mathcal{D}$
   3. sample mini-batch from $\mathcal{D}$
   4. compute target $y = r + \gamma \max_{\mathbf{a}^\prime} Q_{\bar\theta}(\mathbf{s}^\prime, \mathbf{a}^\prime)$
   5. update $\theta$ by minimizing $(Q_\theta(\mathbf{s}, \mathbf{a}) - y)^2$
   6. update target network (hard or soft)

Implementation tip: terminal states

Store a done flag $d$ w/ each transition, $d = 1$ if $\mathbf{s}^\prime$ is terminal, else $d = 0$:

$y = r + \gamma (1 - d) \max_{\mathbf{a}^\prime} Q_{\bar\theta}(\mathbf{s}^\prime, \mathbf{a}^\prime)$

The $(1 - d)$ term makes sure we don't bootstrap from terminal states, where there is no future return. Every target below carries this factor.

---

DDPG (Deep Deterministic Policy Gradient)

DDPG adapts DQN's success (experience replay, target networks) into continuous action spaces using deterministic policies.

TODO: add picture — actor-critic gradient flow, $\nabla_\mathbf{a} Q$ passing through the actor.

Core theory: the DPG theorem

In discrete action spaces $\max_\mathbf{a} Q(\mathbf{s}, \mathbf{a})$ is an enumeration; in continuous spaces it is its own optimization problem at every step. Instead, train a deterministic actor $\mathbf{a} = \mu_\phi(\mathbf{s})$ to output a high-value action, and let the critic tell it which way to move. The actor learns an approximate maximizer; it does not compute an exact $\arg\max$.

For a fixed state, the differentiable path is $\phi \to \mu_\phi(\mathbf{s}) \to Q_\theta(\mathbf{s}, \mu_\phi(\mathbf{s}))$. Holding the critic parameters $\theta$ fixed, the chain rule gives

$\nabla_\phi Q_\theta(\mathbf{s}, \mu_\phi(\mathbf{s})) = \left( \frac{\partial \mu_\phi(\mathbf{s})}{\partial \phi} \right)^\top \left. \nabla_\mathbf{a} Q_\theta(\mathbf{s}, \mathbf{a}) \right|_{\mathbf{a} = \mu_\phi(\mathbf{s})}$

• $\nabla_\mathbf{a} Q_\theta(\mathbf{s}, \mathbf{a})$ — the critic's gradient w.r.t. the action. At the actor's current action, it points in the direction of the steepest local increase in the estimated Q-value: the coach says how to adjust the wrist (手腕怎么压).
• $\partial \mu_\phi(\mathbf{s}) / \partial \phi$ — the actor's Jacobian w.r.t. its parameters. It describes how changing each network parameter changes the output action: how the muscles control the wrist (怎么调整肌肉). For vector actions this is a matrix, so the transpose maps the critic's action-space gradient back into parameter space.

Multiplying the 2 gives the actor a parameter update direction. In the scalar case this is simply $\frac{\partial Q}{\partial a}\frac{\partial \mu}{\partial \phi}$: “教练指出动作往哪改，Actor 再把这个方向传回自己的参数。” Gradient ascent moves along this direction; minimizing the negative Q-value below implements the same update.

What the DPG theorem adds. The chain rule above only differentiates a fixed-state critic output. The deterministic policy gradient theorem (Silver et al., 2014) connects it to the expected long-term return: with the true critic $Q^{\mu_\phi}$, the appropriate discounted state-visitation weights under $\mu_\phi$, and standard differentiability assumptions, integrating this local gradient gives the policy's return gradient. We do not need to differentiate the environment dynamics or explicitly differentiate the state-visitation distribution. Determinism gives one action per state and a direct gradient path; differentiability is also required.

DDPG's practical approximation. Replace the true critic by $Q_\theta$ and sample states from the replay buffer. With $\theta$ and the replay sampling distribution fixed during the actor step, optimize the surrogate $J_{\mathcal{D}}(\phi) = \mathbb{E}_{\mathbf{s} \sim \mathcal{D}}[Q_\theta(\mathbf{s}, \mu_\phi(\mathbf{s}))]$:

$\nabla_\phi J_{\mathcal{D}} = \mathbb{E}_{\mathbf{s} \sim \mathcal{D}} \left[ \left( \frac{\partial \mu_\phi(\mathbf{s})}{\partial \phi} \right)^\top \left. \nabla_\mathbf{a} Q_\theta(\mathbf{s}, \mathbf{a}) \right|_{\mathbf{a} = \mu_\phi(\mathbf{s})} \right]$

This is the exact gradient of the replay surrogate, but generally an approximation to the true return gradient because the critic is learned and replay states come from past behavior policies. In code, freeze the critic's parameters during the actor update, while keeping the gradient through its action input; detaching the action or critic output would break the chain.

Critic loss (evaluation)

MSE against the target network:

$\begin{aligned} y &= r + \gamma (1 - d) Q_{\bar\theta}(\mathbf{s}^\prime, \mu_{\bar\phi}(\mathbf{s}^\prime)) \\ \mathcal{L}_Q(\theta) &= \mathbb{E} \left[ (Q_\theta(\mathbf{s}, \mathbf{a}) - y)^2 \right] \end{aligned}$

Actor loss (improvement)

Maximize the critic's Q-value estimation, minimized via the negative sign:

$\mathcal{L}_\mu(\phi) = - \mathbb{E} \left[ Q_\theta(\mathbf{s}, \mu_\phi(\mathbf{s})) \right]$

Target networks: soft update

Both the critic and the actor keep a target copy, updated by Polyak averaging every step rather than hard-copied every $N$:

$\begin{aligned} \bar\theta &\leftarrow \tau \theta + (1 - \tau) \bar\theta \\ \bar\phi &\leftarrow \tau \phi + (1 - \tau) \bar\phi \end{aligned}$

w/ $\tau \approx 0.005$. This is the single change that makes deep Q-learning workable in continuous spaces — without it the actor and critic chase each other and diverge.

Flaws

-: suffers from severe overestimation bias in Q-values
-: exploration is rigid, relying on hard-coded additive noise (e.g., Gaussian or Ornstein-Uhlenbeck) whose scale is a hyperparameter you must tune per env

---

TD3: 3 fixes for DDPG

TD3 (Fujimoto et al., 2018) keeps DDPG's structure and patches its failure modes. SAC borrows the first fix; the other 2 are worth knowing b/c they explain why the first one is needed.

1. Clipped double Q-learning. Keep 2 independent critics $Q_{\theta_1}, Q_{\theta_2}$ and take the min for the target:

$y = r + \gamma (1 - d) \min_{i=1,2} Q_{\bar\theta_i}(\mathbf{s}^\prime, \mathbf{a}^\prime)$

Double DQN's select/evaluate split is not enough here: in actor-critic the actor is trained through the critic, so the online and target nets stay highly correlated and the decoupling breaks down. Taking the min of 2 critics instead induces a deliberate underestimate. See why underestimation is the safer error below.

2. Target policy smoothing. Add clipped noise to the target action, so the critic can't exploit a sharp spurious peak:

$\mathbf{a}^\prime = \mu_{\bar\phi}(\mathbf{s}^\prime) + \text{clip}(\epsilon, -c, c), \quad \epsilon \sim \mathcal{N}(0, \sigma^2)$

This is a regularizer that enforces the prior that similar actions should have similar values.

3. Delayed policy updates. Update the actor (and the target nets) once every $k$ critic updates, typically $k = 2$. A policy trained against a high-variance critic amplifies the critic's error; letting the critic settle first reduces that.

---

SAC (Soft Actor-Critic)

SAC is built on maximum-entropy RL, optimizing for both maximum reward and maximum policy randomness to ensure robust exploration and prevent local optima.

From Q-learning to actor-critic

Problem w/ continuous actions: can't compute $\max_{\mathbf{a}^\prime} Q(\mathbf{s}^\prime, \mathbf{a}^\prime)$ exactly.
Solution: learn a policy (actor) $\pi_\phi(\mathbf{a} | \mathbf{s})$ alongside the Q-function (critic).

• critic $Q_\theta(\mathbf{s}, \mathbf{a})$: estimates expected return
• actor $\pi_\phi(\mathbf{a} | \mathbf{s})$: outputs actions, replacing the $\arg\max$

The maximum-entropy objective

SAC adds an entropy bonus to the usual RL objective:

$J(\pi) = \sum_t \mathbb{E} \left[ r(\mathbf{s}_t, \mathbf{a}_t) + \alpha \mathcal{H}(\pi(\cdot | \mathbf{s}_t)) \right]$

where $\mathcal{H}(\pi(\cdot | \mathbf{s})) = -\mathbb{E}_{\mathbf{a} \sim \pi(\cdot | \mathbf{s})} [\log \pi(\mathbf{a} | \mathbf{s})]$ is the conditional entropy.

Why entropy regularization?

+: encourages exploration — the policy doesn't collapse to deterministic too early
+: more robust, b/c it captures multiple good solutions instead of committing to one
+: better convergence in practice

The temperature $\alpha$ controls the exploration vs. exploitation tradeoff. Note the contrast w/ DDPG: exploration is now part of the objective rather than noise bolted on at action-selection time.

TODO: add picture — reward vs. entropy tradeoff as $\alpha$ varies.

Soft policy iteration: the 2 theoretical pillars

The whole algorithm is a practical approximation of soft policy iteration, which alternates 2 steps, each w/ its own guarantee.

Lemma 1 (soft policy evaluation). First fold the entropy bonus into the state value:

$V(\mathbf{s}) = \mathbb{E}_{\mathbf{a} \sim \pi} \left[ Q(\mathbf{s}, \mathbf{a}) - \alpha \log \pi(\mathbf{a} | \mathbf{s}) \right]$

Then the soft Bellman backup operator is

$\mathcal{T}^\pi Q(\mathbf{s}, \mathbf{a}) \triangleq r(\mathbf{s}, \mathbf{a}) + \gamma \, \mathbb{E}_{\mathbf{s}^\prime \sim p(\cdot | \mathbf{s}, \mathbf{a})} \left[ V(\mathbf{s}^\prime) \right]$

Substituting $V$ gives the form the critic actually regresses on:

$\mathcal{T}^\pi Q(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma \, \mathbb{E}_{\mathbf{s}^\prime \sim p, \, \mathbf{a}^\prime \sim \pi} \left[ Q(\mathbf{s}^\prime, \mathbf{a}^\prime) - \alpha \log \pi(\mathbf{a}^\prime | \mathbf{s}^\prime) \right]$

The lemma: repeatedly applying $\mathcal{T}^\pi$ to any bounded initial $Q$ converges to the true soft Q-value $Q^\pi$. The proof is the standard one — $\mathcal{T}^\pi$ is still a $\gamma$-contraction, w/ the entropy term riding along as part of an augmented reward $r_\pi(\mathbf{s}, \mathbf{a}) = r + \gamma\alpha\mathbb{E}_{\mathbf{s}^\prime}[\mathcal{H}(\pi(\cdot|\mathbf{s}^\prime))]$. Compare this against the SAC critic target below: they are the same expression, w/ $\bar\theta$ and the $\min$ as the practical approximations.

Lemma 2 (soft policy improvement). Update the policy by projecting it onto the Boltzmann distr of the current soft Q:

$\pi_{\text{new}} = \arg\min_{\pi^\prime \in \Pi} D_{KL} \left( \pi^\prime(\cdot | \mathbf{s}) \,\middle\|\, \frac{\exp\left( \frac{1}{\alpha} Q^{\pi_{\text{old}}}(\mathbf{s}, \cdot) \right)}{Z^{\pi_{\text{old}}}(\mathbf{s})} \right)$

The lemma: $Q^{\pi_{\text{new}}}(\mathbf{s}, \mathbf{a}) \geq Q^{\pi_{\text{old}}}(\mathbf{s}, \mathbf{a})$ for all $(\mathbf{s}, \mathbf{a})$ — monotone improvement. The restriction $\pi^\prime \in \Pi$ is what makes this a projection: over all distrs the argmin is the Boltzmann distr exactly, but we're confined to a tractable family, so we settle for its closest member.

Theorem (soft policy iteration). Alternating the 2 converges to the optimal maximum-entropy policy in $\Pi$. SAC is the function-approximation version: rather than running each step to convergence, it takes a single gradient step on each per env step.

The mathematical bridge: max-entropy = min KL divergence

Lemma 2 projects the policy onto $\exp(\frac{1}{\alpha}Q)/Z$ — but why that target? Because minimizing that KL divergence is mathematically identical to maximizing "expected Q-value $+$ entropy". The 2 formulations are the same optimization written 2 ways, and this is the equivalence the actor loss rests on.

Here is the step-by-step derivation:

1. The objective: the actor wants to find a policy $\pi$ that maximizes the expected Q-value and its own entropy:

   $\pi^* = \arg\max_\pi \mathbb{E}_{\mathbf{a} \sim \pi} \left[ Q(\mathbf{s}, \mathbf{a}) + \alpha \mathcal{H}(\pi) \right]$

   Since $\mathcal{H}(\pi) = \mathbb{E}_{\mathbf{a} \sim \pi} [-\log \pi(\mathbf{a} | \mathbf{s})]$, we can rewrite this as a minimization problem:

   $\pi^* = \arg\min_\pi \mathbb{E}_{\mathbf{a} \sim \pi} \left[ \alpha \log \pi(\mathbf{a} | \mathbf{s}) - Q(\mathbf{s}, \mathbf{a}) \right]$

   This expression is already the actor loss — everything that follows just shows it is a KL projection.

   Dividing the entire expression by $\alpha$ (which doesn't change the argmin location):

   $\pi^* = \arg\min_\pi \mathbb{E}_{\mathbf{a} \sim \pi} \left[ \log \pi(\mathbf{a} | \mathbf{s}) - \frac{1}{\alpha} Q(\mathbf{s}, \mathbf{a}) \right]$

2. The Boltzmann target: define a target prob distr $q(\mathbf{a} | \mathbf{s})$ where the prob of choosing an action is proportional to the exponential of its Q-value. $Z(\mathbf{s})$ is the normalizing constant (partition function):

   $q(\mathbf{a} | \mathbf{s}) = \frac{1}{Z(\mathbf{s})} \exp \left( \frac{1}{\alpha} Q(\mathbf{s}, \mathbf{a}) \right)$

   Taking the log of both sides gives us:

   $\frac{1}{\alpha} Q(\mathbf{s}, \mathbf{a}) = \log q(\mathbf{a} | \mathbf{s}) + \log Z(\mathbf{s})$

3. The substitution: substitute this $\frac{1}{\alpha} Q(\mathbf{s}, \mathbf{a})$ back into our minimization objective from step 1:

   $\pi^* = \arg\min_\pi \mathbb{E}_{\mathbf{a} \sim \pi} \left[ \log \pi(\mathbf{a} | \mathbf{s}) - \log q(\mathbf{a} | \mathbf{s}) - \log Z(\mathbf{s}) \right]$

   By the definition of KL divergence, $\mathbb{E}_{\mathbf{a} \sim \pi} [\log \pi - \log q]$ is exactly $D_{KL}(\pi \| q)$:

   $\pi^* = \arg\min_\pi \left[ D_{KL}(\pi(\cdot | \mathbf{s}) \| q(\cdot | \mathbf{s})) - \log Z(\mathbf{s}) \right]$

4. The conclusion: b/c $\log Z(\mathbf{s})$ only depends on the state and is a constant w.r.t. the policy $\pi$, we can ignore it during optimization. Thus, the objective perfectly simplifies to:

   $\pi^* = \arg\min_\pi D_{KL} \left( \pi(\cdot | \mathbf{s}) \,\middle\|\, \frac{\exp(\frac{1}{\alpha} Q(\mathbf{s}, \cdot))}{Z(\mathbf{s})} \right)$

   which is exactly Lemma 2's update.

Updating the actor to maximize "reward $+$ entropy" is mathematically equivalent to projecting the policy directly onto the Q-value's Boltzmann distr. Practically, the useful direction is step 1: the thing you implement is $\mathbb{E}[\alpha\log\pi - Q]$, and the KL reading is what tells you it's a principled projection rather than an arbitrary penalty.

What SAC changes relative to DDPG

If you have to name it in one breath: SAC changes the actor and the critic, one thing each.

1. The actor becomes stochastic — a reparameterized squashed Gaussian $\pi_\phi(\mathbf{a}|\mathbf{s})$ instead of DDPG's deterministic $\mu_\phi(\mathbf{s})$, w/ an entropy term $-\alpha\log\pi_\phi$ carried into both the target and the actor loss. Exploration moves into the objective; DDPG's hand-tuned additive noise is gone.
2. The critic becomes 2 critics w/ a $\min$ — clipped double-Q borrowed from TD3, to cancel DDPG's overestimation bias.

Everything else DDPG already had: replay buffer, Polyak-averaged target networks, and an actor trained by backprop through the critic.

Those 2 changes map exactly onto DDPG's 2 flaws — rigid exploration and overestimation. The comparison table breaks all of this down line by line.

The rest of this section works through the machinery each change needs: the reparameterization trick and the squashed Gaussian for change 1, clipped double-Q for change 2, and $\alpha$ auto-tuning as a convenience on top of change 1 rather than a change of its own.

Stochastic policy w/ reparameterization

The reparameterization trick bypasses the inability to backprop through random sampling. The network outputs deterministic $\mu_\phi$ and $\sigma_\phi$, and the randomness comes from external standard noise:

$\mathbf{a} = \tanh \left( \underbrace{\mu_\phi(\mathbf{s}) + \sigma_\phi(\mathbf{s}) \odot \epsilon}_{u} \right), \quad \epsilon \sim \mathcal{N}(0, I)$

The $\tanh$ squashes actions to $(-1, 1)$; the reparameterization allows backprop through sampling, since $\epsilon$ carries no parameters and the path $\phi \to \mu, \sigma \to \mathbf{a}$ is fully differentiable.

Is the Gaussian specific to SAC, or just a convenient choice?

Just a convenient choice. Nothing in Lemma 1 or Lemma 2 assumes a Gaussian — the ideal target is always the Boltzmann distr $\exp(Q/\alpha)/Z$, which is generally not Gaussian. Lemma 2 says the policy improves when we project onto the target within our chosen policy class $\Pi$, and the squashed Gaussian is simply a tractable $\Pi$.

What SAC actually requires of the policy class is 3 things:

1. reparameterizable, so the actor loss has a low-variance pathwise gradient
2. a closed-form $\log \pi(\mathbf{a} | \mathbf{s})$, b/c $\log \pi$ appears in all three losses
3. bounded support matching the action space

The Gaussian happens to satisfy all 3 cheaply. Other valid choices: a Categorical distr for discrete SAC (there you can drop reparameterization entirely and sum the entropy over actions exactly), or a mixture of Gaussians / normalizing flow when the action distr is genuinely multi-modal. What breaks the recipe is a policy w/ no tractable density — which is exactly the problem diffusion policies run into.

Clipped double-Q (from TD3)

To reduce overestimation, SAC uses 2 Q-networks and takes the minimum. Combined w/ the entropy term, the target is:

$y = r + \gamma (1 - d) \left[ \min_{i=1,2} Q_{\bar\theta_i}(\mathbf{s}^\prime, \mathbf{a}^\prime) - \alpha \log \pi_\phi(\mathbf{a}^\prime | \mathbf{s}^\prime) \right], \quad \mathbf{a}^\prime \sim \pi_\phi(\cdot | \mathbf{s}^\prime)$

Note $\mathbf{a}^\prime$ is a fresh sample from the current policy, not the action stored in the replay buffer — this is the standard off-policy correction.

Why does taking the min suppress overestimation?

Because the 2 error directions are not symmetric in their consequences.

Write each critic as truth plus noise, $Q_{\theta_i} = Q^* + \varepsilon_i$. The $\max$ in the target systematically selects whichever action drew a positive $\varepsilon$, so $\mathbb{E}[\max] \geq \max \mathbb{E}$ and the target is biased upward. Taking $\min_{i=1,2}$ over 2 critics pulls in the opposite direction — $\mathbb{E}[\min(Q_1, Q_2)] \leq \min(\mathbb{E}[Q_1], \mathbb{E}[Q_2])$ — so the 2 biases partly cancel.

The deeper reason to prefer erring low is that the 2 kinds of error have very different dynamics under bootstrapping:

• an overestimate is self-reinforcing: an action that looks spuriously good gets selected by the $\max$/actor, its inflated value becomes the next regression target, and the error propagates backward through the whole value function
• an underestimate is self-correcting: an action that looks spuriously bad simply stops being taken, so the wrong value never gets amplified into other targets

So a deliberate pessimistic bias is cheap insurance. The cost is real but bounded — the agent can be slow to discover genuinely good actions it currently underrates.

Automatic temperature tuning

Instead of manually setting $\alpha$, learn it by minimizing

$\mathcal{L}_\alpha = \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \, \mathbf{a} \sim \pi_\phi(\cdot | \mathbf{s})} \left[ -\alpha \left( \log \pi_\phi(\mathbf{a} | \mathbf{s}) + \bar{\mathcal{H}} \right) \right]$

where $\bar{\mathcal{H}}$ is the target entropy, typically $-\dim(\mathcal{A})$.

The gradient is $\partial \mathcal{L}_\alpha / \partial \alpha = \mathcal{H} - \bar{\mathcal{H}}$, so if the current entropy $\mathcal{H} < \bar{\mathcal{H}}$, $\alpha$ increases to force exploration, and vice versa. It's a feedback controller holding entropy at a setpoint.

Implementation tip: optimize $\log \alpha$ instead of $\alpha$ directly to guarantee $\alpha > 0$.

SAC loss functions

1. Critic loss, for each $Q_{\theta_i}, i \in \{1, 2\}$ (the squared TD error):

   $\mathcal{L}_Q(\theta_i) = \mathbb{E}_{(\mathbf{s}, \mathbf{a}, \mathbf{s}^\prime) \sim \mathcal{D}} \left[ (Q_{\theta_i}(\mathbf{s}, \mathbf{a}) - y)^2 \right]$

2. Actor loss (derived directly from the KL divergence proof):

   $\mathcal{L}_\pi(\phi) = \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \, \mathbf{a} \sim \pi_\phi(\cdot | \mathbf{s})} \left[ \alpha \log \pi_\phi(\mathbf{a} | \mathbf{s}) - \min_{i=1,2} Q_{\theta_i}(\mathbf{s}, \mathbf{a}) \right]$

3. Temperature loss:

   $\mathcal{L}_\alpha = \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \, \mathbf{a} \sim \pi_\phi(\cdot | \mathbf{s})} \left[ -\alpha \left( \log \pi_\phi(\mathbf{a} | \mathbf{s}) + \bar{\mathcal{H}} \right) \right]$

Note $\mathbf{a}$ in the actor and temperature losses is a fresh reparameterized sample, while $\mathbf{a}$ in the critic loss comes from the buffer.

---

DDPG vs. TD3 vs. SAC

	DDPG	TD3	SAC
policy	deterministic $\mu_\phi(\mathbf{s})$	deterministic $\mu_\phi(\mathbf{s})$	stochastic $\pi_\phi(\mathbf{a} \mid \mathbf{s})$
# critics	1	2, take $\min$	2, take $\min$
exploration	additive noise at act time	additive noise at act time	entropy term in the objective
target action	$\mu_{\bar\phi}(\mathbf{s}^\prime)$	$\mu_{\bar\phi}(\mathbf{s}^\prime) + \text{clipped noise}$	$\mathbf{a}^\prime \sim \pi_\phi(\cdot \mid \mathbf{s}^\prime)$
entropy term	none	none	$-\alpha \log \pi_\phi$ in target & actor loss
actor update	every step	every $k$ critic steps	every step
target update	Polyak, $\tau \approx 0.005$	Polyak, $\tau \approx 0.005$	Polyak, $\tau \approx 0.005$
key hyperparameter	noise scale	noise scale, $k$	$\bar{\mathcal{H}}$ (and $\alpha$ auto-tuned)

---

The squashed Gaussian policy & math corrections

The problem: standard Gaussian distrs have infinite bounds $[-\infty, +\infty]$. If an environment clips an out-of-bound action (e.g., $2.5$ clipped to $1.0$), the critic evaluates an action that never actually occurred, ruining the Q-value.

The fix: SAC passes the sampled action $u \sim \mathcal{N}(\mu, \sigma^2)$ through a $\tanh$ to strictly bound it to $(-1, 1)$:

$\mathbf{a} = \tanh(u)$

TODO: add picture — how $\tanh$ squashes the Gaussian density near the boundary.

The derivation of the Jacobian correction: b/c $\tanh$ squashes the prob density, we must use the change of variables formula to correct the entropy calculation:

$\log p(\mathbf{a}) = \log p(u) - \log \left| \frac{d\mathbf{a}}{du} \right|$

The derivative of $\mathbf{a} = \tanh(u)$ w.r.t. $u$ is exactly $1 - \tanh^2(u)$. Plugging this in yields the SAC correction term:

$\log \pi(\mathbf{a} | \mathbf{s}) = \log p(u | \mathbf{s}) - \sum_i \log(1 - \tanh^2(u_i))$

Implementation pitfalls

NaN / gradient explosion. If $u$ is large, $\tanh(u) \approx 1$, making $1 - \tanh^2(u)$ effectively $0$. Computing $\log(0)$ gives -inf or NaN.

A common patch is to add a tiny epsilon, $\log(1 - \tanh^2(u) + \epsilon)$ w/ $\epsilon = 10^{-6}$. But this is only a band-aid, and it biases the log-prob. There is an exact numerically stable form — from $1 - \tanh^2 u = 4 / (e^u + e^{-u})^2$:

$\log(1 - \tanh^2(u)) = 2 \left( \log 2 - u - \text{softplus}(-2u) \right)$

softplus is stable for both signs of its argument, so this needs no epsilon and introduces no bias. Prefer it.

Clamp $\log \sigma$. The actor outputs $\log \sigma$, not $\sigma$; clamp it to roughly $[-20, 2]$. Unbounded $\log \sigma$ is the other main upstream source of NaN: too large blows up the sample $u$ (feeding straight into the pitfall above), too small makes $\log \pi$ diverge.

Optimize $\log \alpha$. Same reasoning as $\sigma$: parameterize the temperature in log space so $\alpha > 0$ holds by construction.

Rescale to the real action bounds. $\tanh$ gives $(-1, 1)$; for an env w/ bounds $[\text{low}, \text{high}]$, apply $\mathbf{a} = c \cdot \tanh(u) + b$, where $c = (\text{high} - \text{low}) / 2$ (action_scale) and $b = (\text{high} + \text{low}) / 2$ (action_bias). The Jacobian term then picks up a constant $\log c$ per dim, which doesn't affect gradients but does shift the reported entropy — so make sure $\bar{\mathcal{H}}$ is set in the same units.

Deterministic evaluation. At test time use $\mathbf{a} = \tanh(\mu_\phi(\mathbf{s}))$ rather than sampling. The entropy bonus is a training-time exploration device, not part of the behavior you want to deploy.

---

Modern applications: is squashed Gaussian still used?

Standard continuous control: yes. For traditional physics simulations (e.g., Mujoco, robotic arms, quadrupeds), the squashed Gaussian policy remains the standard default baseline, and SAC is still the reference off-policy algorithm.

VLA (Vision-Language-Action) models: mostly no — but the reason is not simply "VLAs are discrete". What actually happens is that the unimodal Gaussian head gets replaced, and there are 2 distinct families of replacement:

• action tokenization (discrete): VLAs built directly on an LLM backbone, e.g. RT-2 and OpenVLA, discretize each continuous action dim into bins (typically 256 tokens per dim, covering end-effector delta pose $+$ gripper) and emit them as ordinary tokens w/ a categorical distr. This inherits the LLM's next-token machinery unchanged.
• continuous generative heads: the trend in recent robot policies goes the other way — back to continuous actions, but w/ an expressive generative model instead of a Gaussian. Diffusion policies frame action generation as a denoising process, and the $\pi_0$-style flow matching action experts attach a continuous action head to a VLM backbone. Neither one tokenizes.

The common thread is multi-modality: a squashed Gaussian can only express one mode per state, so it averages over distinct valid behaviors (go left vs. go right around an obstacle) into a bad middle action. Discretization and diffusion/flow are 2 different ways to buy multi-modality back.

The tradeoff is the one flagged in is the Gaussian specific to SAC: both replacements give up a cheap closed-form $\log \pi(\mathbf{a} | \mathbf{s})$ — categorical keeps it but only over a discretized space, while diffusion/flow have no tractable density at all. That is exactly why RL post-training on top of these policies is harder than plugging in SAC, and why methods in this space lean on value-weighted or advantage-conditioned supervised objectives rather than an entropy-regularized actor loss.