RL Background for VLA Post-training

Explain why Vision-Language-Action (VLA) post-training needs more than imitation learning, and why real-world robot RL is both an algorithm problem and a systems problem.

VLA RL Background

This note is intended as conceptual background before studying methods such as RECAP / π*_0.6 and Learning While Deploying (LWD).

Document boundary:

• This file owns the reusable foundations.
• 02_recap_method.md owns RECAP's advantage-conditioned policy extraction.
• 03_lwd_main.md owns LWD's fleet learning system, DIVL, and QAM.

The central question is:

A pretrained VLA policy can already imitate demonstrations. Why do we still need reinforcement learning, human-in-the-loop corrections, and real robot experience?

Short answer:

• Imitation learning gives a strong initial policy.
• DAgger / HIL corrections help with states induced by the learned policy.
• RL / value learning extracts signal from success, failure, partial progress, and practice data.
• Real-world interaction samples the true deployment distribution.
• Embodied RL systems are much more complex than standard digital RL pipelines.

---

1. Human learning analogy

A useful analogy is how humans learn a physical skill.

Usually we do not learn only by watching an expert once. We learn through a combination of:

1. Instruction
   Someone teaches the basics: what strategy works, what mistakes are common, and what techniques matter.

2. Coaching
   A teacher watches us practice, corrects mistakes, and points out better ways to act.

3. Practice
   We repeatedly try the task ourselves, fail, recover, and gradually make the behavior reliable.

Robot policy post-training follows a similar pattern:

Human learning	Robot learning analogue
Instruction	demonstrations / SFT / behavior cloning
Coaching	human interventions / corrections / DAgger-style data
Practice	robot rollouts / successes / failures / RL experience

The key idea is:

Demonstrations teach what good behavior looks like, but practice teaches how behavior actually works under real physical interaction.

TODO: Insert/redraw motivation figure about offline learning vs online interaction.
Suggested filename: figs/offline_vs_online_learning.png.

---

2. Why VLA needs RL

A pretrained VLA policy is usually imitation-first. It learns from demonstrations, teleoperation data, or large robot datasets.

This gives a powerful starting point, but it has several limitations.

2.1 Demonstration data is expensive

Robot demonstrations usually require human effort:

• teleoperation,
• kinesthetic teaching,
• puppeteering,
• scripted collection,
• or expert reset and supervision.

This makes data expensive and hard to scale.

For long-horizon manipulation tasks, a single demonstration may take minutes. Collecting enough demonstrations to cover object layouts, recovery cases, and rare failures can become impractical.

2.2 Imitation is limited by demonstration coverage

A behavior-cloned policy can only learn from what the dataset shows.

If the dataset mostly contains clean successful trajectories, the policy may not know:

• what to do after a failed grasp,
• how to recover from a displaced object,
• when to retry,
• how to handle partial progress,
• how to avoid repeated mistakes,
• or how to react to deployment-time distribution shifts.

The policy may look strong on familiar states but brittle when it drifts outside the demonstration distribution.

2.3 Deployment reveals failures absent from the dataset

In real deployment, the policy encounters:

• new object poses,
• new object instances,
• clutter,
• lighting changes,
• camera calibration noise,
• gripper slip,
• contact-rich dynamics,
• ambiguous instructions,
• user-specific preferences,
• and long-tail failure modes.

These are not just “more data points”; they are often exactly the cases that determine whether a robot is deployment-ready.

2.4 RL provides a way to learn from experience

Reinforcement learning gives a framework for using outcomes:

attempted behavior
  -> success / failure / partial progress
  -> value estimate
  -> policy improvement

Unlike pure imitation learning, RL can use both successful and failed trials.

Failed robot rollouts are not useless. They tell us:

• which actions are bad,
• where the policy tends to drift,
• which states are hard,
• what recovery attempts look like,
• and which partial progress states are promising.

So the practical goal is not to replace imitation learning with RL from scratch. The recipe is:

pretrained VLA
  + real robot data
  + human corrections
  + off-policy / offline RL updates
  = stronger deployment policy

---

3. Imitation Learning

3.1 Behavior Cloning / SFT

Given expert demonstrations:

$\mathcal D_{\text{expert}} = \{(s_i, a_i)\}_{i=1}^N$

behavior cloning trains a policy to assign high probability to the expert action:

$\max_\theta \mathbb E_{(s,a)\sim \mathcal D_{\text{expert}}} [ \log \pi_\theta(a|s) ]$

Equivalently, it minimizes negative log likelihood:

$\mathcal L_{\text{BC}}(\theta) = - \mathbb E_{(s,a)\sim \mathcal D_{\text{expert}}} [ \log \pi_\theta(a|s) ]$

For deterministic continuous actions, this is often written as supervised regression:

$\min_\theta \mathbb E_{(s,a)\sim \mathcal D_{\text{expert}}} [ \|a-\pi_\theta(s)\|^2 ]$

In VLA training, behavior cloning is often called Supervised Fine-Tuning (SFT).

3.2 Expressive policy distributions

Simple regression can fail when the action distribution is multimodal.

For example, in a driving-like setting, two different demonstrators may take different but valid actions:

state: obstacle ahead
demo 1: go left
demo 2: go right

A simple L2 regression policy may average these actions and output an invalid middle action.

This is why modern robot policies often use expressive generative action models:

• Gaussian mixture models,
• discretized autoregressive actions,
• diffusion policies,
• flow matching policies.

Instead of predicting only one mean action, the policy models:

$p_\theta(a|s)$

This is crucial for VLA policies, where the same instruction and observation may admit multiple valid action chunks.

TODO: Insert figure: multimodal demonstrations and mean-action failure.
Suggested filename: figs/multimodal_bc_mean_failure.png.

TODO: Insert figure: generative policies for continuous actions.
Suggested filename: figs/generative_policy_types.png.

3.3 Why imitation is a strong baseline

Imitation learning has major advantages:

• It is simple and stable.
• It does not require reward design.
• It does not require unsafe online exploration.
• It can leverage large-scale human or robot datasets.
• It gives a capable initial policy for later post-training.

This is why VLA pipelines usually start with SFT / BC.

3.4 Why imitation is not enough

The main limitation is that imitation learning treats data as action labels.

It asks:

What action did the expert take here?

But it does not directly ask:

Did this action lead to success?
Was this action better than another possible action?
Was this recovery useful?
Did this partial progress matter later?

So imitation learning is a strong baseline, but not a complete framework for learning from experience.

---

4. Compounding errors and HIL corrections

4.1 Covariate shift

In ordinary supervised learning, prediction errors do not change the next input.

In robot control, actions affect future states:

$s_t \xrightarrow{\pi_\theta} a_t \xrightarrow{\mathcal T} s_{t+1}$

If the policy makes a small mistake, it may enter states that were rare or absent in the expert dataset.

So the state distribution under the learned policy differs from the expert state distribution:

$p_{\pi_\theta}(s) \neq p_{\text{expert}}(s)$

This is covariate shift.

4.2 Compounding errors

Over a long horizon, small errors accumulate.

small action error
  -> state drift
  -> less familiar observation
  -> larger action error
  -> more drift
  -> failure

This is the classic compounding error problem in imitation learning.

TODO: Insert figure: compounding errors / covariate shift.
Suggested filename: figs/compounding_errors.png.

4.3 DAgger

Dataset Aggregation (DAgger) addresses this by collecting expert labels on states visited by the learned policy.

Algorithmically:

1. Roll out the current policy $\pi_\theta$.
2. Observe the states it actually visits.
3. Ask the expert for the correct action at those states.
4. Add these labeled states to the dataset.
5. Retrain the policy.

In notation:

$\mathcal D \leftarrow \mathcal D \cup \{(s'_i,a^*_i)\}$

where $s'_i$ are states visited by the learned policy and $a^*_i$ are expert corrections.

This makes the training distribution closer to the policy-induced state distribution.

4.4 Human-Gated DAgger / HIL interventions

In robotics, querying an expert action at every state can be inconvenient.

A more practical interface is Human-Gated DAgger (HG-DAgger) or human-in-the-loop (HIL) intervention:

1. The policy starts executing.
2. A human watches.
3. When the policy is about to fail, the human intervenes.
4. The intervention segment is added to the dataset.
5. The policy is retrained.

This is useful because it collects data exactly around failure-prone states.

TODO: Insert figure: DAgger / HG-DAgger correction loop.
Suggested filename: figs/dagger_hil_loop.png.

4.5 What correction data fixes

HIL correction data is useful because it teaches:

• how to recover,
• what the policy should have done in a failure state,
• how to handle policy-induced states,
• and which mistakes occur in deployment.

4.6 What correction data does not fix

DAgger-style methods are still mostly imitation learning. They mainly use corrections as action labels:

$\min_\theta - \log \pi_\theta(a^*|s)$

This does not fully use:

• failed autonomous rollouts,
• partial progress,
• sparse terminal rewards,
• differences between good and bad attempts,
• or long-horizon credit assignment.

So HIL corrections help, but we still want value learning / RL to extract richer signals from experience.

---

5. Why real-world RL, not only sim RL?

A natural question is:

If RL is expensive and risky on real robots, why not train everything in simulation?

Simulation is extremely useful, but for VLA robot post-training it is usually not enough by itself.

5.1 Simulation cannot fully match deployment

A VLA policy deployed in the real world faces messy variation:

• lighting changes,
• camera noise,
• object pose variation,
• clutter,
• reflective or transparent objects,
• gripper slip,
• latency,
• hardware wear,
• user instruction variation,
• and long-tail failures.

Simulation can randomize many of these factors, but it is hard to cover the full deployment distribution.

In simulation, we choose the distribution.
In deployment, the world chooses the distribution.

5.2 Contact-rich manipulation is hard to simulate

Many manipulation tasks depend on subtle physical details:

• friction,
• compliance,
• grasp stability,
• pushing and sliding,
• liquid behavior,
• tool contact,
• object deformation,
• container opening,
• collision with clutter.

For long-horizon tasks, small contact errors can accumulate:

slightly bad grasp
  -> object rotates
  -> next subtask becomes harder
  -> policy enters unfamiliar state
  -> failure

These are exactly the kinds of failures real-world deployment reveals.

5.3 VLA failures are semantic + physical

For VLA policies, failures are not only low-level dynamics.

A policy may fail because it misunderstands:

• which object the instruction refers to,
• whether a subtask is complete,
• where an item should be placed,
• when to retry,
• which object is blocking the next step,
• or how to recover from an earlier mistake.

So the real deployment problem combines:

real images
+ real language
+ real objects
+ real robot dynamics
+ real failures

Simulation can help, but real-world interaction samples the actual target distribution.

5.4 Real-world RL learns from the target distribution

The strongest reason to do real-world RL is simple:

The best data for deployment is deployment data.

Real-world rollouts reveal the failures the current policy actually makes.

Post-training should learn not only ideal behavior, but also:

• where the policy is weak,
• what states it drifts into,
• which failures are recoverable,
• what human interventions look like,
• which partial progress states are valuable,
• and which retries are useful.

5.5 Not real-world RL from scratch

Real-world RL is valuable, but training from scratch on real robots is usually impractical.

Problems include:

• unsafe exploration,
• expensive robot time,
• hardware wear,
• sparse rewards,
• long-horizon credit assignment,
• human reset cost,
• and low sample efficiency.

So modern robot post-training usually means:

pretrained VLA / BC policy
  + offline real data
  + real robot rollouts
  + human interventions
  + off-policy RL updates

not:

random policy + real robot RL from scratch

5.6 Simulation is still useful

Simulation can help with:

• pretraining,
• debugging algorithms,
• cheap exploration,
• safety testing,
• generating diverse initial behaviors,
• and testing controlled hypotheses.

A practical recipe is:

simulation:
  cheap exploration, debugging, pretraining

offline real data:
  demonstrations, historical rollouts, play data

online real data:
  deployment distribution, real failures, interventions

offline-to-online RL:
  reuse all data to improve the policy

So the question is not:

sim RL or real-world RL?

but:

how much can we learn in sim,
and when do we need real-world data to close the gap?

TODO: Insert/redraw simulation vs real-world comparison table.
Suggested filename: figs/sim_vs_real_rl.png.

---

6. Online, off-policy, and offline RL

A useful way to understand robot RL is through dataflow.

TODO: Insert/redraw left column from RLinf talk: on-policy RL vs off-policy RL vs offline RL.
Suggested filename: figs/rl_dataflow_modes.png.

6.1 On-policy RL

The current policy interacts with the environment, collects trajectories, and updates itself using that fresh data.

policy π_k
  -> environment
  -> trajectories
  -> update
  -> policy π_{k+1}

This is conceptually clean, but it is expensive for robots because every update needs fresh real-world interaction.

Typical on-policy methods include PPO-style algorithms.

6.2 Off-policy RL

A behavior policy collects data into a replay buffer. The learner samples from the replay buffer and updates another policy or critic.

behavior policy
  -> environment
  -> replay buffer
  -> learner
  -> target policy

This is more data-efficient because experience can be reused many times.

For robotics, this is attractive because each real trajectory is expensive.

6.3 Offline RL

In offline RL, the dataset is fixed during training.

static dataset
  -> trainer
  -> deployed policy

The policy does not interact with the environment during training.

This is attractive for robotics because:

• data collection can be separated from training,
• training can reuse existing logs,
• unsafe exploration can be avoided,
• and the robot does not need to be online during every gradient step.

But it also creates a hard problem: the policy must improve from a fixed dataset without trying new actions.

6.4 Why robot RL wants replay reuse

Robot data is expensive. We want to reuse:

• demonstrations,
• old policy rollouts,
• failed attempts,
• human interventions,
• play data,
• and online deployment data.

This naturally points toward offline / off-policy RL rather than pure on-policy RL.

6.5 Offline RL problem: OOD action maximization

Naive Q-learning uses the Bellman optimality backup:

$Q(s,a) \leftarrow r(s,a) + \gamma \max_{a'}Q(s',a')$

The problem in offline RL is the max:

$\max_{a'}Q(s',a')$

The critic may assign high values to actions that are outside the dataset. The max operator will select these out-of-distribution actions, producing overestimated targets.

Failure mode:

unreliable Q on OOD action
  -> max selects OOD action
  -> overestimated TD target
  -> critic learns inflated value
  -> even worse max later

For robot VLA policies, this is especially dangerous because actions are high-dimensional continuous action chunks.

This motivates offline RL methods that avoid explicit maximization over unsupported actions.

---

7. Why VLA RL is also a systems problem

Real-world VLA RL is not only about choosing a learning objective. It is also a systems problem.

A practical embodied RL setup may involve:

• robot-side policy inference,
• physical environment interaction,
• replay collection and storage,
• centralized or cloud-based training,
• checkpoint deployment,
• optional simulation,
• human-in-the-loop interventions,
• edge-cloud communication,
• and GPU/CPU scheduling.

This is much messier than the clean loop often assumed in algorithm papers.

7.1 Dataflow differs across RL algorithms

Different RL algorithms require different dataflows:

on-policy RL:
  policy -> environment -> immediate update

off-policy RL:
  behavior policy -> replay buffer -> learner -> target policy

offline RL:
  static dataset -> trainer -> deployed policy

A flexible embodied RL system must support these different modes.

7.2 Embodied RL components are heterogeneous

In simulation, rollout, training, and inference can often run inside a controlled software stack.

In real-world robot learning, the environment is physical, slow, asynchronous, and expensive. The system must coordinate robots, humans, networks, storage, and learners.

A real robot fleet does not behave like a batch of GPU environments.

7.3 Systems bottlenecks

Existing scheduling modes often create tradeoffs:

• high component-switching overhead,
• poor resource flexibility,
• expensive idle time,
• communication bottlenecks,
• slow robot reset,
• and difficulty coordinating HIL interventions.

Therefore, scalable embodied RL requires not only good algorithms, but also systems infrastructure.

TODO: Insert/redraw middle and right columns from RLinf talk: heterogeneous embodied RL components and scheduling bottlenecks.
Suggested filename: figs/embodied_rl_system_complexity.png.

---

8. Compared with LLM RL

LLM RL is also computationally expensive, but the environment/dataflow is usually simpler than embodied real-world RL.

Classic RLHF is often presented as PPO-style optimization or related reward/preference optimization. Regardless of the exact optimizer, the rollout loop is mostly digital:

prompt
  -> model response
  -> reward model / verifier / preference signal
  -> policy update

The environment is usually the model plus a reward model, verifier, or human preference pipeline. Rollouts can be massively parallelized on GPU clusters. Failed samples usually do not damage hardware.

Embodied VLA RL is different:

real sensor state
  -> policy action
  -> physical robot execution
  -> real environment changes
  -> possible failure / collision / human intervention
  -> replay upload
  -> learner update
  -> redeployment

Aspect	LLM RL	Embodied VLA RL
Environment	digital prompts / reward model / verifier	physical world
Rollout cost	mostly GPU time	robot time, reset cost, hardware risk
Parallelism	easier to scale on GPU clusters	limited by robot fleet and real-time execution
Failure cost	bad sample text	failed task, collision, object damage, reset
State	text / tokens	images, proprioception, language, physical scene
Action	tokens	continuous high-dimensional action chunks
Dataflow	relatively centralized	robot actors + replay + learner + redeployment
Human role	preference labels / reward data	teleop, intervention, reset, safety monitoring

The point is not that LLM RL is easy. It is that embodied RL is tightly coupled to physical systems.

This is one reason methods like LWD should be viewed as both algorithmic and systems contributions.

---

9. From RL to VLA post-training methods

Modern VLA post-training methods combine imitation, correction, and RL-style value learning.

9.1 Data types

Data type	Who generates it?	What it teaches
Demonstration	expert / human	clean successful behavior
Rollout	current or old policy	what the policy actually does
Intervention	human takes over during failure	recovery / correction
Play data	human-guided exploration	failure-adjacent coverage
Practice data	broad umbrella	successes, failures, partial progress

9.2 Value learning extracts signal from experience

A value function estimates future return:

$V^\pi(s) = \mathbb E_\pi \left[ \sum_{t=0}^\infty \gamma^t r_t \mid s_0=s \right]$

A Q-function estimates the return of first taking action $a$:

$Q^\pi(s,a) = \mathbb E_\pi \left[ \sum_{t=0}^\infty \gamma^t r_t \mid s_0=s,a_0=a \right]$

From these, we can define advantage:

$A(s,a)=Q(s,a)-V(s)$

Advantage tells us whether an action is better or worse than expected for that state.

9.3 Advantage-based policy improvement

A common offline RL idea is to imitate good dataset actions more strongly.

For example, Advantage-Weighted Regression (AWR) uses:

$\mathcal L_{\text{AWR}}(\theta) = - \mathbb E_{(s,a)\sim\mathcal D} \left[ \exp(A(s,a)/\lambda) \log \pi_\theta(a|s) \right]$

High-advantage actions get larger weights.

BC:
  imitate all dataset actions equally

AWR:
  imitate high-advantage actions more

Aside: KL view of AWR

AWR can also be viewed as approximately solving a KL-constrained policy improvement problem: improve expected value, but keep the new policy close to the behavior policy.

$\pi_{\text{new}} = \arg\max_\pi \mathbb E_{a\sim\pi(\cdot|s)} [ Q(s,a) ] \quad \text{s.t.} \quad D_{\mathrm{KL}}(\pi\|\pi_\beta) < \epsilon$

The solution has an exponential-advantage weighting form, so AWR looks like behavior cloning with weights $\exp(A(s,a)/\lambda)$.

9.4 Bridge to RECAP and LWD

RECAP and LWD can be understood as two different ways to turn experience into VLA policy improvement.

RECAP:
  learn value / advantage
  -> condition the VLA policy on high advantage

LWD:
  learn critic / value from fleet replay
  -> use critic gradients to update a flow-based policy via QAM

A rough spectrum:

Method	Main signal	Policy extraction
BC / SFT	expert actions	imitate demonstrations
DAgger / HIL	corrective actions	imitate corrections
AWR / IQL-style	advantage values	weighted BC
RECAP	advantage labels/conditions	advantage-conditioned policy
LWD	critic gradients + distributional value	QAM for flow policy

---

10. Takeaways

The main ideas:

1. Imitation learning is necessary but not sufficient.
   It gives a strong initial VLA policy, but it is limited by dataset coverage.

2. DAgger / HIL corrections address covariate shift.
   They teach the policy how to act in states induced by its own mistakes.

3. RL extracts more signal from experience.
   Successes, failures, partial progress, and recovery attempts can all inform value learning.

4. Real-world RL matters because deployment data is the target distribution.
   Simulation is useful, but cannot fully replace real physical interaction.

5. Offline / off-policy RL is attractive for robots because data is expensive.
   We want to reuse demonstrations, rollouts, interventions, and play data.

6. Naive offline Q-learning is dangerous.
   Explicit $\max_a Q(s,a)$ can select out-of-distribution actions.

7. Embodied RL is also a systems problem.
   Real robot learning requires coordinating inference, replay, training, humans, edge/cloud communication, and redeployment.

8. RECAP and LWD are post-training methods built on these ideas.
   RECAP uses advantage-conditioned policies; LWD extends the story to fleet-scale offline-to-online RL with DIVL and QAM.