RECAP in π*_0.6

Explain RECAP / $\pi^*_{0.6}$ as a concrete VLA post-training method: how it uses robot experience, corrections, value learning, and advantage-conditioned policy extraction.

Scope of this note:

• 01_vla_rl_background.md owns the general background: BC/SFT, DAgger/HIL, replay, offline RL, $Q$, $V$, and advantage.
• This file keeps only the local definitions needed to understand RECAP.
• 03_lwd_main.md owns the detailed IQL $\rightarrow$ DIVL and QAM story; here LWD appears only as a comparison point.

Minimal local glossary:

Term	Meaning in this note
SFT / BC	supervised imitation of demonstration actions
HIL / intervention	human correction when the policy enters a bad state
Experience	demonstrations, autonomous rollouts, failures, successes, corrections
Advantage	$A(s,a)=Q(s,a)-V(s)$, a label for whether an action was better or worse than expected
Policy extraction	turning learned value information into an improved VLA policy

---

0. RECAP in one sentence

RECAP stands for:

RL with Experience and Corrections via Advantage-conditioned Policies

The high-level idea is:

Start from a pretrained VLA, collect real robot experience and human corrections, learn value / advantage signals from that data, then train the policy to generate actions conditioned on “good” advantage.

In short:

experience + corrections
        ↓
value / advantage learning
        ↓
advantage-conditioned policy
        ↓
condition on high advantage at inference

RECAP is important because it is a direct prior method to LWD. It already tries to answer:

How can a pretrained VLA improve from real-world deployment data, not just demonstrations?

TODO: Insert/redraw a one-page RECAP overview.
Suggested filename: figs/recap_overview.png.
Source to redraw from: recent attachment /mnt/data/fbb7f5d1-edac-466f-adb2-3f91ab0ccc79.png, the English screenshot titled roughly “π*_0.6 / RECAP — High-Level Understanding”. Use its high-level framing, not the exact screenshot.

---

1. Where RECAP fits

RECAP sits between imitation-only post-training and full fleet-scale offline-to-online RL.

A useful progression is:

BC / SFT:
  imitate demonstrations

DAgger / HIL:
  imitate human corrections on policy-induced states

AWR / IQL-style offline RL:
  learn values and imitate high-advantage replay actions more

RECAP:
  learn values and condition the VLA policy on advantage

LWD:
  learn distributional values from fleet replay and update flow policy via critic gradients

So RECAP is not “just DAgger” and not “standard actor-critic RL.”

It is a VLA-friendly policy extraction method:

learn which experience is good
        ↓
turn goodness into a condition
        ↓
ask the model for good behavior at inference

---

2. Why RECAP is needed

2.1 SFT gives actions, not outcome rankings

SFT / behavior cloning teaches the policy to imitate demonstration actions.

That is a strong initialization, but it does not directly learn from:

• failed rollouts,
• partial progress,
• autonomous practice,
• sparse task success,
• human interventions,
• or recovery attempts.

If the dataset contains only clean successful trajectories, the model may not learn which failure-adjacent states are recoverable or which actions made success more likely.

2.2 HIL corrections give labels, not full credit assignment

DAgger / HIL collects corrections when the policy enters bad states. This helps with covariate shift.

But the learning signal is still mostly supervised imitation of the human correction. This teaches recovery actions, but it does not fully use the outcome structure of autonomous experience.

For example, suppose the robot tries a task three times:

trial 1: fails immediately
trial 2: makes partial progress but fails late
trial 3: succeeds

Pure imitation does not naturally convert these outcomes into a value ranking over actions and states. RECAP adds value learning so this experience can become a policy-improvement signal.

2.3 Problem with standard RL for VLAs

Standard policy-gradient RL usually requires quantities like:

$\log \pi_\theta(a|s)$

and updates such as:

$\nabla_\theta \log \pi_\theta(a|s) A(s,a)$

But modern VLA policies often use diffusion or flow matching action generators. For these models, likelihoods and direct policy-gradient updates can be awkward or expensive.

RECAP avoids this by turning policy improvement into a conditional generation problem.

Instead of saying:

directly push policy parameters with policy gradients

RECAP says:

teach the model to generate actions conditioned on how good they are

---

3. RECAP algorithm map

RECAP can be decomposed into three core pieces.

1. Data collection
   demonstrations, autonomous rollouts, corrections / interventions

2. Value learning
   learn Q / V from robot experience

3. Policy extraction
   train an advantage-conditioned VLA policy

At inference time:

condition on high advantage
        ↓
generate better actions

This is the key move.

The VLA does not need to expose a simple Gaussian log-probability or support a fragile policy-gradient update. It only needs to learn a conditional action distribution.

TODO: Insert/redraw algorithm-map figure.
Suggested filename: figs/recap_algorithm_map.png.
Source to redraw from: recent attachment /mnt/data/3be763d8-b33e-42ff-912f-cdb0b8dd81a3.png, the English screenshot titled roughly “π*_0.6 RECAP Training Method”. Focus on its value-learning / advantage-conditioning pipeline.

---

4. Data used by RECAP

RECAP uses heterogeneous robot data.

At a high level:

Data type	Description	Why it matters
Demonstrations	Expert successful executions	Teach clean task behavior
Autonomous rollouts	Policy attempts, successes or failures	Show what the policy actually does
Corrections / interventions	Human teleoperated recovery or guidance	Show how to fix failures
Experience data	Broad umbrella over robot trials	Enables value / advantage learning

The important point is that RECAP tries to learn from experience, not only from clean demonstrations.

4.1 Demonstrations

Demonstrations provide the initial supervised signal.

They answer:

What does successful behavior look like?

This is useful for initializing a capable policy and preventing random unsafe exploration.

4.2 Autonomous experience

Autonomous rollouts show how the current policy behaves when it is actually deployed.

They answer:

What does the policy do by itself?
Where does it succeed?
Where does it fail?
What partial progress does it make?

This is valuable because it reveals the model's own failure distribution.

4.3 Corrections / interventions

Human corrections provide targeted recovery data.

They answer:

When the policy is about to fail, what should it do instead?

This makes the data more informative than only passively observing failures.

4.4 Why mixing data matters

The mixture is important:

demonstrations:
  clean success

rollouts:
  policy-induced states

corrections:
  recovery behavior

rewards/outcomes:
  good vs bad experience

RECAP combines them through value learning and advantage-conditioned policy training.

---

5. Value learning

5.1 Value function

A value function estimates expected future return from a state:

$V^\pi(s) = \mathbb E_\pi \left[ \sum_{t=0}^{\infty} \gamma^t r_t \mid s_0=s \right]$

It answers:

How good is this state if we continue with the policy?

5.2 Q-function

A Q-function estimates expected return after taking a particular action:

$Q^\pi(s,a) = \mathbb E_\pi \left[ \sum_{t=0}^{\infty} \gamma^t r_t \mid s_0=s, a_0=a \right]$

It answers:

How good is this action in this state?

5.3 Advantage

The advantage is:

$A(s,a) = Q(s,a)-V(s)$

It answers:

Is this action better or worse than the typical action at this state?

If:

$A(s,a)>0$

then action $a$ is better than expected.

If:

$A(s,a)<0$

then action $a$ is worse than expected.

This is the signal RECAP uses for policy extraction.

---

6. Advantage-conditioned policy extraction

6.1 From advantage weighting to advantage conditioning

A common offline RL policy extraction method is Advantage-Weighted Regression (AWR).

AWR trains:

$\mathcal L_{\text{AWR}}(\theta) = - \mathbb E_{(s,a)\sim\mathcal D} \left[ \exp(A(s,a)/\lambda) \log \pi_\theta(a|s) \right]$

High-advantage actions get larger imitation weights.

AWR:
  advantage -> weight

RECAP uses a different idea:

RECAP:
  advantage -> condition

Instead of only weighting actions by advantage, RECAP trains the VLA policy to model actions conditioned on an advantage-related input:

$\pi_\theta(a|s,\hat A)$

where $\hat A$ is an advantage label, bin, token, or condition.

TODO: Verify exact advantage representation / binning / tokenization from the $\pi^*_{0.6}$ paper before final slides. Keep this doc at the conceptual level unless exact notation is checked.

6.2 Training-time behavior

During training, each action is paired with some indication of whether it was good or bad according to value learning.

Conceptually:

state s
action a
estimated advantage A(s,a)
        ↓
train πθ(a | s, advantage condition)

So the model learns:

when condition says low advantage:
  generate behavior like low-quality experience

when condition says high advantage:
  generate behavior like high-quality experience

This is a conditional generation problem, which is natural for VLA models.

6.3 Inference-time behavior

At inference time, we do not ask for arbitrary behavior.

We condition the model on high advantage:

state s
desired condition: high advantage
        ↓
πθ(a | s, high advantage)
        ↓
action a

The policy is therefore steered toward the subset of behavior that value learning judged to be good.

6.4 Simple mental model

Suppose the dataset contains two actions at similar states.

State:
  robot needs to grasp a cup

Action A:
  clean grasp
  task succeeds
  high advantage

Action B:
  bumps cup sideways
  task fails
  low advantage

Behavior cloning may imitate both depending on dataset frequency.

RECAP instead trains:

(clean grasp | high advantage)
(bump cup | low advantage)

At inference, it queries:

action | high advantage

So it asks the policy to produce the kind of action associated with successful experience.

---

7. Why advantage conditioning is VLA-friendly

7.1 It avoids direct policy gradients

For diffusion / flow-based VLA policies, direct policy-gradient RL can be hard because the policy may not provide convenient action likelihoods or stable gradients through the full action generation process.

Advantage conditioning turns the problem into supervised conditional modeling.

This is closer to what VLAs are already good at:

condition on input tokens / observations
        ↓
generate output action sequence

7.2 It can use bad data constructively

A failed action is not simply discarded. It can still be useful if the model knows it was low-advantage.

This is different from pure behavior cloning, where imitating failed experience would be harmful unless carefully filtered.

RECAP can use bad data because it labels or conditions it differently.

7.3 It separates “modeling behavior” from “choosing behavior”

During training, the model may learn to represent both good and bad behaviors.

At inference, we choose the desired condition:

high advantage

So the policy is not merely copying the marginal data distribution. It is sampling from a conditioned subset.

---

8. Policy extraction in context

This section is only for positioning RECAP. The detailed IQL derivation belongs in 03_lwd_main.md, because LWD modifies IQL into DIVL.

Method family	What value learning produces	How the policy uses it
AWR / IQL-style extraction	advantage weights	imitate high-advantage replay actions more
RECAP	advantage labels / conditions	generate actions conditioned on high advantage
LWD / QAM	critic action gradients	update a flow actor through adjoint matching

The key distinction is:

AWR / IQL:
  advantage -> weight

RECAP:
  advantage -> condition

LWD / QAM:
  critic gradient -> flow update

For RECAP, the important move is not the exact value-learning variant. It is the decision to turn "how good was this action?" into a generation condition that a VLA policy can consume.

TODO: Insert comparison diagram.
Suggested filename: figs/policy_extraction_comparison.png.
Source to redraw from: combine concepts from recent attachment /mnt/data/3be763d8-b33e-42ff-912f-cdb0b8dd81a3.png and the QAM section later in 03_lwd_main.md.

---

9. RECAP training pipeline

At a high level, RECAP can be described as a repeated loop:

1. Start from pretrained VLA
2. Collect / aggregate robot experience and corrections
3. Train value functions
4. Estimate advantages
5. Train advantage-conditioned policy
6. Deploy / evaluate improved policy

9.1 Pretrained VLA initialization

RECAP starts from a pretrained VLA policy.

This initial model is trained mainly through imitation / SFT on large-scale robot data.

It provides:

• language grounding,
• visual representation,
• action generation ability,
• and a reasonable starting behavior.

This is crucial because real-world RL from scratch would be too expensive and unsafe.

9.2 Experience collection

The policy is deployed to collect task attempts.

These attempts can include:

• successful rollouts,
• failed rollouts,
• partial completion,
• human teleoperated corrections,
• and recovery behavior.

The important part is that the data is not limited to perfect expert demonstrations.

9.3 Value function training

The value model learns from the collected experience.

The goal is to estimate which states/actions are associated with better outcomes.

Conceptually:

$Q(s,a) \leftarrow r+\gamma V(s')$

or with a multi-step / offline RL variant.

TODO: Verify exact distributional value objective used in RECAP from the paper. The method has a section on distributional value function training, so the final doc should eventually include its exact equations.

9.4 Advantage labeling / conditioning

Once $Q$ and $V$ are available, compute:

$A(s,a)=Q(s,a)-V(s)$

Then convert $A(s,a)$ into a policy condition.

Possible forms:

• binary good/bad label,
• discretized advantage bin,
• scalar condition,
• special token.

TODO: Verify exact condition format from the $\pi^*_{0.6}$ paper.

9.5 Policy training

Train:

$\pi_\theta(a|s,\hat A)$

using the replay data.

At inference, set:

$\hat A = \text{high advantage}$

so the policy generates actions associated with better outcomes.

---

10. Interventions in RECAP

RECAP uses human interventions as part of the experience-and-corrections pipeline.

10.1 Difference from pure HG-DAgger

In HG-DAgger, intervention data is mainly used as imitation labels:

$(s,a^*) \in \mathcal D$

and the policy is trained to imitate $a^*$.

In RECAP, interventions are still useful as corrective behavior, but the broader method also uses outcome/value information.

So intervention data serves two roles:

1. It gives the policy examples of recovery behavior.
2. It contributes to estimating which behavior is better.

10.2 Why this matters

Imagine a robot starts making espresso and misplaces the cup.

A human intervenes and fixes the cup position.

Pure imitation says:

imitate the human correction

RECAP can also use the correction as part of the broader experience:

state before correction:
  low value

recovery action:
  higher value

post-recovery state:
  closer to success

This gives a richer training signal.

---

11. What RECAP achieves

According to the public paper/abstract, RECAP is designed to improve VLA policies through real-world deployments using heterogeneous data, including demonstrations, on-policy collection, and expert teleoperated interventions.

The reported task domains include:

• folding diverse laundry,
• assembling boxes,
• making espresso drinks.

The high-level empirical claim is that the full RECAP-trained $\pi^*_{0.6}$ improves task performance and efficiency over the initial VLA policy.

TODO: Add exact result numbers and table after reading the paper carefully.
Suggested figure source: use the original $\pi^*_{0.6}$ paper figures/tables if available, not the current screenshots.

---

12. Limitations and bridge to LWD

RECAP is a strong prior method, but LWD is motivated by several remaining challenges.

12.1 Value learning under heterogeneous fleet data

Fleet replay is more heterogeneous than the single-method RECAP setting: many robots, many policy versions, sparse rewards, interventions, and rare success modes. LWD owns the detailed answer here with DIVL, a distributional value-learning method.

12.2 Policy extraction for flow-based VLA policies

RECAP uses advantage conditioning.

For flow-based VLA policies, LWD asks a different question: can the critic's action gradient directly guide the actor? 03_lwd_main.md owns the detailed QAM derivation.

12.3 System scale

RECAP studies real-world improvement from experience and corrections.

LWD pushes the idea toward a fleet-scale offline-to-online data flywheel:

robot fleet
  -> online replay
  -> central learner
  -> updated policy
  -> redeployment

This is why LWD should be viewed as both:

• an algorithmic contribution: DIVL + QAM,
• and a systems contribution: learning while deploying.

---

13. Summary comparison

Method	Data	Value learning	Policy extraction	Main idea
BC / SFT	demonstrations	no	imitate actions	learn initial behavior
HG-DAgger	corrections	no explicit value learning	imitate corrections	fix covariate shift
AWR	replay	Q/V advantage	weighted BC	imitate good data more
IQL	offline replay	expectile V + Q	advantage-weighted regression	avoid OOD action max
RECAP	experience + corrections	value / advantage learning	advantage-conditioned policy	ask VLA for high-advantage behavior
LWD	fleet offline + online replay	distributional value + critic	QAM	critic-guided flow policy update

---

14. Figure TODO list

TODO 1: RECAP high-level motivation

Suggested filename:

figs/recap_high_level_motivation.png

Source:

recent attachment:
  /mnt/data/fbb7f5d1-edac-466f-adb2-3f91ab0ccc79.png

description:
  English screenshot of previous RECAP high-level note.
  Use the parts that frame RECAP as combining imitation, coaching/corrections, and experience.

Use this in:

Section 0 or Section 1

TODO 2: RECAP method pipeline

Suggested filename:

figs/recap_method_pipeline.png

Source:

recent attachment:
  /mnt/data/3be763d8-b33e-42ff-912f-cdb0b8dd81a3.png

description:
  English screenshot of previous RECAP training-method note.
  Use the value-learning -> advantage -> advantage-conditioned policy structure.

Use this in:

Section 3: RECAP algorithm map

TODO 3: Chinese updated background reference

Suggested filename:

figs/recap_background_chinese_reference.png

Source:

recent attachment:
  /mnt/data/a930c264-94b2-4c51-a768-dadd733caba6.png
  or
  /mnt/data/8a86dace-146c-48d6-afd0-c9b7604e1208.png

description:
  Chinese versions are more updated than English versions.
  Use them as content reference rather than inserting directly.

Use this for:

checking wording / structure before final raw Markdown cleanup

TODO 4: Policy extraction comparison

Suggested filename:

figs/awr_recap_qam_comparison.png

Source:

custom redraw based on this doc:
  AWR: advantage -> weight
  RECAP: advantage -> condition
  LWD/QAM: critic gradient -> flow update

Use this in:

Section 8: Policy extraction in context

TODO 5: Original paper figures

Suggested filename:

figs/pi_star_06_results_or_pipeline.png

Source:

original π*_0.6 / RECAP paper or official blog, not from screenshots.

Use this in:

Section 11: What RECAP achieves

---

15. Personal takeaway

RECAP's key conceptual move is:

Do not force a VLA into a fragile low-level policy-gradient update. Instead, learn which actions are good, then condition the generative policy to produce good actions.

A compact summary:

BC:
  imitate expert behavior

DAgger:
  imitate corrections

AWR:
  imitate high-advantage actions more

RECAP:
  condition the policy on high advantage

LWD:
  use critic gradients to update the flow action generator

This makes RECAP a natural bridge between imitation-heavy VLA post-training and LWD's critic-gradient-based fleet-scale RL.