Learning While Deploying (LWD)

Explain Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies as the next step after RECAP-style VLA post-training.

Scope of this note:

• 01_vla_rl_background.md owns the general VLA RL background.
• 02_recap_method.md owns RECAP and advantage-conditioned policy extraction.
• This file owns LWD: fleet-scale offline-to-online learning, DIVL, and QAM.

The intended reading flow is:

01_vla_rl_background.md
  why VLA post-training needs imitation + HIL + RL + real-world data

02_recap_method.md
  how RECAP uses experience/corrections and advantage-conditioned policies

03_lwd_main.md
  how LWD scales the idea into fleet-scale offline-to-online RL with DIVL + QAM

---

0. LWD in one sentence

Learning While Deploying (LWD) turns robot deployment into a continuous learning loop.

Instead of treating deployment as the endpoint of training, LWD treats deployment as a source of new robot experience:

deploy shared VLA policy to robot fleet
        ↓
collect real-world rollouts and interventions
        ↓
update critic, value model, and policy
        ↓
redeploy improved policy
        ↓
repeat

Algorithmically, LWD combines:

1. DIVL: Distributional Implicit Value Learning
   Stable value/critic learning from heterogeneous offline + online replay.

2. QAM: Q-learning via Adjoint Matching
   A way to use critic action gradients to update a flow-based VLA policy.

Systemically, LWD is a fleet-scale offline-to-online RL system.

TODO: Insert LWD overview / data flywheel figure.
Suggested filename: figs/lwd_data_flywheel.png.
Source: LWD paper PDF, Fig. 1 on page 1, “Learning While Deploying (LWD): Fleet-scale Reinforcement Learning for Generalist Robot Policies.”

---

1. Motivation

1.1 Deployment is not the end of training

A pretrained VLA policy can be strong, but real deployment is not a fixed benchmark.

Once deployed, robots encounter:

• new objects,
• new layouts,
• new instructions,
• lighting/camera variation,
• long-tail failures,
• partial progress states,
• recovery situations,
• and human corrections.

So the deployment distribution is not fully covered by the offline training data:

$p_{\text{offline}}(s,a) \neq p_{\text{deploy}}(s,a)$

LWD’s key claim is:

Deployment should not be treated only as evaluation. It should become a source of continual policy improvement.

1.2 Why fleet-scale?

A single robot samples only a small slice of the deployment distribution.

A robot fleet samples more diversity:

different tasks
different scenes
different objects
different users / instructions
different failures
different recovery cases

If all robots share one policy and upload experience to a central learner, the fleet becomes a data flywheel:

more robots
  -> more deployment data
  -> better shared policy
  -> better future deployment
  -> more useful data

This is the main systems-level idea behind LWD.

1.3 Why not pure imitation or DAgger?

Local refresher:

Method	What it gives LWD	What it leaves unused
BC / SFT	strong initial policy from demonstrations	failures, partial progress, temporal credit
DAgger / HIL	recovery actions in policy-induced states	broader outcome/value structure of rollouts

For LWD, the missing signals are:

• failed autonomous rollouts,
• sparse terminal rewards,
• partial progress,
• temporal credit assignment,
• or value differences between actions.

For long-horizon robot tasks, these are crucial.

1.4 Why not just RECAP?

RECAP already uses experience, corrections, value learning, and advantage-conditioned policy extraction.

LWD is motivated by remaining challenges:

1. Fleet-scale data is heterogeneous
   Data comes from many robots, tasks, policy versions, successes, failures, play data, and interventions.

2. Scalar values may be insufficient
   The return distribution for similar states can be multi-modal or heavy-tailed.

3. Flow-based VLA policy extraction is hard
   Advantage-weighted regression or advantage conditioning may not fully use critic action gradients.

4. The system needs offline-to-online continuous learning
   The learner must keep mixing old offline data with fresh online deployment data.

So LWD introduces:

DIVL:
  distributional value learning + quantile bootstrap

QAM:
  critic gradient -> adjoint matching -> flow actor update

---

2. Problem setting

2.1 MDP

LWD formulates robot control as a Markov Decision Process:

$\mathcal M = (\mathcal S, \mathcal A, \mathcal T, r, \gamma)$

where:

• $s\in\mathcal S$ is the state,
• $a\in\mathcal A$ is the action,
• $\mathcal T$ is the transition dynamics,
• $r$ is reward,
• $\gamma$ is the discount factor.

For VLA policies, the state includes both robot observation and language instruction:

$s = (o,\ell_k)$

where:

• $o$ is the robot observation,
• $\ell_k$ is the language instruction for task $k$.

2.2 Action chunks

The policy outputs an action chunk, not just a single action:

$a_t \equiv a_{t:t+H} = [a_t,a_{t+1},\dots,a_{t+H-1}]$

This chunk is executed before the policy replans.

The corresponding chunk reward is:

$r_t \equiv r_{t:t+H} = \sum_{i=0}^{H-1} \gamma^i r_{t+i}$

So a replay transition is written as:

$(s_t,a_t,r_t,s_{t+H})\sim\mathcal D$

Important:

In LWD, $Q_\phi(s_t,a_t)$ is a critic over action chunks, not single low-level actions.

2.3 Sparse binary rewards

LWD uses sparse binary rewards:

$r = \begin{cases} 1, & \text{episode terminates successfully}\\ 0, & \text{otherwise} \end{cases}$

This is difficult for long-horizon tasks because reward may only appear at the end.

Therefore, value learning and TD backup are important for propagating success signals backward through the trajectory.

2.4 Data sources

LWD uses offline and online replay.

Offline buffer:

$\mathcal B_{\text{off}}$

Online buffer:

$\mathcal B_{\text{on}}$

Data types include:

Data type	Meaning
Demonstrations	expert successful trajectories
Rollouts	historical or current policy attempts, successes or failures
Play data	human-guided exploration, often around failure modes
Interventions	human corrective segments during online deployment

Play data can be viewed as HIL-style exploratory data, but it is not the same as clean demonstrations or online interventions.

A compact taxonomy:

demo:
  how to succeed cleanly

rollout:
  what the policy actually does

intervention:
  how to recover when the policy is failing

play data:
  human-guided exploration of hard/failure-adjacent regions

---

3. LWD system overview

LWD has two stages:

Stage 1:
  offline RL pretraining on B_off

Stage 2:
  online post-training with mixed replay from B_off ∪ B_on

TODO: Insert LWD pipeline figure.
Suggested filename: figs/lwd_offline_online_pipeline.png.
Source: LWD paper PDF, Fig. 2(a) on page 5, “Pipeline”.

3.1 Stage 1: Offline RL pretraining

The offline stage trains:

• policy $\pi_\theta$,
• critic $Q_\phi$,
• distributional value model $V_\psi$,

using a static offline buffer:

$\mathcal B_{\text{off}}$

This provides a strong initialization for real-world deployment.

3.2 Stage 2: Continuous online post-training

The online stage deploys the current policy to a robot fleet.

Each robot:

1. executes the policy,
2. collects online transitions,
3. optionally receives human interventions,
4. uploads experience to the online buffer.

The central learner samples mixed replay:

$\mathcal B_{\text{off}} \cup \mathcal B_{\text{on}}$

and continues updating:

$V_\psi,\quad Q_\phi,\quad \pi_\theta$

Updated policy checkpoints are periodically redeployed to the robots.

3.3 Why this is a systems contribution

LWD is not only an objective function. It is also a real-world learning system.

A practical LWD system must coordinate:

• robot-side inference,
• real-world execution,
• online replay upload,
• optional human intervention,
• centralized training,
• checkpoint synchronization,
• and redeployment.

This connects to the systems discussion in 01_vla_rl_background.md.

TODO: Insert/redraw embodied RL systems complexity figure.
Suggested filename: figs/embodied_rl_system_complexity.png.
Source: recent attachment /mnt/data/B32A1F25-C578-42F0-9DAC-CEA4B4D63DE4.jpeg, use the middle and right columns about heterogeneous components and scheduling/resource bottlenecks.
Do not use the left column here; the left column belongs in 01_vla_rl_background.md for online/off-policy/offline RL.

---

4. Algorithm structure

LWD’s learner has two coupled parts:

Value learning:
  DIVL learns V_psi and Q_phi

Policy extraction:
  QAM uses Q_phi to update flow actor f_theta

TODO: Insert LWD algorithm structure figure.
Suggested filename: figs/lwd_algorithm_structure.png.
Source: LWD paper PDF, Fig. 2(b) on page 5, “RL Algorithm Structure”.

The full loop is:

replay transition (s, a, r, s')
        ↓
DIVL:
  update distributional V and critic Q
        ↓
QAM:
  compute ∇_a Q
  update flow actor
        ↓
deploy improved policy

---

5. LWD-specific background: from IQL to DIVL

IQL is detailed here because DIVL is easiest to understand as "IQL, but with a distributional value model and quantile bootstrap." In 02_recap_method.md, IQL is only a comparison point for policy extraction.

5.1 Why IQL matters

Standard Q-learning uses:

$Q(s,a) \leftarrow r+\gamma\max_{a'}Q(s',a')$

In offline RL, this max is dangerous because it may select out-of-distribution actions.

IQL avoids explicit maximization by learning a value function as a high expectile over dataset action-values:

$V(s) \approx \operatorname{Expectile}_\tau [ Q(s,a), a\sim\mathcal D(\cdot|s) ]$

This gives an optimistic in-dataset value estimate without maximizing over unsupported actions.

5.2 IQL losses

IQL fits scalar value using expectile regression:

$\mathcal L_V^{\text{IQL}}(\psi) = \mathbb E_{\mathcal D} \left[ \rho_{\tau,2} ( Q_{\bar\phi}(s_t,a_t)-V_\psi(s_t) ) \right]$

where:

$\rho_{\tau,2}(u) = |\tau-\mathbb I(u<0)|u^2$

For $\tau>0.5$, this biases $V(s)$ toward higher dataset action-values.

The critic target is:

$y_t^{\text{IQL}} = r_t+\gamma^H V_\psi(s_{t+H})$

and the critic loss is:

$\mathcal L_Q^{\text{IQL}}(\phi) = \mathbb E_{\mathcal D} \left[ (Q_\phi(s_t,a_t)-y_t^{\text{IQL}})^2 \right]$

5.3 Why LWD modifies IQL

IQL uses a scalar value:

$V_\psi(s)\in\mathbb R$

But fleet replay can be heterogeneous:

same or similar state
  -> old policy failure
  -> new policy partial success
  -> human intervention recovery
  -> rare autonomous success

So the value distribution may be:

• multi-modal,
• heavy-tailed,
• high-variance,
• and sparse in high-return modes.

A scalar value can compress away rare but reproducible success modes.

LWD’s solution is Distributional Implicit Value Learning (DIVL).

---

6. DIVL: Distributional Implicit Value Learning

DIVL is LWD’s value-learning component.

It replaces scalar value regression with a distributional value model.

6.1 Distributional value model

DIVL learns:

$p_\psi(v|s_t) = P ( v=Q_\phi(s_t,a_t) \mid a_t\sim\mathcal D(\cdot|s_t) )$

This means:

Given state $s_t$, $V_\psi(s_t)$ represents a distribution over the Q-values of replay actions at that state.

So $V_\psi(s)$ is not a scalar. It is a distribution.

6.2 Fitting the value distribution

DIVL fits this distribution using scalar targets from the EMA critic:

$\mathcal L_V(\psi) = \mathbb E_{(s_t,a_t)\sim\mathcal D} \left[ -\log p_\psi ( Q_{\bar\phi}(s_t,a_t) \mid s_t ) \right]$

In implementation, the distribution is represented with categorical discretization.

The critic target value is projected onto a discrete value support, similar to C51-style distributional RL.

6.3 Quantile bootstrap

After fitting the distribution, DIVL extracts a $\tau$-quantile:

$\operatorname{Quant}_\tau(V_\psi(s_t)) = \inf \{v:F_\psi(v|s_t)\ge\tau\}$

where $F_\psi$ is the CDF of $p_\psi$.

The critic TD target is:

$y_Q = r_t + \gamma^H \operatorname{Quant}_\tau ( V_\psi(s_{t+H}) )$

The critic loss is:

$\mathcal L_Q(\phi) = \mathbb E_{\mathcal D} \left[ ( Q_\phi(s_t,a_t)-y_Q )^2 \right]$

6.4 DIVL vs IQL

IQL:

learn scalar V(s)
use high expectile as bootstrap

DIVL:

learn distribution p(v|s)
use high quantile as bootstrap

So the shared principle is:

avoid explicit max over all actions
but still favor high-value in-dataset actions

6.5 Why distribution helps fleet replay

Suppose replay contains values:

$Q(s,a)\in\{0.1,0.2,0.3,0.9\}$

A scalar value may average them into something like:

$0.375$

This hides the rare high-return mode $0.9$.

A distributional value can represent both low and high modes, and quantile extraction can preserve optimistic but in-distribution information.

This is especially useful for:

• sparse rewards,
• long-horizon tasks,
• intervention data,
• partial recovery,
• and heterogeneous task replay.

6.6 Adaptive $\tau$

DIVL also adapts $\tau$ based on uncertainty.

Given a categorical distribution with $C$ categories:

$\mathcal H(s) = - \frac{1}{\log C} \sum_{c=1}^C p_{\psi,c}(s)\log p_{\psi,c}(s)$

Then:

$\tau(s_{t+H}) = \operatorname{clip} ( \tau_{\text{base}} - \alpha\mathcal H(s_{t+H}), \tau_{\min}, \tau_{\max} )$

Intuition:

high uncertainty:
  lower τ
  more conservative target

low uncertainty:
  higher τ
  more optimistic target

This helps reduce overestimation when the value distribution is diffuse.

---

7. QAM: Q-learning via Adjoint Matching

DIVL learns a critic:

$Q_\phi(s,a)$

QAM answers:

Given this critic, how do we improve a flow-based VLA policy?

7.1 Why policy extraction is hard for flow policies

If the actor were a simple deterministic policy:

$a=\pi_\theta(s)$

we could directly optimize:

$\max_\theta Q_\phi(s,\pi_\theta(s))$

using:

$\nabla_\theta Q_\phi(s,\pi_\theta(s)) = \nabla_a Q_\phi(s,a) \frac{\partial \pi_\theta(s)}{\partial\theta}$

But LWD’s actor is a flow-based generative policy.

It generates an action chunk through a multi-step flow / denoising process.

Directly backpropagating critic gradients through the full generation process is:

• expensive,
• memory-heavy,
• numerically unstable,
• and not ideal for large VLA policies.

QAM provides a more stable local regression objective.

7.2 Flow matching recap

Flow Matching represents a generative policy as a time-dependent vector field.

Given:

$a^0\sim\mathcal N(0,I)$

and clean action:

$a^1=a$

define interpolation:

$a^w=(1-w)a^0+wa^1,\quad w\in[0,1]$

The flow model learns:

$f_\theta(s,a^w,w)$

which predicts the velocity from noise to action.

Standard flow matching learns:

$f_\theta(s,a^w,w)\approx a^1-a^0$

7.3 KL-regularized policy improvement target

QAM starts from a fixed reference policy:

$\pi_\beta(a|s)$

and defines an improved target:

$\pi^*(a|s) \propto \pi_\beta(a|s) \exp(Q_\phi(s,a)/\lambda)$

Interpretation:

start from reference behavior
tilt distribution toward high-Q actions

The temperature $\lambda$ controls how strongly the critic affects the policy:

large λ:
  conservative update

small λ:
  stronger critic-guided update

7.4 Residual flow

Let:

$f_\beta$

be the fixed reference flow.

Let:

$f_\theta$

be the trainable improved flow.

Define residual flow:

$f_\delta(s,a^w,w) = f_\theta(s,a^w,w)-f_\beta(s,a^w,w)$

So the improved policy is:

reference flow + critic-guided correction

7.5 Adjoint intuition

The critic directly gives a gradient at the final action:

$\nabla_a Q_\phi(s,a^1)$

This tells us how the final action should move to increase Q.

But the flow policy needs supervision at intermediate generation times:

$w\in[0,1]$

The adjoint state $\tilde g_w$ propagates the endpoint critic gradient backward along the reference flow trajectory.

Intuition:

critic gradient:
  how should final action move?

adjoint:
  how should each flow step contribute?

QAM loss:
  regress vector field to those local targets

7.6 Terminal adjoint condition

QAM sets:

$\tilde g_1 = - \nabla_a \left( \frac{Q_\phi(s,a^1)}{\lambda} \right)$

or:

$\tilde g_1 = -\frac{1}{\lambda}\nabla_a Q_\phi(s,a^1)$

Then the adjoint dynamics produce $\tilde g_w$ for earlier $w$.

7.7 QAM loss

QAM optimizes:

$\mathcal L_{\text{QAM}}(\theta) = \mathbb E \left[ \int_0^1 \left\| \frac{2f_\delta(s,a^w,w)}{\sigma_w} + \sigma_w\tilde g_w \right\|_2^2 dw \right]$

where:

$\sigma_w = \sqrt{2(1-w)w}$

At optimum:

$f_\delta^*(s,a^w,w) = -\frac{\sigma_w^2}{2}\tilde g_w$

Since:

$\sigma_w^2=2(1-w)w$

we get:

$f_\delta^*(s,a^w,w) = -(1-w)w\tilde g_w$

So QAM trains the actor to match a local critic-guided correction to the reference flow.

7.8 QAM in LWD

Algorithmically:

1. sample state s and Gaussian noise a^0
2. roll out reference flow f_beta to get trajectory a^w
3. evaluate critic gradient ∇_a Q_phi(s,a^1)
4. initialize terminal adjoint g_1
5. solve adjoint backward along trajectory
6. update f_theta using QAM regression loss

DIVL and QAM connect as:

DIVL learns Q_phi(s,a)
        ↓
QAM uses ∇_a Q_phi(s,a)
        ↓
flow actor is updated

TODO: Insert QAM intuition diagram.
Suggested filename: figs/qam_intuition.png.
Source: custom redraw. Use this structure:

noise a^0 + state s
  -> reference flow f_beta
  -> endpoint a^1
  -> critic gradient ∇_a Q
  -> adjoint backward
  -> local targets for f_theta

TODO: Insert LWD policy extraction figure.
Suggested filename: figs/lwd_qam_policy_extraction.png.
Source: LWD paper PDF, Fig. 2(b) on page 5, right half: “Policy Extraction (QAM)” showing critic gradient $\nabla_a Q(s,a)$ feeding action head update.

---

8. Offline-to-online training pipeline

8.1 Stage 1: offline training

Offline training uses:

$\mathcal B_{\text{off}}$

containing:

• demonstrations,
• historical rollouts,
• play data,
• successes,
• failures.

The learner updates:

$V_\psi,\quad Q_\phi,\quad \pi_\theta$

using the same DIVL + QAM objectives.

8.2 n-step TD target for offline long-horizon tasks

For long-horizon sparse-reward tasks, one-step targets propagate reward slowly.

LWD uses an n-step chunk-level target in offline training:

$y_Q = \sum_{i=0}^{n-1} \gamma^{iH}r_{t+iH} + \gamma^{nH} \operatorname{Quant}_\tau ( V_\psi(s_{t+nH}) )$

If the episode terminates inside the n-step window, the return is truncated and the bootstrap term is removed.

This accelerates sparse reward propagation.

8.3 Stage 2: online training

Online training deploys the policy to the robot fleet.

Robots collect:

• autonomous transitions,
• success/failure outcomes,
• and optional human intervention transitions.

The learner trains on mixed replay:

$\mathcal B_{\text{off}}\cup\mathcal B_{\text{on}}$

8.4 Why online uses 1-step targets

The paper reports that long multi-step targets were less effective in online training.

One possible reason:

online trajectories mix policy transitions and human interventions

Long backups may cross different behavior sources, so the TD path may not correspond to a single coherent policy execution.

Because the critic already has offline initialization, LWD uses 1-step chunk-level TD targets online.

8.5 What gets frozen / updated

In the online QAM stage:

• the policy VLM backbone is frozen,
• the action expert is updated,
• value and critic networks continue to be fully fine-tuned.

This helps:

• preserve pretrained vision-language representations,
• reduce online update cost,
• prevent catastrophic model drift,
• and still adapt the action generator to deployment data.

TODO: Insert algorithm pseudocode figure.
Suggested filename: figs/lwd_algorithm_pseudocode.png.
Source: LWD paper PDF, Algorithm 1 and Algorithm 2 on pages 6–7.

---

9. Architecture

9.1 Actor

The actor follows a flow-based VLA architecture.

It contains:

• a vision-language backbone,
• an action expert,
• and a flow-based action generation head.

The actor generates action chunks.

9.2 Value and critic

The value and critic networks are separate from the policy.

The value model:

$V_\psi(s)$

outputs a distribution over value support atoms.

The critic:

$Q_\phi(s,a)$

conditions on both state representation and action chunk.

The critic uses a clipped double-Q design to reduce overestimation.

9.3 Why separate actor and critic?

Separating actor and critic is useful because:

• only the actor needs to run on robot actors,
• critic/value stay in the centralized learner,
• actor inference remains efficient,
• critic can be updated freely without deployment overhead.

---

10. Experiments

10.1 Tasks

LWD is evaluated on eight real-world manipulation tasks.

Grocery restocking tasks:

• flat-shelf restocking,
• misplaced-item correction,
• freezer restocking with door operation,
• open-cooler restocking with carton handling.

Long-horizon tasks:

• brewing Gongfu Tea,
• making Fruit Juice,
• making Cocktail,
• packing shoes into a Shoebox.

The long-horizon tasks last roughly 3–5 minutes and require multiple substeps.

TODO: Insert task illustration figure.
Suggested filename: figs/lwd_tasks.png.
Source: LWD paper PDF, Fig. 3 on page 9, task illustrations A–E.

10.2 Robot fleet

The experiments use a fleet of 16 dual-arm robots.

TODO: Insert robot fleet figure.
Suggested filename: figs/lwd_robot_fleet.png.
Source: LWD paper PDF, Fig. 4 on page 9, robot fleet.

10.3 Baselines

The paper compares:

Method	Description
SFT	supervised fine-tuning on demonstrations
RECAP	prior VLA RL post-training baseline
HG-DAgger	human-gated DAgger
LWD Offline	offline stage only
LWD Online	full offline-to-online LWD

10.4 Main results

The key reported result:

LWD Online achieves the best average performance across tasks.

The gains are especially large on long-horizon tasks, where RL can propagate sparse rewards and learn from partial progress.

TODO: Insert main result bar plot / table.
Suggested filename: figs/lwd_main_results.png.
Source: LWD paper PDF, Fig. 5 and Table I on page 10.

10.5 Cycle time

LWD also reduces cycle time relative to the SFT reference policy on long-horizon tasks.

Interpretation:

critic-guided policy updates reduce hesitation, retries, and unstable intermediate behavior

TODO: Insert cycle-time plot.
Suggested filename: figs/lwd_cycle_time.png.
Source: LWD paper PDF, Fig. 5 bottom plot on page 10.

10.6 Value visualization

The paper visualizes learned value estimates during successful and failed Gongfu Tea episodes.

The successful trajectory shows value increasing as the robot approaches task completion.

The failed trajectory shows value staying lower or failing to track progress after execution goes off course.

TODO: Insert value visualization figure.
Suggested filename: figs/lwd_value_visualization.png.
Source: LWD paper PDF, Fig. 6 on page 11.

10.7 Ablations

Important ablations:

1. DIVL vs scalar expectile regression

   • DIVL performs better, especially on long-horizon tasks.

2. Adaptive $\tau$ vs constant $\tau$

   • adaptive $\tau$ improves consistency across tasks.

TODO: Insert ablation table.
Suggested filename: figs/lwd_divl_ablation.png.
Source: LWD paper PDF, Table II and Table III on page 11–12.

---

11. What is actually new?

LWD’s contribution has three layers.

11.1 Algorithmic contribution

DIVL:
  distributional value learning for heterogeneous fleet replay

QAM:
  critic-gradient-based policy extraction for flow VLA policies

11.2 Training-pipeline contribution

same RL objective in offline and online stages
offline pretraining
  -> online post-training

This reduces offline-to-online mismatch.

11.3 Systems contribution

robot fleet
  -> online data
  -> mixed replay
  -> central learner
  -> redeployed policy

This is the “learning while deploying” loop.

---

12. Relation to RECAP

RECAP and LWD share the broad goal:

Improve pretrained VLA policies using real robot experience and corrections.

But they differ in value learning and policy extraction.

Aspect	RECAP	LWD
Data setting	real experience + corrections	fleet-scale offline + online replay
Value learning	value / advantage learning	distributional value + critic
Policy extraction	advantage-conditioned policy	QAM with critic action gradients
Main update style	conditional generation	local flow regression
System emphasis	post-training with experience/corrections	continuous fleet-scale learning while deploying

Concise comparison:

RECAP:
  learn advantage
  condition policy on high advantage

LWD:
  learn distributional value + critic
  use critic gradient to update flow policy

---

13. Limitations / open questions

13.1 Online update schedule

The current online learning pipeline uses a straightforward update schedule.

At larger deployment scale, more sophisticated scheduling and data selection may matter.

13.2 Safety

LWD does not explicitly solve safe exploration or safety-constrained policy learning.

Real robots need safety constraints beyond reward maximization.

13.3 Human intervention dependence

Although LWD uses autonomous rollouts, interventions still matter.

Questions:

• How much human intervention is needed?
• How should intervention data be weighted?
• When should the system ask for human help?

13.4 Long-horizon reasoning

The policy uses a high-level instruction such as “Make Tea”.

Complex tasks may require more explicit decomposition, planning, or closed-loop error recovery.

13.5 Evaluation reliability

Real-world robot evaluation is noisy.

Questions:

• How many trials are enough?
• Can scenes be reset consistently?
• How should we measure generalization?
• How do we separate memorization from robustness?

---

14. Personal takeaways

14.1 The big idea

LWD is compelling because it reframes deployment:

deployment is not the end of training
deployment is the data source for continued learning

14.2 Why DIVL matters

DIVL addresses the fact that fleet replay is messy.

A scalar value may blur together successes, failures, recoveries, and rare high-return modes.

A distributional value model can preserve more information and support quantile-based optimistic backup.

14.3 Why QAM matters

QAM addresses the policy extraction problem for flow-based VLA actors.

It lets the critic guide the actor without unstable full backpropagation through the generation process.

14.4 Why the system matters

LWD is not just “RL loss on a VLA”.

It is a real-world learning system:

robot fleet
  + replay buffers
  + human interventions
  + centralized learner
  + policy redeployment

This is exactly why embodied RL is harder than LLM RLHF-style digital post-training.

---

15. Figure TODO list

TODO 1: LWD data flywheel

Suggested filename:

figs/lwd_data_flywheel.png

Source:

LWD paper PDF:
  Fig. 1, page 1

Use in:

Section 0 or Section 1

TODO 2: Offline-to-online pipeline

Suggested filename:

figs/lwd_offline_online_pipeline.png

Source:

LWD paper PDF:
  Fig. 2(a), page 5

Use in:

Section 3

TODO 3: Algorithm structure

Suggested filename:

figs/lwd_algorithm_structure.png

Source:

LWD paper PDF:
  Fig. 2(b), page 5

Use in:

Section 4

TODO 4: Systems complexity

Suggested filename:

figs/embodied_rl_system_complexity.png

Source:

recent attachment:
  /mnt/data/B32A1F25-C578-42F0-9DAC-CEA4B4D63DE4.jpeg

Use:
  middle and right columns only:
  - heterogeneous embodied RL components
  - framework scheduling / resource bottlenecks

Do not use:
  left column, because online/off-policy/offline RL belongs in 01_vla_rl_background.md.

Use in:

Section 3.3

TODO 5: QAM intuition

Suggested filename:

figs/qam_intuition.png

Source:

custom redraw

Diagram content:

state s + noise a^0
  -> reference flow f_beta
  -> trajectory a^w
  -> endpoint a^1
  -> critic gradient ∇_a Q(s,a^1)
  -> adjoint backward
  -> local targets for f_theta

Use in:

Section 7

TODO 6: LWD tasks

Suggested filename:

figs/lwd_tasks.png

Source:

LWD paper PDF:
  Fig. 3, page 9

Use in:

Section 10.1

TODO 7: Robot fleet

Suggested filename:

figs/lwd_robot_fleet.png

Source:

LWD paper PDF:
  Fig. 4, page 9

Use in:

Section 10.2

TODO 8: Main results

Suggested filename:

figs/lwd_main_results.png

Source:

LWD paper PDF:
  Fig. 5 and Table I, page 10

Use in:

Section 10.4

TODO 9: Value visualization

Suggested filename:

figs/lwd_value_visualization.png

Source:

LWD paper PDF:
  Fig. 6, page 11

Use in:

Section 10.6

TODO 10: DIVL ablations

Suggested filename:

figs/lwd_divl_ablation.png

Source:

LWD paper PDF:
  Table II, page 11
  Table III, page 12

Use in:

Section 10.7

---

16. One-paragraph summary

LWD is a fleet-scale offline-to-online RL framework for post-training generalist VLA policies. It starts from a pretrained policy, deploys it across a robot fleet, collects real-world rollouts and interventions, and continuously updates a shared policy. Algorithmically, DIVL learns a distributional value model and critic from heterogeneous replay, using quantile bootstrap to preserve optimistic in-dataset value information. QAM then uses the critic’s action gradient to update a flow-based VLA actor through adjoint matching, avoiding unstable direct backpropagation through the full generation process. The key conceptual shift is that deployment becomes a source of training data: the robot improves by being used.