Interview for Embodied AI

franklinqin0 1/20/2026 tech

# RAFT

# 1. feature extraction

feature encoder $g_\theta$ : maps the input images $I_1$ and $I_2$ to dense feature maps at a lower resolution context network $h_\theta$ : extracts features only from the first input image $I_1$

# 2. compute visual similarity

# Correlation Pyramid

The correlation pyramid $\{ \mathbf{C}^1, \mathbf{C}^2, \mathbf{C}^3, \mathbf{C}^4 \}$ is constructed by pooling the last two dimensions of the full correlation volume $\mathbf{C}$ with kernel sizes of $1 \times 1, 2 \times 2, 4 \times 4,$ and $8 \times 8$ . Consequently, the volume at level $k$ , denoted as $\mathbf{C}^k$ , has dimensions $H \times W \times H/2^k \times W/2^k$ .

# Correlation Lookup

# correlation volume

correlation volume: single matrix multiplication

\mathbf{C}(g_{\theta}(I_1), g_{\theta}(I_2)) \in \mathbb{R}^{H \times W \times H \times W}, \quad C_{ijkl} = \sum_{h} g_{\theta}(I_1)_{ijh} \cdot g_{\theta}(I_2)_{klh}

# correlation pyramid

${\mathbf{C}_1, \mathbf{C}_2, \mathbf{C}_3, \mathbf{C}_4}$
pooled by last two dimensions of the correlation volume with kernel sizes 1, 2, 4, and 8 and equivalent stride

volume $\mathbf{C}$ has dimensions $H \times W \times H/2^k \times W/2^k$ , first 2 dimensions are from $I_1$ .

# correlation lookup

map each pixel $x = (u, v)$ in $I_1$ to its estimated correspondence in $I_2$ : $x' = (u + f^1(u), v + f^2(v))$ . Using a local grid around $\mathbf{x}'$ and bilinear sampling:

\mathcal{N}(\mathbf{x}')_r = \{ \mathbf{x}' + \mathbf{dx} \mid \mathbf{dx} \in \mathbb{Z}^2, \|\mathbf{dx}\|_1 \leq r \}

can index from correlation volume. The features from each level are concat into a single feature map.

# 3. iterative updates

Input: Current flow $\mathbf{f}_k$ , correlation features $L_C$ , and a latent hidden state $h_{k-1}$ .
Output: Flow update $\Delta \mathbf{f}$ and an updated hidden state $h_k$ .

Update Rule: Each iteration produces $f_{k+1} = \mathbf{f}_k + \Delta \mathbf{f}$ .
Convergence: The sequence of updates is designed such that $f_k \rightarrow f^*$ .

Convolutional GRU Block

The update operator is modeled after a Gated Recurrent Unit (GRU), replacing fully connected layers with convolutions.

\begin{aligned} z_t &= \sigma(\text{Conv}([h_{t-1}, x_t], W_z)) \\ r_t &= \sigma(\text{Conv}([h_{t-1}, x_t], W_r)) \\ \tilde{h}_t &= \tanh(\text{Conv}([r_t \odot h_{t-1}, x_t], W_h)) \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \end{aligned}

Input $x_t$ : flow, correlation features, and context features.
Flow Head: hidden state outputted by the GRU is passed through two convolutional layers to predict the flow update $\Delta \mathbf{f}$ . Upsampling:

TODO: xxx

# GDRNPP

# Pose Estimation (1st network, est)

# world model

p(s_{t} \mid a_{t}, s_{<t})

$s_t$ : video frames or some representations of them (+ other modalities)
$a_t$ : intervention to the video

pretrain on all web-scale video data, learn basic physics and world info.
post-train on generating challenging tasks. Narrow down the scope, long-term planning / reasoning.
rate / pick interesting or high reward samples.
train the agent.

Human perception varies in sensitivity to different bits of a visual signal
Less sensitive to high-frequency details.
More sensitive to low-frequency content.

# Closing Thoughts

Video Generation: Developing rapidly. Initial usage in real-world content creation.
Controllable generation: Need more thoughts on how to define and unify actions.
Interleaved generation: a nice way to combine world models and agents. Need more thoughts on unified representations.
More challenges to tackle:
- long-term consistency and real-time cheap interactions.
- More efficient representations 💗 video models

# Diffusion

$\beta_t$ : A variance schedule that controls how much noise is added at each step.
$\mathbf{I}$ : The identity matrix, indicating isotropic Gaussian noise.

The probability of the state at time $t$ given the state at $t-1$ is defined as:

q(\mathbf{x}_t \vert \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I})

The "Reparameterization Trick" by defining $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$ ,

sample $\mathbf{x}_t$ directly from the starting image $\mathbf{x}_0$ without iterating through all intermediate steps:

q(\mathbf{x}_t \vert \mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t) \mathbf{I})

TODO: xxx

# Neural Rendering

Rendering:
Given 3D model (geometry, texture, material & lighting), camera params, etc., can obtain photo-realistic 2D images

in CV, the reverse process: given 2D images, try to get 3D reconstruction e.g., Building Rome in a Day, Agarwal et al. ICCV 2009

Neural Rendering: Novel View point synthesis
Given: 6 DoF Camera Pose / View Point
Neural Network encodes entire scene description, lighting, materials, etc.
Output: photo-realistic image Synthesis

Import 3D structure from CG

# Pix2Pix

use 2D Convs

Ground truth for training

Pose + Target Image (e.g., observed from real world)
Constrain with re-rendering loss

Testing

Given unseen pose, generate image

# Deep Voxels

Main idea for video generation: instead of learning 3D operations with 2D Convs, can use knowledge of how 3D transformations work. E.g., 6 DoF rigid pose $[R | t]$ . Once Incorporate these into the architectures (need to be differentiable), can apply to novel view point synthesis: given rigid pose, generate image for that view.

simplified overview novel view synthesis

Issue: we don’t know the depth for the target!
$\rightarrow$ Per-pixel softmax along the ray
$\rightarrow$ Network learns the depth

Insights

Lifting from 2D to 3D works great
- No need to take specific care for temp. coherency!
All 3D operations are differentiable
Currently, only for novel view-point synthesis
- I.e., cGAN for new pose in a given scene
But: limited resolution due to dense 3D voxel grid

# Scene Representation Networks

use MLP as implicit representation, query how far can I shoot rays to hit a surface

scene representation: ReLU MLP

renderer: generalized (learned) sphere-tracing
iterative query how far can you step before you find the surface point
each step has a length equal to the signed distance to the closest surface point of the scene

once shooting enough rays, the hypernetwork extrapolates and generalizes to look at different scenes.

# NeRF (Neural Radiance field)

input: a set of calibrated RGB images output: a 3D scene representation that renders novel views

scene representation: ReLU MLP, positional encoding, view direction
no explicit 3D structure
an implicit network, takes a ray, predict the color along the ray.

renderer: volumetric, stratified sampling

generalization: none
for every surface point in 3D, the network memorizes the color learns the mapping from ray to surface point

# 3DGS

TODO

# Representation Learning

encode raw data into feature embedding
lower dimensional represenatation used for downstream tasks (classification, segmentation, etc.)

given a backbone classfier trained on ImageNet, generalize to a different task like segmentation

# Good Representation

smoothness: close inputs map to close outputs
compactness: input dimension >> output dimension
robustness: features are insensitive to input noise
abstraction and invariances -> problem driven

abstraction: if too high level, not good for generalization invariances: feature descriptor invariant to focal length, but not invariant to day/night

good encoding for priors

# What are representations?

A representation performs the task of converting an observation in the real world (e.g., an image, a recorded speech signal, a word in a sentence) into a mathematical form (e.g., a vector).

The feature can be used by other models to produce outputs (classification, reconstruction, generation).

Examples

hand-crafted

binary (one-hot vector)

embedding vector

# Representation in CV

supervised constrained on tasks For classification, train on ResNet50 on ImageNet, then use the features in the last layer as image representations

unsupervised constrained on data For reconstruction,

clustering (K-Means) mean vectors as representations

...

# DINO (self-distillation with no labels)

a pair of 2 random transformations of the input image

centering: teacher's output is centered over the batch

prevent 1 dim dominates

# Nvidia

# Isaac Gym, Sim, Lab

Isaac Gym: 物理仿真和 RL 训练流程完全放在 GPU 上运行
Isaac Sim: General-purpose robotics simulation and synthetic data. Focus on high-fidelity single/multi-robot scenes.
Isaac Lab: Robot learning (RL) and loco-manipulation research. Focus on extreme parallelism (thousands of robots).

typical workflow: use Isaac Lab to drive Isaac Sim

Scene Creation: You might use Isaac Sim’s GUI to import a URDF of a robot or layout a factory floor in USD.
Logic & Learning: You switch to Isaac Lab to write a Python script that spawns 4,096 copies of that robot. Isaac Lab tells Isaac Sim: "Reset all these robots, apply these joint torques, and give me back the camera tensors."

# Omniverse

提到 Omniverse，一定要抓住这四个关键词：

USD (Universal Scene Description): 这是 Omniverse 的灵魂（由皮克斯开源）。它不仅是文件格式，更像是一种“3D 场景的 HTML”。它支持非破坏性编辑和图层管理，是实现复杂大规模场景的基础。
RTX 渲染 (Physically Based Rendering): Omniverse 拥有极其强大的光线追踪渲染能力。对于 VLA (Vision-Language-Action) 模型来说，这非常关键，因为你需要仿真环境产生的图像（Synthetic Data）在光影和材质上尽可能接近真实世界，以减少 Sim-to-Real gap。
PhysX (Physics Simulation): 它是 Omniverse 内置的物理引擎。现在的 PhysX 5.x 支持刚体、流体、布料、软体以及复杂的关节约束（Articulation），这是机器人仿真（Isaac Sim）的核心。
Connectors (插件生态): 它能连接 CAD 软件、Maya、Houdini 等。对于 SA 岗，如果你能提到“如何把客户现有的工业 CAD 模型通过 Omniverse Connector 快速导入 Isaac Sim 进行 AI 训练”，面试官会觉得你非常懂业务。

Choose mode