FEEL: Force Intuition from Egocentric Experience Learning

Abstract

Overview

When humans interact with familiar objects, they use visual cues to anticipate weight, stiffness, and deformability, then refine force after contact. Although current embodied AI models excel at semantic understanding and action generation, whether they can ground continuous physical quantities remains unclear.
We introduce FEEL (Force Estimation from Egocentric Learning), a controlled, multi-participant benchmark that pairs first-person RGB and 7-D hand state with synchronized fingertip-force targets. The v2.1 public release contains 4 participants, 52 trials, 3,317 episodes, and 101,219 synchronized frames, spanning fragile, deformable, and variable-load objects. Boundary, fill-level, and visual-physical mismatch samples probe safety limits and visual force priors.
Across representative embodied models, compact architectures with force-specific temporal and uncertainty-aware inductive biases outperform much larger VLA backbones. We retrain every model on data from all four participants and evaluate it on held-out episodes. EgoForce achieves the best overall MAE (6.583 N) and R² (0.271). On the separate empty-carton visual-mismatch test, it also obtains the best low-force MAE (1.481 N) and contact F1 (0.880). A controlled closed-loop simulation further shows that increasing force-action error reduces manipulation success from 84.7% to 0%.

Motivation

The Physical Intuition Gap

Human manipulation relies on internal sensorimotor models that generate visual feedforward priors before contact. Current VLA models lack this capacity.

🧠

Human Internal Model

Feedforward + Feedback

Cognitive science shows that humans employ visual feedforward priors before contact. Seeing a full cup triggers anticipatory scaling of grasp force. A deceptive empty milk carton causes measurable force overshoot—proof of visual prior formation.

Feedforward: Initial force ∝ visual water level
Feedback: Tactile correction after contact transient

🤖

Current VLA Models

Vision-Only · No Physical Prior

Models like RT-2 and OpenVLA excel at semantic understanding and instruction following, yet they lack explicit physical grounding. They may correctly pick up a potato chip, but without force intuition, the result is often catastrophic failure at safety boundaries.

Problem 1: No force scaling from visual cues
Problem 2: Brittle under visual-physical mismatch

Core Scientific Question

"Can current VLA models learn this human-like physical intuition, and under what circumstances do they struggle?"

Closed-loop validation · Isaac Sim

Force-estimation accuracy materially changes manipulation success

We perturb only the predicted bilateral force-action channels of a closed-loop Tabero-VTLA policy, while holding initial states and sampling noise paired. The rollout shows that small force errors remain recoverable, but larger errors prevent successful transport and placement.

Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, and Renjing Xu. Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language. ICML 2026. arXiv:2605.27886 ↗

84.7%0 N MAE · success

53.3%4 N MAE · success

28.0%6 N MAE · success

0%16–20 N MAE · success

150 trials per error level. Force-tracking MAE rises from 10.4 N to 21.2 N as injected output error increases from 0 N to 16–20 N.

Benchmark

The FEEL Benchmark & Data Engine

A reproducible visuo-tactile collection setup with hardware blueprints and spatio-temporal synchronization algorithms.

Data Collection Pipeline

👓

Meta Aria Glasses

6-DoF wrist & finger poses via MPS · mm accuracy

📐

3D-Print Blueprint

Open-source STL + BOM for low-cost sensor mount

✋

Paxini PX6AX Tactile Sensor

PX6AX-GEN3-DP-S2015-Elite on thumb & index · high-res fingertip force / deformation

⏱️

Spatio-Temporal Sync

Browser-based temporal alignment with manual refinement for Aria RGB, MPS hand poses, and tactile force

📦

FEEL Dataset

F/D/V taxonomy · boundary & mismatch samples

FEEL v2.1 at a Glance

The latest FEEL release expands both scale and participant diversity, enabling cross-participant evaluation and per-axis fingertip-force research.

4participants

52recording trials

3,317segmented episodes

101,219synchronized frames

4.62 GBdownload size

Release growth · v1 → v2.1

Participants

1 → 4

Episodes

700 → 3,317

Frames

39.5K → 101.2K

Duration

65.8 → 168.7 min

Episodes by object category

Bottle

1,534

Sponge

436

Water

398

Wangzai

341

Jelly

330

Chip

259

Coke

↗

Designed for generalization: four anonymized participants enable cross-participant evaluation, while two synchronized 3-axis fingertip-force vectors per frame support future shear-aware modeling. Models receive egocentric RGB and a 7-D hand state; tactile force is the prediction target, not an input.

Best prediction on held-out episodes from the expanded dataset

6.583 N ↓MAE · average force error

0.271 ↑R² · explained variation

All models are retrained on the four-participant FEEL dataset, then evaluated on 871 held-out episodes (22,630 frames) that are not used for training. EgoForce ranks first: lower MAE means predictions are closer to the measured force; higher R² means they follow changes in force more faithfully.

Lower MAE · higher R² is better

Best low-force performance when appearance is misleading

1.481 N ↓Low-force MAE · 0–5 N

0.880 ↑Contact F1 · touch detection

This separate OOD test uses empty milk cartons that look like the full cartons seen during training. It tests whether a model can handle a familiar appearance with a different true load. EgoForce has the smallest error in delicate 0–5 N interactions and the best contact/no-contact classification.

Lower error · higher F1 is better

90.4% less over-/under-estimation on deceptive empty cartons

22.6 NBefore expansion

→ 2.168 N ↓After 4-participant training

For each prediction, signed error tells us whether force is too high or too low. |MBE| is the magnitude of that average signed error: a large value means the model consistently predicts too much or too little force. Falling from 22.6 N to 2.168 N means the expanded training data largely removes this one-direction error on deceptive empty cartons.

Closer to 0 N is better

Scale & generalization

From a single participant to a multi-participant benchmark

The public-release totals are shown above. The quality-controlled evaluation split below contains 3,069 episodes after excluding invalid or unaligned samples.

Quality-controlled evaluation split

Episode split3,069 total

63.1%

28.4%

8.5%

Frame split85,177 total

65.2%

26.6%

8.2%

Train1,937 eps · 55,525 frames

Test871 eps · 22,630 frames

OOD261 eps · 7,022 frames

Held-out episodes after four-participant training · MAE ↓

EgoForce

6.583

GR00T

7.226

OpenTouch

7.356

ACT

7.492

WALL-X

7.999

π0.5

8.653

π0

8.947

SmolVLA

9.066

X-VLA

11.158

Chunk size = 1 for a controlled comparison. EgoForce uses the final MoE model; seed 1000, 10,000 training steps.

Visual-mismatch test · low-force MAE ↓

EgoForce

1.481

ACT

1.980

GR00T

2.928

π0.5

5.348

π0

5.383

OpenTouch

5.501

WALL-X

8.871

X-VLA

10.934

SmolVLA

11.825

MAE on frames whose mean ground-truth fingertip force is in [0, 5) N. Lower is better.

FEEL v2.1 public-release statistics. Explore the dataset on Hugging Face →

F/D/V Physical Taxonomy

Fragile

Brittle failure modes requiring precise force thresholding to avoid catastrophic damage.

Potato chipsCrackers

Deformable

Elastic/plastic behaviors requiring visual deformation-to-force mapping.

SpongeBreadBottles

Variable-Load

Objects whose effective weight varies with content or fill level, requiring force estimation under a changing, visually-inferred load.

Water cupsRice bagsMilk cartons

Three Controlled Probe Types

⚠️

Boundary Samples

Demonstrators deliberately approach or exceed unsafe force limits (e.g., crushing a potato chip). Tests safety-boundary prediction.

🌊

Water-Level Prior Samples

Cups filled at 0%, 25%, 50%, 75%, 100% capacity. Tests whether initial force scales with visual perception of weight.

🎭

Visual-Physical Mismatch

Visual appearance conflicts with true mass (empty can). Tests force overshoot and human-like "size-weight illusion" reproduction.

Method

Technical Contributions for Force Learning

Three targeted contributions that together improve continuous contact force prediction in egocentric VLA settings.

Laplace NLL Uncertainty-Aware Prediction Head

Standard deterministic regression treats all force errors equally, failing to account for label noise and the inherent variability of boundary samples. We replace the regression head with an a Laplace NLL formulation, making the model simultaneously predict force mean and uncertainty. A clamped scale prevents uncertainty escape while retaining calibrated predictions near contact boundaries.

L_Laplace = |F − μ| / b + log b ; log b ≤ 1.0

High scale b on boundary samples means the model correctly signals "I am uncertain here"—actionable for downstream safety planning. EgoForce w/o MoE (NLL+GRU+Contact) reaches Sponge R²=0.656 and the lowest 0–5 N MAE (1.66 N) overall.

Finger-Distance-Conditioned Contact State Encoding

RGB images alone cannot distinguish approach, first contact, or firm grasp. We inject the thumb–index distance as an additional input feature whose trajectory continuously encodes contact phase—information invisible to the camera.

d_ti = ‖p_thumb − p_index‖₂ ; z_ACS = [s_6-DoF ⊕ d_ti]

Ablation: removing this input raises Jelly MAE from 5.115→5.854 (+14%) and drops R² by 0.172. Contact-phase context that pure visual features cannot recover.

AdaMoE: Cross-Scenario Expert Routing

Different materials (Fragile, Deformable, Soft) exhibit heterogeneous force patterns that a single model struggles to capture. AdaMoE uses a scene-adaptive router to dynamically combine a shared prediction head with multiple expert heads (top-k selection). A Scale Adapter decouples expert selection from contribution weighting, avoiding the load-balance–accuracy trade-off of vanilla MoE.

ŷ = w_shared·h_shared + Σ_k w_k·h_expert-k

Final configuration: 4 experts, top-2 selection, routed only by the GRU hidden state. AdaMoE raises overall R² from 0.629 to 0.703 while redistributing capacity across different force regimes.

Model architecture

EgoForce: from synchronized perception to calibrated force

A compact causal pipeline fuses egocentric vision with contact-phase hand state, then routes temporal features to force-specialized experts. Tactile sensing supplies synchronized training targets rather than an inference input.

~37M · chunk size 1

Synchronized inputs1

▧Egocentric RGB sequenceappearance · deformation · load cues

→

ResNet-18 visual encoderz_vision

⌁7-D hand state6-DoF + d_ti

→

Linear ACS state encoderz_state

Token fusion2

Self-Attention Encoderwithin-stream context

Cross-Attention Decodercross-modal tokens aligned per frame

x_1:T · fused token sequence

Temporal memory3

GRU

h_t ∈ ℝ²⁵⁶
sub-100 ms → seconds

AdaMoE routing4

Router
softmax

top-2 / step

Shared

Scale Adapter weightsΣ

Prediction heads5

Laplace force headμ_L/R, b_L/R
log b ≤ 1.0

Contact headp(touch_L/R)

Continuous bilateral forceF̂_t ∈ ℝ² · chunk 1

Synchronized tactile labels: F_L/R + touch_L/R → End-to-end objective L_Laplace+ λ_contactL_BCE+ λ_balanceL_balance

Contact grounding. Finger distance exposes approach, contact onset, and stable grasp phases that RGB alone cannot recover.

Multi-timescale memory. The GRU bridges fast contact transients with slower deformation and load evolution.

Input–target separation. At inference, EgoForce uses RGB and hand state; fingertip tactile force is the supervised prediction target.

Experiments

Offline Evaluation Results

Real results on Jelly & Sponge materials (3 runs ±σ). Strong VLA backbones do not automatically outperform targeted force models on contact force estimation.

Method	MAE (N) ↓	RMSE (N) ↓	R² ↑	Pearson r ↑	Contact F1 ↑	Speed
Ours (NLL + Temporal)
EgoForce ⭐	4.550	8.272	0.703	0.858	0.903	~1.67 s/step
Baselines
ACT	5.318	9.960	0.569	0.830	0.939	~1.2 s/step
WALL-X	5.420	9.455	0.611	0.803	0.867	~1.22 s/step
Large VLA Models
SmolVLA	7.627	12.440	0.327	0.715	0.862	~2.4 s/step
Pi0	6.243	11.069	0.468	0.707	0.865	~4.3 s/step
Pi0.5	6.451	11.245	0.451	0.698	0.879	~5.4 s/step
X-VLA	8.707	14.339	0.108	0.585	0.857	~2.1 s/step
Pi0-FAST	16.088	32.543	−4.574	0.015	0.843	~10.8 s/step

Strong VLA backbones fail on continuous force estimation

Pi0-FAST collapses to overall R²=−4.574 and X-VLA to R²=0.108—both billion-parameter VLAs perform worse than the lightweight ACT baseline (R²=0.569) on continuous force curves. FEEL's controlled protocol isolates this failure mode that standard action benchmarks completely miss. Our EgoForce achieves overall R²=0.703.

✅ Normal Case — Full Can

Visual appearance matches actual weight. No force overshoot. Model correctly scales force.

Correct Prior · No Overshoot

⚡ Mismatch — Empty Can (Deceptive)

Model applies heavy-can prior. Dramatic initial overshoot followed by correction—mirroring human "size-weight illusion."

Visual Overshoot · Human-Like Misjudgment

✓

Models reproduce human-like force overshoot under visual deception

The force overshoot in visuo-tactile models mirrors the human "size-weight illusion," confirming that FEEL models build genuine visual force priors—not just kinematic imitation. This misjudgment is a feature, not a bug: it proves that visual cues influence force prediction.

✅ Normal Case — Full Can

Visual appearance matches actual weight. No force overshoot. Model correctly scales initial force.

Correct Prior · No Overshoot

⚡ Mismatch — Empty Can (Deceptive)

Model applies heavy-can prior. Dramatic initial overshoot—mirroring the human "size-weight illusion." This misjudgment is evidence of prior formation, not a bug.

Visual Overshoot · Human-Like Misjudgment

✓

Visual-mismatch overshoot proves prior formation, not kinematic imitation

A model that overshoots on the empty-can probe has built a genuine visual force prior—the same "size-weight illusion" humans experience. Models that don't overshoot are simply not forming physical priors, regardless of their semantic task performance. FEEL uses this as a diagnostic signal.

Two paper-aligned analyses unpack what each inductive bias contributes (single seed = 1000). Top: ablation across our three GRU-based prediction heads (paper Table 2). Bottom: per-model MAE stratified by ground-truth force magnitude (paper Table 3).

GRU Head Ablation (Paper Table 2)

Variant	Overall R² ↑	Overall MAE ↓	Sponge R² ↑	Empty B. R² ↑	Full B. R² ↑	0–5 N MAE ↓
EgoForce w/o MoE & NLL & Contact (L1 head)	0.677	4.916	0.123	0.641	0.388	3.47
EgoForce w/o MoE & Contact	0.630	4.986	0.623	0.623	0.277	2.84
EgoForce w/o MoE (NLL+GRU+Contact)	0.629	4.831	0.656	0.368	0.284	1.66

All three variants share the ResNet-18 tactile encoder, ACS state encoder, and GRU temporal backbone—differing only in the prediction head. Plain L1 wins overall R² and the high-force Empty / Full Bottle categories; NLL+Contact wins the low-force regimes (Sponge R²=0.656, 0–5 N MAE=1.66 N) thanks to heteroscedastic uncertainty + auxiliary contact loss. No single head wins everywhere — motivating the material-aware AdaMoE router (overall R²=0.703).

Force-Range-Stratified MAE (Paper Table 3)

Model	0–5 N MAE ↓	5–20 N MAE ↓	20–50 N MAE ↓
EgoForce w/o MoE	1.66	3.81	13.42
EgoForce ⭐	2.66	3.11	10.95
EgoForce w/o MoE & Contact	2.84	3.62	12.77
ACT	3.22	3.86	13.73
EgoForce w/o MoE & NLL & Contact	3.47	3.47	11.05
WALL-X	4.44	3.59	11.69
Pi0	4.38	4.09	13.08
Pi0.5	4.75	4.31	13.34
SmolVLA	7.59	5.37	14.49
X-VLA	9.42	5.80	13.33
Pi0-FAST	12.76	13.29	22.19

Every billion-parameter VLA degrades faster in the low-force regime than mid/high — SmolVLA and X-VLA lose 7.6–9.4 N on 0–5 N samples while keeping 13–15 N error on the much larger 20–50 N targets — suggesting they bias toward the population-mean force rather than tracking contact-onset transients.

✓

Different inductive biases win different force regimes

Within the GRU family, NLL+Contact decisively wins the low-force regime (0–5 N MAE=1.66 N) where heteroscedastic uncertainty pays off, while plain L1 takes over in the high-force Empty / Full Bottle categories where mean prediction matters more. AdaMoE routes per-step to a shared head plus top-2 specialised experts, lifting overall R² from 0.629 to 0.703 and winning Jelly, Sponge, and Full Bottle simultaneously.

For each material, all 9 models predict on the same source clip with the predicted force curve overlaid against the ground truth. Switching the material tab tears down the previous 9 videos and rebuilds a fresh 3×3 grid, keeping at most 9 active video decoders in the DOM.

Fragile

Deformable

Rigid

Variable-Load

Order (row-major): EgoForce (ours), WALL-X, ACT, Pi0, Pi0.5, SmolVLA, X-VLA, GR00T, OpenTouch. All 9 clips play muted-autoplay-loop with drift correction against the top-left reference.

Core Insight from Real Data

R²=0.703

EgoForce (Ours) — best overall R²

R² < 0

Pi0-FAST — large VLA fails on force

R²=0.569

ACT — lightweight baseline beats massive VLAs

Key Findings

What FEEL Reveals

Four empirically grounded findings that distinguish FEEL from conventional action benchmarks.

Strong VLA backbones ≠ better force estimation

Massive VLA backbones produce low or negative overall R² on continuous force (X-VLA R²=0.108, SmolVLA 0.327, Pi0-FAST −4.574), often worse than the lightweight ACT baseline (R²=0.569). Their action-generation strengths do not transfer to continuous contact force prediction. FEEL's controlled protocol exposes this failure mode that standard task-success metrics completely hide.

Both visual input and hand state are necessary signals

Within our GRU family, removing the proprioceptive state cuts Jelly R² roughly in half, with thumb–index finger-distance d_ti as the dominant recoverable component (paper §5.2). Egocentric vision provides material/deformation cues; finger-distance+pose provides contact geometry. Models that ignore either modality cannot predict realistic force curves.

Temporal modeling + uncertainty awareness outperforms bigger backbones

EgoForce (~37M params) outperforms every billion-parameter foundation baseline (Pi0 578M, WALL-X 4.2B, Pi0-FAST 3.3B). The bottleneck for contact-force learning is not model scale but the right inductive biases: explicit temporal context (GRU), calibrated uncertainty (NLL head with auxiliary contact loss), and material-aware Mixture-of-Experts routing.

Material heterogeneity requires adaptive modeling; single models trade off

EgoForce's AdaMoE router lifts overall R² from 0.629 (no-MoE baseline) to 0.703 and is the only configuration simultaneously best on Jelly, Sponge, and Full Bottle. Routing analysis reveals partial expert specialisation along low-force (Sponge / Empty Bottle) versus high-force (Full Bottle) regimes, with Jelly shared—evidence that FEEL's F/D/V taxonomy captures physically meaningful variation.

Overview

The Physical Intuition Gap

Human Internal Model

Current VLA Models

Force-estimation accuracy materially changes manipulation success

The FEEL Benchmark & Data Engine

Data Collection Pipeline

Meta Aria Glasses

3D-Print Blueprint

Paxini PX6AX Tactile Sensor

Spatio-Temporal Sync

FEEL Dataset

FEEL v2.1 at a Glance

Release growth · v1 → v2.1

Episodes by object category

Best prediction on held-out episodes from the expanded dataset

Best low-force performance when appearance is misleading

90.4% less over-/under-estimation on deceptive empty cartons

From a single participant to a multi-participant benchmark

Quality-controlled evaluation split

Held-out episodes after four-participant training · MAE ↓

Visual-mismatch test · low-force MAE ↓

F/D/V Physical Taxonomy

Fragile

Deformable

Variable-Load

Three Controlled Probe Types

Boundary Samples

Water-Level Prior Samples

Visual-Physical Mismatch

Egocentric Footage

Technical Contributions for Force Learning

Laplace NLL Uncertainty-Aware Prediction Head

Finger-Distance-Conditioned Contact State Encoding

AdaMoE: Cross-Scenario Expert Routing

EgoForce: from synchronized perception to calibrated force

Offline Evaluation Results

Strong VLA backbones fail on continuous force estimation

Models reproduce human-like force overshoot under visual deception

Visual-mismatch overshoot proves prior formation, not kinematic imitation

GRU Head Ablation (Paper Table 2)

Force-Range-Stratified MAE (Paper Table 3)

Different inductive biases win different force regimes

What FEEL Reveals

Strong VLA backbones ≠ better force estimation

Both visual input and hand state are necessary signals

Temporal modeling + uncertainty awareness outperforms bigger backbones

Material heterogeneity requires adaptive modeling; single models trade off

Open-Source Hardware Blueprint

BibTeX