FEEL: Force Intuition from Egocentric Experience Learning

Abstract

Overview

FEEL (Force Estimation from Egocentric Learning) is a controlled benchmark for estimating thumb and index-finger forces from egocentric RGB and a 7-D hand state. Four demonstrators contribute 3,069 episodes and 85,177 processed frames across 10 object/fill-state categories. Tactile measurements supply supervision; they are not model inputs.
EgoForce has 36,177,101 parameters (36.18 M): a visual–state Transformer and shared GRU feed shared Laplace heads, while two softly weighted residual MLP experts refine only the force means. On Test it achieves contact MAE 7.2846 N, contact R² 0.1950, contact r 0.2992, and pooled F1 0.8677. On empty-milk OOD, contact r is 0.4367 and pooled F1 is 0.8737.
All main comparisons use seed 1000 and 10,000-step checkpoints. Controlled force-action perturbations reduce simulated task success from 84.7% to 0%; a separate tactile-feedback controller tracks prescribed real-gripper forces with final-window MAE 0.0207 N. Connecting EgoForce predictions to robot control remains future work.

Motivation

The Physical Intuition Gap

Human visual priors and tactile correction motivate controlled tests of force estimation, especially when appearance conceals load.

🧠

Human Internal Model

Feedforward + Feedback

Humans use visual cues to anticipate load before contact and tactile feedback to correct force. Controlled fill-state and opaque-container probes motivate FEEL’s evaluation.

Feedforward: Initial force ∝ visual water level
Feedback: Tactile correction after contact transient

🤖

Force estimation remains a separate challenge

Accuracy varies across models and metrics

Action-generation pretraining does not guarantee accurate current-force estimation. FEEL compares RGB-and-state models using contact magnitude, temporal variation and contact detection.

Question 1: Can force magnitude be estimated?
Question 2: Do trend tracking and contact detection generalize?

Core Scientific Question

"Can current VLA models learn this human-like physical intuition, and under what circumstances do they struggle?"

Closed-loop validation · Isaac Sim

Force-estimation accuracy materially changes manipulation success

We perturb only the predicted bilateral force-action channels of a closed-loop Tabero-VTLA policy, while holding initial states and sampling noise paired. The rollout shows that small force errors remain recoverable, but larger errors prevent successful transport and placement.

Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, and Renjing Xu. Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language. ICML 2026. arXiv:2605.27886 ↗

84.7%0 N MAE · success

53.3%4 N MAE · success

28.0%6 N MAE · success

0%16–20 N MAE · success

150 trials per error level. Force-tracking MAE rises from 10.4 N to 21.2 N as injected output error increases from 0 N to 16–20 N.

Benchmark

The FEEL Benchmark & Data Engine

A reproducible visuo-tactile collection setup with hardware blueprints and spatio-temporal synchronization algorithms.

Data Collection Pipeline

👓

Meta Aria Glasses

MPS hand tracking provides thumb and index fingertip positions for the 7-D state.

📐

3D-Print Blueprint

Open-source STL + BOM for low-cost sensor mount

✋

Paxini PX6AX Tactile Sensor

PX6AX-GEN3-DP-S2015-Elite on thumb & index · high-res fingertip force / deformation

⏱️

Spatio-Temporal Sync

Browser-based temporal alignment with manual refinement for Aria RGB, MPS hand poses, and tactile force

📦

FEEL Dataset

F/D/V taxonomy · boundary & mismatch samples

FEEL benchmark at a glance

The ICLR benchmark uses the processed, quality-controlled collection. Raw recording duration and processed frame counts refer to different stages of the data engine.

4participants

10object / fill-state categories

3,069segmented episodes

85,177synchronized frames

449.5raw recording minutes

Chip · Jelly · Sponge · Empty Bottle · P25 / P50 / P75 / Full Bottle · Filled Milk · Empty Milk. F/D/V properties overlap.

Train / Test / OOD

Nine categories train and test the model; Empty Milk is reserved for OOD. Episodes stay intact across splits. Partitioning combines episode stratification on some recordings and source-session splits on others.

Episodes3,069 total

63.1%

28.4%

8.5%

Frames85,177 total

65.2%

26.6%

8.2%

Train1,937 eps · 55,525 frames

Test871 eps · 22,630 frames

OOD · Empty Milk261 eps · 7,022 frames

↗

Evaluation scope: 9 categories are used for Train/Test; empty milk containers form the 10th, OOD category. Four demonstrators contribute data, but this split does not establish held-out-person generalization. Inputs are RGB and 7-D hand state; targets are two scalar fingertip forces.

Source: ICLR 2027 manuscript §3 and §5; 449.5 min of raw recording. Hugging Face →

F/D/V Physical Taxonomy

Fragile

Fragile materials such as potato chips, including recorded fracture boundaries.

Potato chips

Deformable

Bottles, jelly and sponges exhibit visible deformation during contact.

SpongeBottles

Variable-Load

Bottle fill levels and filled/empty opaque milk containers vary load. F/D/V properties may overlap.

Bottle fill levelsFilled / Empty Milk

Three Controlled Probe Types

⚠️

Boundary Samples

Demonstrators deliberately approach or exceed unsafe force limits (e.g., crushing a potato chip). Tests safety-boundary prediction.

🌊

Water-Level Prior Samples

Bottles at 0%, 25%, 50%, 75% and 100% fill support controlled comparisons of visible load cues.

🎭

Visual-Physical Mismatch

Empty opaque containers conceal their true load, probing force estimation under misleading appearance.

Method

Technical Contributions for Force Learning

Shared visual–state temporal modeling, clamped Laplace prediction, and residual force-mean refinement.

Finger-specific shared Laplace heads

Each finger uses a 512 → 256 → 2 GELU head to predict a base mean and log scale in quantile-normalized force space. The same clipped log scale is used in both terms of the Laplace objective.

L = E[ |y − μ| / exp(s̄) + s̄ ]; s̄ = clip(s, −5, 2)

The log scale is clipped to [−5, 2]. An auxiliary 512 → 64 → 1 GELU/sigmoid contact branch adds BCE with weight 0.1. Its training target uses max normalized finger force > 1; evaluation contact uses physical force > 0 N.

7-D fingertip geometry and four-frame GRU

The hand state contains thumb xyz, index xyz and their distance. An ImageNet-initialized ResNet-18 and a 4-layer encoder / 1-layer decoder fuse RGB and state. A shared 512-D GRU processes four observations and predicts the force at the final timestep.

state = [p_thumb,xyz ⊕ p_index,xyz ⊕ d_ti]; n_obs = 4

The GRU hidden state resets for each window. Transformer width: 512; attention heads: 8; feed-forward width: 3200. Images: 480 × 480. This is a local four-frame context, not persistent episode-wide memory.

Two softly weighted residual MLP experts

A 512 → 2 linear router takes the final GRU hidden state. Both 512 → 128 → 2 GELU experts are active for every prediction; their softmax-weighted outputs refine the shared force means. Log scales stay in the shared heads.

μ = μ^base + Σ_k=1,2 softmax(W_rh_t + b_r)_k Δμ_k(h_t)

Zero-initialized expert output projections start with no correction. Experts and router add 132,870 parameters (+0.37%): 36,044,231 → 36,177,101.

Model architecture

EgoForce: shared temporal features, residual force refinement

RGB and 7-D hand state → shared Transformer/GRU → shared force heads + two residual MLP experts. Tactile forces are supervision targets. Source: paper Figure 4 and Appendix B.1.

36.18 M · n_obs = 4 · chunk 1

Synchronized inputs1

Egocentric RGB sequence480 × 480 RGB

→

ResNet-18 visual encoderImageNet · z_vision

7-D hand statethumb xyz ⊕ index xyz ⊕ d_ti

→

Linear state encoderz_state

Token fusion2

4-layer Transformer encoderd = 512 · heads = 8 · FFN = 3200

1-layer Transformer decoderRGB + hand-state fusion

x_t−3:t · n_obs = 4

Shared temporal memory3

GRU

h_t ∈ ℝ⁵¹²
4 observations · reset per window

Residual force head4

Shared base μ, s2 × (512 → 256 → 2)

Router
512 → 2

softmax(h_t)

Δμ₁
512→128→2

Δμ₂
512→128→2

μ = μ^base + π₁Δμ₁ + π₂Δμ₂

Outputs & contact5

Corrected mean + shared log scaleμ_L/R, s_L/R
s̄ = clip(s, −5, 2)

Two scalar fingertip forcesF̂_thumb, F̂_index · current t

Auxiliary contact (from hₜ)h_t → 512→64→1
GELU · sigmoid

L = L_Laplace + 0.1 L_BCE+ constant balance term (0.02)

7-D state. Thumb xyz ⊕ index xyz ⊕ thumb–index distance. These are fingertip coordinates, not a 6-DoF wrist pose.

Residual means only. Two experts correct μ; shared heads retain the scale outputs. Both experts contribute on every sample.

Training. AdamW, LR 1e−5, batch 4, weight decay 1e−4, 10,000 steps, seed 1000. ImageNet initializes the visual backbone.

Training and model details · Appendix B

ImageNet-initialized ResNet-18; four-layer Transformer encoder, one-layer decoder; width 512, 8 heads, FFN 3200. Shared GRU: hidden 512, four observations, reset per window. Finger heads: 512→256→2. Contact branch: 512→64→1. Router: 512→2; residual experts: 2 × (512→128→2). AdamW: LR 1e−5, batch 4, weight decay 1e−4, gradient clip 1.0; 480×480 images, quantile state/force normalization.

Experiments

Offline Evaluation Results

ICLR 2027 protocol: 871 Test episodes / 22,630 frames, seed 1000, 10,000-step checkpoints. Main comparisons use chunk size 1; default chunks are reported separately.

Test: contact magnitude and detection

7.2846 N ↓

0.8677 F1 ↑

Best Test contact MAE, contact R² (0.1950) and pooled F1. Contact r = 0.2992; the shared-head variant has higher r = 0.3094.

OOD: trend tracking and contact detection

0.4367 r ↑

0.8737 F1 ↑

Best contact r and pooled F1 among the Table 1 configurations on OOD. MAE = 5.2113 N and R² = −3.2403; this is not best OOD magnitude accuracy.

Residual experts: small cost, mixed Test gains

+132,870

36.18 M total

Versus the shared head, Test MAE improves by 0.0070 N, R² by 0.0109 and F1 by 0.0087. Test r decreases by 0.0102; all four OOD metrics improve.

How the metrics are computed

Predictions are inverse-normalized and clipped at 0 N; contact threshold is 0 N. Contact MAE averages contact-frame errors per finger and episode, then equally across episodes. Contact R² pools contact frames per finger, then averages fingers. Contact r averages eligible finger/episode correlations (≥5 contact frames and force SD ≥0.1 N). F1 pools all frames and both fingers, including noncontact frames.

Test contact MAE · N ↓

EgoForce (MoE)

7.2846

EgoForce w/o MoE

7.2916

GR00T

7.6107

OpenTouch

7.7258

WALL-X

7.8986

ACT

8.4400

Pi0.5

9.1171

SmolVLA

9.4473

Pi0

9.5923

X-VLA

11.4716

The residual model is best on Test MAE, R² and F1; the shared-head model leads on contact r. Models are evaluated from seed-1000, 10,000-step checkpoints, not repeated-seed means.

Empty-Milk OOD · contact r ↑

EgoForce

0.437

ACT

0.219

WALL-X

0.241

OpenTouch

0.320

GR00T

0.340

Pi0.5

0.155

Pi0

0.223

SmolVLA

0.146

X-VLA

0.051

Figure 7 values, rounded to three decimals; all models use chunk 1. Exact EgoForce values: r 0.4367, F1 0.8737.

Paper Table 1 · chunk size 1

Model	Total parameters	Contact MAE (N) ↓	Contact R² ↑	Contact Pearson r ↑	Pooled F1 ↑
EgoForce (MoE)	36.18 M	7.2846	0.1950	0.2992	0.8677
EgoForce w/o MoE	36.04 M	7.2916	0.1841	0.3094	0.8591
ACT	34 M	8.4400	0.0670	0.1933	0.6492
WALL-X	4.2 B	7.8986	0.0860	0.1226	0.8322
OpenTouch	87 M	7.7258	0.1044	0.1807	0.8110
GR00T	~3 B	7.6107	0.0157	0.2286	0.8310
Pi0.5	~3.6 B	9.1171	-0.1236	0.1286	0.8318
Pi0	~3.5 B	9.5923	-0.1654	0.1102	0.8124
SmolVLA	460 M	9.4473	-0.1834	0.1017	0.7882
X-VLA	880 M	11.4716	-0.5462	0.0415	0.8217

The residual model is best on Test MAE, R² and F1; the shared-head model leads on contact r. Models are evaluated from seed-1000, 10,000-step checkpoints, not repeated-seed means.

Default-chunk results · same Table 1

Model	Chunk size	Contact MAE (N) ↓	Contact R² ↑	Contact Pearson r ↑	Pooled F1 ↑
ACT	100	8.4349	0.0905	0.2022	0.8209
WALL-X	32	8.8121	-0.0515	0.0676	0.8245
Pi0.5	50	10.1447	-0.2900	0.0695	0.8332
Pi0	50	9.8835	-0.2325	0.0862	0.8295
SmolVLA	50	9.7020	-0.2393	0.0908	0.7840
X-VLA	30	9.7409	-0.3939	-0.0463	0.8246
GR00T	16	7.8585	-0.0393	0.1663	0.8326

Per-material contact MAE · paper Figure 5 (N ↓)

Model	Jelly	Sponge	Empty Bottle	Full Bottle	Filled Milk	Chip	P25	P50	P75
EgoForce	6.98	1.47	2.01	6.87	23.48	1.26	16.19	9.93	4.45
ACT	8.10	2.34	4.47	9.37	23.79	1.82	16.75	9.30	7.62
WALL-X	8.07	2.50	3.34	7.53	29.54	3.15	15.24	9.89	5.46
OpenTouch	7.44	1.66	2.33	8.57	32.49	1.60	15.47	9.38	5.89
GR00T	9.36	1.59	2.07	7.70	31.47	1.87	16.43	8.32	4.71
Pi0	9.71	3.08	3.80	8.72	34.60	3.67	15.51	14.59	6.84

Displayed models and two-decimal values follow Figure 5. EgoForce has the lowest MAE in 7 of these 9 categories; WALL-X leads on P25 Bottle and GR00T on P50 Bottle.

OOD on empty opaque milk containers

261 episodes / 7,022 frames across multiple demonstrators and four source sessions. These are held-out container conditions, not a held-out-person test.

Empty-Milk OOD · contact r ↑

EgoForce

0.437

ACT

0.219

WALL-X

0.241

OpenTouch

0.320

GR00T

0.340

Pi0.5

0.155

Pi0

0.223

SmolVLA

0.146

X-VLA

0.051

Figure 7 values, rounded to three decimals; all models use chunk 1. Exact EgoForce values: r 0.4367, F1 0.8737.

Empty-Milk OOD · pooled F1 ↑

EgoForce

0.874

ACT

0.549

WALL-X

0.845

OpenTouch

0.836

GR00T

0.857

Pi0.5

0.843

Pi0

0.812

SmolVLA

0.831

X-VLA

0.840

Figure 7 values, rounded to three decimals; all models use chunk 1. Exact EgoForce values: r 0.4367, F1 0.8737.

All benchmark configurations have negative contact R² on OOD. GR00T (chunk 1) has lower MAE (4.8675 N) and higher R² (−1.0332) than EgoForce, but lower r (0.3405) and F1 (0.8574). Default-chunk X-VLA (30) reaches MAE 4.5071 N / R² −0.9889, with negative r −0.0342. Magnitude, trends and contact detection must be read together.

Residual-head ablation · paper Table 5

Shared encoder, GRU, optimizer and force loss; seed 1000, 10,000 steps. This isolates the residual force-mean correction.

Split	Variant	Contact MAE (N) ↓	Contact R² ↑	Contact Pearson r ↑	Pooled F1 ↑
Test	EgoForce	7.2846	0.1950	0.2992	0.8677
Test	w/o MoE	7.2916	0.1841	0.3094	0.8591
OOD	EgoForce	5.2113	-3.2403	0.4367	0.8737
OOD	w/o MoE	5.5983	-3.8860	0.3706	0.8696

Expert-count comparison · paper Table 6

Split	K	top-k	Contact MAE (N) ↓	Contact R² ↑
Test	0	—	7.2916	0.1841
Test	2	2	7.2846	0.1950
Test	4	2	7.3342	0.1905
OOD	0	—	5.5983	-3.8860
OOD	2	2	5.2113	-3.2403
OOD	4	2	5.1035	-1.2025

K=2 uses both experts for every sample and is the final 36.18 M model. K=4 uses top-2 routing: it is worse on all four Test metrics, but has lower OOD MAE and higher OOD R²/F1; OOD r is slightly lower. There is no universal winner across splits and metrics.

Existing qualitative clips are retained as demonstrations. Their checkpoint provenance is not established as the latest ICLR 10,000-step models; use the tables above for current quantitative comparisons.

Fragile

Deformable

Opaque containers

Variable-Load

Nine model overlays from the existing demo_v6 assets; these videos are not used as evidence for the updated numerical results.

Real-world experiments

Real-world validation: tactile feedback enables precise force control

Given a prescribed target force, can a real gripper execute it accurately? Electronic skin on both fingers closes the feedback loop for 1–5 N tracking and gentle manipulation.

What does this experiment establish?

It demonstrates that tactile feedback can turn prescribed force targets into accurate physical execution. Tracking a tissue package provides quantitative evidence (final-window MAE: 0.0207 N); lifting a potato chip without visible fracture illustrates gentle manipulation qualitatively.

The experiments address complementary questions: offline evaluation measures force-prediction accuracy; simulation measures how force-command errors affect task success; the real-gripper study measures how accurately the hardware follows a given target force.

0.0487 NFull-interval tracking MAE2,457 samples · transitions included

0.0207 NFinal-window tracking MAE412 samples · last 10 s per interval

0.0291 NFinal-window tracking RMSEAcross all nine target intervals

92.7%Within ±0.05 N of targetFinal-window samples only

Electronic-skin gripper holding a tissue package beside recorded target, reference and feedback force curves, rising from 1 to 5 N and returning to 1 N — Paper Fig. 9. Left: the gripper setup. Right: measured mean fingertip force against the prescribed target and rate-limited reference. Time begins after initial grasp establishment; the reference changes at 0.5 N/s.

Robot platform: Synria Robotics Alicia M series, equipped with an electronic-skin gripper for bilateral tactile feedback.

One continuous sequence, nine force intervals

1 → 2 → 3 → 4 → 5 → 4 → 3 → 2 → 1 N

After establishing a stable 1 N grasp, the same controller tracks nine targets for 60 s each. Feedback is the mean of the two tactile readings, acquired at approximately 4.54 Hz. Each final window is the fixed last 10 s, rather than a manually selected stable segment.

Four real-gripper frames showing a potato chip before grasping, during grasping, lifting and suspended holding without visible fracture — Paper Fig. 10. Before grasping → grasping → lifting → suspended holding. The chip remains without visible fracture; this is a qualitative demonstration, with no task success rate reported.

Per-stage measurements · paper Table 8

Target change (N)	Final-window mean (N)	Final-window MAE (N) ↓	Within ±0.05 N (%) ↑
1	1.0101	0.0101	100.0
1 → 2	2.0007	0.0143	100.0
2 → 3	2.9866	0.0134	100.0
3 → 4	4.0001	0.0062	100.0
4 → 5	5.0004	0.0186	95.7
5 → 4	3.9688	0.0312	97.7
4 → 3	3.0004	0.0179	95.7
3 → 2	2.0255	0.0481	60.5
2 → 1	0.9998	0.0308	81.4

Scope: the real-gripper targets are prescribed, rather than supplied by EgoForce. This validates force execution, but does not establish improved robot task success from EgoForce predictions. Connecting the estimator to the controller and comparing task outcomes remains future work. Source: paper §5.4 and Appendix D. Force values use the sensor’s protocol conversion; independent absolute-force calibration was not performed in this run.

Overview

The Physical Intuition Gap

Human Internal Model

Force estimation remains a separate challenge

Force-estimation accuracy materially changes manipulation success

The FEEL Benchmark & Data Engine

Data Collection Pipeline

Meta Aria Glasses

3D-Print Blueprint

Paxini PX6AX Tactile Sensor

Spatio-Temporal Sync

FEEL Dataset

FEEL benchmark at a glance

Train / Test / OOD

F/D/V Physical Taxonomy

Fragile

Deformable

Variable-Load

Three Controlled Probe Types

Boundary Samples

Water-Level Prior Samples

Visual-Physical Mismatch

Egocentric Footage

Technical Contributions for Force Learning

Finger-specific shared Laplace heads

7-D fingertip geometry and four-frame GRU

Two softly weighted residual MLP experts

EgoForce: shared temporal features, residual force refinement

Offline Evaluation Results

Test: contact magnitude and detection

OOD: trend tracking and contact detection

Residual experts: small cost, mixed Test gains

Test contact MAE · N ↓

Empty-Milk OOD · contact r ↑

Paper Table 1 · chunk size 1

OOD on empty opaque milk containers

Empty-Milk OOD · contact r ↑

Empty-Milk OOD · pooled F1 ↑

Residual-head ablation · paper Table 5

Expert-count comparison · paper Table 6

Real-world validation: tactile feedback enables precise force control

What does this experiment establish?

One continuous sequence, nine force intervals

What FEEL Reveals

Model scale does not consistently predict force accuracy

A compact residual branch yields metric-specific gains

OOD trends and contact improve, magnitude remains difficult

Simulation and hardware test complementary links

Open-Source Hardware Blueprint

BibTeX