Patch Policy: Efficient Embodied Control via Dense Visual Representations

PDF arXiv Code

Gaoyue Zhou1*, Zichen Jeff Cui1*, Ada Langford1, Bowen Tan1, Yann LeCun1,3, Lerrel Pinto1,2

1. Courant Institute, New York University 2. Meta-FAIR 3. AMI Labs

*Equal contribution.

Continuous Plug/Unplug Demo (2x)

Abstract

Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information.


Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control.

Method

In this work, we aim to give visuomotor policies direct access to the dense, spatially-rich features that large-scale Vision Transformer pre-training already provides. Instead of compressing each observation into a single global vector such as a pooled feature or a ViT CLS token, which discards the fine-grained detail needed for precise manipulation, Patch Policy feeds the full grid of patch features from a frozen, pretrained ViT to a lightweight transformer-based policy. A block-causal attention mask lets the model integrate spatial information within each frame while preserving temporal causality across frames.

Patch Policy method overview

Real Robot Environments

We evaluate Patch Policy on three real-world manipulation tasks: cable insertion, pen collection, and tool hanging. The rollouts below are examples from our human-collected demonstration data for each task, and the sections that follow break down where Patch Policy succeeds and fails against the baselines.

Cable Insertion (10x)

Pen Collection (10x)

Tool Hanging (10x)

Simulated Environments

The videos below are sample rollouts from each of our four simulated environment datasets. The sections that follow show how Patch Policy performs on each task.

Push-T
Libero Goal
BlockPush
Cube

Push-T

Patch Policy VQ-BeT DINOv2 (Ours)
Patch Policy DP DINOv2 (Ours)

Libero Goal

Patch Policy VQ-BeT DINOv2 (Ours)
Patch Policy DP DINOv2 (Ours)

BlockPush

Patch Policy VQ-BeT DINOv2 (Ours)
Patch Policy DP DINOv2 (Ours)

Cube

Patch Policy VQ-BeT DINOv2 (Ours)
Patch Policy DP DINOv2 (Ours)

Citation

@misc{zhou2026patchpolicyefficientembodied,
      title={Patch Policy: Efficient Embodied Control via Dense Visual Representations},
      author={Gaoyue Zhou and Zichen Jeff Cui and Ada Langford and Bowen Tan and Yann LeCun and Lerrel Pinto},
      year={2026},
      eprint={2607.18236},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.18236},
}