Token Bottleneck: One Token to Remember Dynamics
Deriving compact and temporally aware visual representations from dynamic
scenes is essential for successful execution of sequential scene understanding
tasks such as visual tracking and robotic manipulation. In this paper, we
introduce Token Bottleneck (ToBo), a simple yet intuitive self-supervised
learning pipeline that squeezes a scene into a bottleneck token and predicts
the subsequent scene using minimal patches as hints. The ToBo pipeline
facilitates the learning of sequential scene representations by conservatively
encoding the reference scene into a compact bottleneck token during the squeeze
step. In the expansion step, we guide the model to capture temporal dynamics by
predicting the target scene using the bottleneck token along with few target
patches as hints. This design encourages the vision backbone to embed temporal
dependencies, thereby enabling understanding of dynamic transitions across
scenes. Extensive experiments in diverse sequential tasks, including video
label propagation and robot manipulation in simulated environments demonstrate
the superiority of ToBo over baselines. Moreover, deploying our pre-trained
model on physical robots confirms its robustness and effectiveness in
real-world environments. We further validate the scalability of ToBo across
different model scales.
686f9aad706a6ea465418a08
Taekyung Kim
claimed_verified
2025-07-11T08:03:22.401Z
67c6a1e75e2443d7d5f85cb3
/avatars/0569b368520411ab828d46725bc3896a.svg
Taekyung Kim
user
taekyung-k
686f9aad706a6ea465418a09
Dongyoon Han
claimed_verified
2025-07-11T08:03:14.509Z
660f8cc1a61244f3df3d4426
/avatars/45d59766122bb3482f6dd7f9d98aa87a.svg
Dongyoon Han
user
calintz
686f9aad706a6ea465418a0a
Byeongho Heo
claimed_verified
2025-07-11T08:03:16.434Z
64b9feed96676e40d0fa89a7
/avatars/2154a6ceb87677ad2c9d9620de5b18ec.svg
Byeongho Heo
user
bhheo
686f9aad706a6ea465418a0b
Jeongeun Park
686f9aad706a6ea465418a0c
Sangdoo Yun
claimed_verified
2025-07-13T07:26:29.372Z
648ac415718bc0670a5a5f56
/avatars/27189e289b808ef01689ff2abb7a56bf.svg
Sangdoo Yun
user
oodgnas
Token Bottleneck: One Token to Remember Dynamics
Deriving compact and temporally aware visual representations from dynamic
scenes is essential for successful execution of sequential scene understanding
tasks such as visual tracking and robotic manipulation. In this paper, we
introduce Token Bottleneck (ToBo), a simple yet intuitive self-supervised
learning pipeline that squeezes a scene into a bottleneck token and predicts
the subsequent scene using minimal patches as hints. The ToBo pipeline
facilitates the learning of sequential scene representations by conservatively
encoding the reference scene into a compact bottleneck token during the squeeze
step. In the expansion step, we guide the model to capture temporal dynamics by
predicting the target scene using the bottleneck token along with few target
patches as hints. This design encourages the vision backbone to embed temporal
dependencies, thereby enabling understanding of dynamic transitions across
scenes. Extensive experiments in diverse sequential tasks, including video
label propagation and robot manipulation in simulated environments demonstrate
the superiority of ToBo over baselines. Moreover, deploying our pre-trained
model on physical robots confirms its robustness and effectiveness in
real-world environments. We further validate the scalability of ToBo across
different model scales.
ToBo, a self-supervised learning pipeline, generates compact and temporally aware visual representations by encoding scenes into a bottleneck token and predicting subsequent scenes with minimal hints, demonstrating superior performance in sequential tasks.
Token Bottleneck self-supervised learning bottleneck token sequential scene representations temporal dynamics video label propagation robot manipulation simulated environments real-world environments scalability