Prompting Depth Anything for 4K Resolution Accurate Metric Depth
  Estimation
Prompts play a critical role in unleashing the power of language and vision
foundation models for specific tasks. For the first time, we introduce
prompting into depth foundation models, creating a new paradigm for metric
depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost
LiDAR as the prompt to guide the Depth Anything model for accurate metric depth
output, achieving up to 4K resolution. Our approach centers on a concise prompt
fusion design that integrates the LiDAR at multiple scales within the depth
decoder. To address training challenges posed by limited datasets containing
both LiDAR depth and precise GT depth, we propose a scalable data pipeline that
includes synthetic data LiDAR simulation and real data pseudo GT depth
generation. Our approach sets new state-of-the-arts on the ARKitScenes and
ScanNet++ datasets and benefits downstream applications, including 3D
reconstruction and generalized robotic grasping.
67638f91d5948d839a423200
Haotong Lin
admin_assigned
2024-12-20T09:52:43.197Z
6489a01b8de3f9d810b0154f
/avatars/f7a0fc6816535945e11bac1212dd7b57.svg
Haotong Lin
user
haotongl
67638f91d5948d839a423201
Sida Peng
admin_assigned
2024-12-20T09:52:36.196Z
62986ca2b58e71e2ac9b8f01
/avatars/83944db5f3dbb6f47c47c46fb2cb2849.svg
Sida Peng
user
pengsida
67638f91d5948d839a423202
Jingxiao Chen
67638f91d5948d839a423203
Songyou Peng
admin_assigned
2024-12-20T09:52:22.648Z
620cb0722d8bc91e1cf2e6e7
https://cdn-avatars.huggingface.co/v1/production/uploads/1644998719617-noauth.jpeg
Songyou Peng
user
syp
67638f91d5948d839a423204
Jiaming Sun
admin_assigned
2024-12-20T09:52:15.584Z
63141d8df839c69a68e73201
/avatars/b25fca555d22987241fd40bb56b8247a.svg
Jiaming  Sun
user
jiamingsun
67638f91d5948d839a423205
Minghuan Liu
admin_assigned
2024-12-20T09:52:02.965Z
630c8f0c8e3ff0c72327140a
/avatars/998de36d102b8bec85624f209272f7f2.svg
Minghuan Liu
user
ericonaldo
67638f91d5948d839a423206
Hujun Bao
67638f91d5948d839a423207
Jiashi Feng
admin_assigned
2024-12-20T09:51:49.781Z
67298e44017b96a1d0101dc4
/avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
Jiashi Feng
user
jshfeng
67638f91d5948d839a423208
Xiaowei Zhou
67638f91d5948d839a423209
Bingyi Kang
claimed_verified
2024-12-20T08:33:34.930Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
Prompting Depth Anything for 4K Resolution Accurate Metric Depth
  Estimation
Prompts play a critical role in unleashing the power of language and vision
foundation models for specific tasks. For the first time, we introduce
prompting into depth foundation models, creating a new paradigm for metric
depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost
LiDAR as the prompt to guide the Depth Anything model for accurate metric depth
output, achieving up to 4K resolution. Our approach centers on a concise prompt
fusion design that integrates the LiDAR at multiple scales within the depth
decoder. To address training challenges posed by limited datasets containing
both LiDAR depth and precise GT depth, we propose a scalable data pipeline that
includes synthetic data LiDAR simulation and real data pseudo GT depth
generation. Our approach sets new state-of-the-arts on the ARKitScenes and
ScanNet++ datasets and benefits downstream applications, including 3D
reconstruction and generalized robotic grasping.
Prompt Depth Anything integrates LiDAR data into depth foundation models for high-resolution depth estimation, achieving state-of-the-art results on ARKitScenes and ScanNet++ datasets and improving downstream applications.
prompting depth foundation models Prompt Depth Anything LiDAR depth decoder synthetic data LiDAR simulation pseudo GT depth generation ARKitScenes ScanNet++ 3D reconstruction generalized robotic grasping