On the Robustness of Language Guidance for Low-Level Vision Tasks:
  Findings from Depth Estimation
Recent advances in monocular depth estimation have been made by incorporating
natural language as additional guidance. Although yielding impressive results,
the impact of the language prior, particularly in terms of generalization and
robustness, remains unexplored. In this paper, we address this gap by
quantifying the impact of this prior and introduce methods to benchmark its
effectiveness across various settings. We generate "low-level" sentences that
convey object-centric, three-dimensional spatial relationships, incorporate
them as additional language priors and evaluate their downstream impact on
depth estimation. Our key finding is that current language-guided depth
estimators perform optimally only with scene-level descriptions and
counter-intuitively fare worse with low level descriptions. Despite leveraging
additional data, these methods are not robust to directed adversarial attacks
and decline in performance with an increase in distribution shift. Finally, to
provide a foundation for future research, we identify points of failures and
offer insights to better understand these shortcomings. With an increasing
number of methods using language for depth estimation, our findings highlight
the opportunities and pitfalls that require careful consideration for effective
deployment in real-world settings
661c87a00cc7cf6c18a2aec1
Agneet Chatterjee
extracted_confirmed
2024-08-06T02:12:46.808Z
6320c537a023aad6a7680c8b
/avatars/057dc492b8f756b83f12ced0b74fae65.svg
Agneet Chatterjee
user
agneet
661c87a00cc7cf6c18a2aec2
Tejas Gokhale
admin_assigned
2024-04-18T11:44:04.772Z
630440407373aacccd8a0897
/avatars/593e71241f4365f41aa6f812506ac640.svg
Tejas Gokhale
user
tgokhale
661c87a00cc7cf6c18a2aec3
Chitta Baral
extracted_pending
2024-04-15T01:49:21.661Z
62687430dc80d589a9f6cd45
/avatars/fe56b8c92657e4f210c86cc097d42813.svg
Chitta Baral
user
cogint-asu
661c87a00cc7cf6c18a2aec4
Yezhou Yang
extracted_confirmed
2024-04-15T01:53:05.210Z
6486a850903d5878eb98110f
https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
'YZ' Yezhou Yang
user
AiWithoutBorders
On the Robustness of Language Guidance for Low-Level Vision Tasks:
  Findings from Depth Estimation
Recent advances in monocular depth estimation have been made by incorporating
natural language as additional guidance. Although yielding impressive results,
the impact of the language prior, particularly in terms of generalization and
robustness, remains unexplored. In this paper, we address this gap by
quantifying the impact of this prior and introduce methods to benchmark its
effectiveness across various settings. We generate "low-level" sentences that
convey object-centric, three-dimensional spatial relationships, incorporate
them as additional language priors and evaluate their downstream impact on
depth estimation. Our key finding is that current language-guided depth
estimators perform optimally only with scene-level descriptions and
counter-intuitively fare worse with low level descriptions. Despite leveraging
additional data, these methods are not robust to directed adversarial attacks
and decline in performance with an increase in distribution shift. Finally, to
provide a foundation for future research, we identify points of failures and
offer insights to better understand these shortcomings. With an increasing
number of methods using language for depth estimation, our findings highlight
the opportunities and pitfalls that require careful consideration for effective
deployment in real-world settings
Recent monocular depth estimation techniques using language guidance show optimal performance with scene-level descriptions but degrade with low-level details and adversarial attacks.
monocular depth estimation natural language language prior object-centric three-dimensional spatial relationships scene-level descriptions adversarial attacks distribution shift