title: On the Robustness of Language Guidance for Low-Level Vision Tasks:
    Findings from Depth Estimation
summary: Recent advances in monocular depth estimation have been made by incorporating
  natural language as additional guidance. Although yielding impressive results,
  the impact of the language prior, particularly in terms of generalization and
  robustness, remains unexplored. In this paper, we address this gap by
  quantifying the impact of this prior and introduce methods to benchmark its
  effectiveness across various settings. We generate "low-level" sentences that
  convey object-centric, three-dimensional spatial relationships, incorporate
  them as additional language priors and evaluate their downstream impact on
  depth estimation. Our key finding is that current language-guided depth
  estimators perform optimally only with scene-level descriptions and
  counter-intuitively fare worse with low level descriptions. Despite leveraging
  additional data, these methods are not robust to directed adversarial attacks
  and decline in performance with an increase in distribution shift. Finally, to
  provide a foundation for future research, we identify points of failures and
  offer insights to better understand these shortcomings. With an increasing
  number of methods using language for depth estimation, our findings highlight
  the opportunities and pitfalls that require careful consideration for effective
  deployment in real-world settings
paper_authors._id: 661c87a00cc7cf6c18a2aec1
paper_authors.name: Agneet Chatterjee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-06T02:12:46.808Z
paper_authors._id: 6320c537a023aad6a7680c8b
paper_authors.avatarUrl: /avatars/057dc492b8f756b83f12ced0b74fae65.svg
paper_authors.fullname: Agneet Chatterjee
paper_authors.type: user
paper_authors.user: agneet
paper_authors._id: 661c87a00cc7cf6c18a2aec2
paper_authors.name: Tejas Gokhale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:44:04.772Z
paper_authors._id: 630440407373aacccd8a0897
paper_authors.avatarUrl: /avatars/593e71241f4365f41aa6f812506ac640.svg
paper_authors.fullname: Tejas Gokhale
paper_authors.type: user
paper_authors.user: tgokhale
paper_authors._id: 661c87a00cc7cf6c18a2aec3
paper_authors.name: Chitta Baral
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-15T01:49:21.661Z
paper_authors._id: 62687430dc80d589a9f6cd45
paper_authors.avatarUrl: /avatars/fe56b8c92657e4f210c86cc097d42813.svg
paper_authors.fullname: Chitta Baral
paper_authors.type: user
paper_authors.user: cogint-asu
paper_authors._id: 661c87a00cc7cf6c18a2aec4
paper_authors.name: Yezhou Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-15T01:53:05.210Z
paper_authors._id: 6486a850903d5878eb98110f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
paper_authors.fullname: 'YZ' Yezhou Yang
paper_authors.type: user
paper_authors.user: AiWithoutBorders
paper_title: On the Robustness of Language Guidance for Low-Level Vision Tasks:
    Findings from Depth Estimation
paper_summary: Recent advances in monocular depth estimation have been made by incorporating
  natural language as additional guidance. Although yielding impressive results,
  the impact of the language prior, particularly in terms of generalization and
  robustness, remains unexplored. In this paper, we address this gap by
  quantifying the impact of this prior and introduce methods to benchmark its
  effectiveness across various settings. We generate "low-level" sentences that
  convey object-centric, three-dimensional spatial relationships, incorporate
  them as additional language priors and evaluate their downstream impact on
  depth estimation. Our key finding is that current language-guided depth
  estimators perform optimally only with scene-level descriptions and
  counter-intuitively fare worse with low level descriptions. Despite leveraging
  additional data, these methods are not robust to directed adversarial attacks
  and decline in performance with an increase in distribution shift. Finally, to
  provide a foundation for future research, we identify points of failures and
  offer insights to better understand these shortcomings. With an increasing
  number of methods using language for depth estimation, our findings highlight
  the opportunities and pitfalls that require careful consideration for effective
  deployment in real-world settings
paper_ai_keywords: monocular depth estimation natural language language prior generalization robustness low-level sentences object-centric three-dimensional spatial relationships scene-level descriptions directed adversarial attacks distribution shift