title: VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language
    Models for Trait Discovery from Biological Images
summary: Images are increasingly becoming the currency for documenting biodiversity on
  the planet, providing novel opportunities for accelerating scientific
  discoveries in the field of organismal biology, especially with the advent of
  large vision-language models (VLMs). We ask if pre-trained VLMs can aid
  scientists in answering a range of biologically relevant questions without any
  additional fine-tuning. In this paper, we evaluate the effectiveness of 12
  state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel
  dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images
  from three groups of organisms: fishes, birds, and butterflies, covering five
  biologically relevant tasks. We also explore the effects of applying prompting
  techniques and tests for reasoning hallucination on the performance of VLMs,
  shedding new light on the capabilities of current SOTA VLMs in answering
  biologically relevant questions using images. The code and datasets for running
  all the analyses reported in this paper can be found at
  https://github.com/sammarfy/VLM4Bio.
paper_authors._id: 66d57e0c64c1e9b732b85e66
paper_authors.name: M. Maruf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:01.849Z
paper_authors._id: 6491f54f85d19ff5a62b092b
paper_authors.avatarUrl: /avatars/abcbb5c2f7fc167db9a834466154b05d.svg
paper_authors.fullname: M. Maruf
paper_authors.type: user
paper_authors.user: sammarfy
paper_authors._id: 66d57e0c64c1e9b732b85e67
paper_authors.name: Arka Daw
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:08.340Z
paper_authors._id: 656ba0e17825b31010e6d969
paper_authors.avatarUrl: /avatars/6ca50115bfdef4d26aad8e51a1bc5642.svg
paper_authors.fullname: Arka
paper_authors.type: user
paper_authors.user: arkadaw9
paper_authors._id: 66d57e0c64c1e9b732b85e68
paper_authors.name: Kazi Sajeed Mehrab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:24.181Z
paper_authors._id: 633b52a43124a257cd20c922
paper_authors.avatarUrl: /avatars/e8930ccaafa17eeaffda3d83d5f9ef59.svg
paper_authors.fullname: Kazi Sajeed Mehrab
paper_authors.type: user
paper_authors.user: ksmehrab
paper_authors._id: 66d57e0c64c1e9b732b85e69
paper_authors.name: Harish Babu Manogaran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:30.930Z
paper_authors._id: 63a486f2769ff94bc946933c
paper_authors.avatarUrl: /avatars/bc57f33b3d6e94902a990fbef28faf12.svg
paper_authors.fullname: Harish Babu Manogaran
paper_authors.type: user
paper_authors.user: harishB97
paper_authors._id: 66d57e0c64c1e9b732b85e6a
paper_authors.name: Abhilash Neog
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:37.101Z
paper_authors._id: 6563ba4a7061c2cbda323a84
paper_authors.avatarUrl: /avatars/7b0a27e55d17cc2b2829942f871356c2.svg
paper_authors.fullname: Abhilash Neog
paper_authors.type: user
paper_authors.user: abhilashneog
paper_authors._id: 66d57e0c64c1e9b732b85e6b
paper_authors.name: Medha Sawhney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:43.344Z
paper_authors._id: 6605ed14b85b7b4ea50de65c
paper_authors.avatarUrl: /avatars/3ba48ae9b1f608192fc5884decb4128d.svg
paper_authors.fullname: Medha Sawhney
paper_authors.type: user
paper_authors.user: medha12
paper_authors._id: 66d57e0c64c1e9b732b85e6c
paper_authors.name: Mridul Khurana
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:26:48.938Z
paper_authors._id: 65147ce8873a37df0f6d0d9a
paper_authors.avatarUrl: /avatars/2e07db81d49e39711c70e6a491ae5eee.svg
paper_authors.fullname: Mridul Khurana
paper_authors.type: user
paper_authors.user: mridulk
paper_authors._id: 66d57e0c64c1e9b732b85e6d
paper_authors.name: James P. Balhoff
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T15:25:07.295Z
paper_authors._id: 64db9b65ff83b3386a4840b4
paper_authors.avatarUrl: /avatars/7bc34ee9bf20b7a1ddb4c462d7e90426.svg
paper_authors.fullname: Jim Balhoff
paper_authors.type: user
paper_authors.user: balhoff
paper_authors._id: 66d57e0c64c1e9b732b85e6e
paper_authors.name: Yasin Bakis
paper_authors._id: 66d57e0c64c1e9b732b85e6f
paper_authors.name: Bahadir Altintas
paper_authors._id: 66d57e0c64c1e9b732b85e70
paper_authors.name: Matthew J. Thompson
paper_authors._id: 66d57e0c64c1e9b732b85e71
paper_authors.name: Elizabeth G. Campolongo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:27:41.298Z
paper_authors._id: 642dbba7c3a53b9e1d6e8726
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642dbba7c3a53b9e1d6e8726/9ifixpoG2cu91WPAdsg-_.jpeg
paper_authors.fullname: Elizabeth Campolongo
paper_authors.type: user
paper_authors.user: egrace479
paper_authors._id: 66d57e0c64c1e9b732b85e72
paper_authors.name: Josef C. Uyeda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:27:50.166Z
paper_authors._id: 62a23da28de73752aa5d37e0
paper_authors.avatarUrl: /avatars/3bd2ac717ef5a1aae5f7405352ab7f0c.svg
paper_authors.fullname: Josef Uyeda
paper_authors.type: user
paper_authors.user: juyeda
paper_authors._id: 66d57e0c64c1e9b732b85e73
paper_authors.name: Hilmar Lapp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:27:55.923Z
paper_authors._id: 6330c479c375586dc3b2aa2c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664141174743-6330c479c375586dc3b2aa2c.jpeg
paper_authors.fullname: Hilmar Lapp
paper_authors.type: user
paper_authors.user: hlapp
paper_authors._id: 66d57e0c64c1e9b732b85e74
paper_authors.name: Henry L. Bart
paper_authors._id: 66d57e0c64c1e9b732b85e75
paper_authors.name: Paula M. Mabee
paper_authors._id: 66d57e0c64c1e9b732b85e76
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:21:56.313Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 66d57e0c64c1e9b732b85e77
paper_authors.name: Wei-Lun Chao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:28:14.898Z
paper_authors._id: 66a144328cce07a5e6db1b0b
paper_authors.avatarUrl: /avatars/4338bc493ab854f5f1b62f2a1b27c764.svg
paper_authors.fullname: Wei-Lun Chao
paper_authors.type: user
paper_authors.user: chao209
paper_authors._id: 66d57e0c64c1e9b732b85e78
paper_authors.name: Charles Stewart
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:28:20.534Z
paper_authors._id: 66aa43e3894e79991e585328
paper_authors.avatarUrl: /avatars/98012824666b184d80175f7094a0e14f.svg
paper_authors.fullname: Charles Stewart
paper_authors.type: user
paper_authors.user: charless969
paper_authors._id: 66d57e0c64c1e9b732b85e79
paper_authors.name: Tanya Berger-Wolf
paper_authors._id: 66d57e0c64c1e9b732b85e7a
paper_authors.name: Wasila Dahdul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:28:33.940Z
paper_authors._id: 6570fd3bb6f7eb7f3109f41c
paper_authors.avatarUrl: /avatars/09060022745673862523d08b9978b9a6.svg
paper_authors.fullname: Wasila Dahdul
paper_authors.type: user
paper_authors.user: wdahdul
paper_authors._id: 66d57e0c64c1e9b732b85e7b
paper_authors.name: Anuj Karpatne
paper_title: VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language
    Models for Trait Discovery from Biological Images
paper_summary: Images are increasingly becoming the currency for documenting biodiversity on
  the planet, providing novel opportunities for accelerating scientific
  discoveries in the field of organismal biology, especially with the advent of
  large vision-language models (VLMs). We ask if pre-trained VLMs can aid
  scientists in answering a range of biologically relevant questions without any
  additional fine-tuning. In this paper, we evaluate the effectiveness of 12
  state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel
  dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images
  from three groups of organisms: fishes, birds, and butterflies, covering five
  biologically relevant tasks. We also explore the effects of applying prompting
  techniques and tests for reasoning hallucination on the performance of VLMs,
  shedding new light on the capabilities of current SOTA VLMs in answering
  biologically relevant questions using images. The code and datasets for running
  all the analyses reported in this paper can be found at
  https://github.com/sammarfy/VLM4Bio.
paper_ai_summary: Pre-trained vision-language models are evaluated for their effectiveness in answering biologically relevant questions using a novel dataset without additional fine-tuning, exploring prompting techniques and reasoning hallucination.
paper_ai_keywords: vision-language models (VLMs) VLM4Bio prompting techniques reasoning hallucination