VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language
  Models for Trait Discovery from Biological Images
Images are increasingly becoming the currency for documenting biodiversity on
the planet, providing novel opportunities for accelerating scientific
discoveries in the field of organismal biology, especially with the advent of
large vision-language models (VLMs). We ask if pre-trained VLMs can aid
scientists in answering a range of biologically relevant questions without any
additional fine-tuning. In this paper, we evaluate the effectiveness of 12
state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel
dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images
from three groups of organisms: fishes, birds, and butterflies, covering five
biologically relevant tasks. We also explore the effects of applying prompting
techniques and tests for reasoning hallucination on the performance of VLMs,
shedding new light on the capabilities of current SOTA VLMs in answering
biologically relevant questions using images. The code and datasets for running
all the analyses reported in this paper can be found at
https://github.com/sammarfy/VLM4Bio.
66d57e0c64c1e9b732b85e66
M. Maruf
admin_assigned
2024-09-02T09:26:01.849Z
6491f54f85d19ff5a62b092b
/avatars/abcbb5c2f7fc167db9a834466154b05d.svg
M. Maruf
user
sammarfy
66d57e0c64c1e9b732b85e67
Arka Daw
admin_assigned
2024-09-02T09:26:08.340Z
656ba0e17825b31010e6d969
/avatars/6ca50115bfdef4d26aad8e51a1bc5642.svg
Arka
user
arkadaw9
66d57e0c64c1e9b732b85e68
Kazi Sajeed Mehrab
admin_assigned
2024-09-02T09:26:24.181Z
633b52a43124a257cd20c922
/avatars/e8930ccaafa17eeaffda3d83d5f9ef59.svg
Kazi Sajeed Mehrab
user
ksmehrab
66d57e0c64c1e9b732b85e69
Harish Babu Manogaran
admin_assigned
2024-09-02T09:26:30.930Z
63a486f2769ff94bc946933c
/avatars/bc57f33b3d6e94902a990fbef28faf12.svg
Harish Babu Manogaran
user
harishB97
66d57e0c64c1e9b732b85e6a
Abhilash Neog
admin_assigned
2024-09-02T09:26:37.101Z
6563ba4a7061c2cbda323a84
/avatars/7b0a27e55d17cc2b2829942f871356c2.svg
Abhilash Neog
user
abhilashneog
66d57e0c64c1e9b732b85e6b
Medha Sawhney
admin_assigned
2024-09-02T09:26:43.344Z
6605ed14b85b7b4ea50de65c
/avatars/3ba48ae9b1f608192fc5884decb4128d.svg
Medha Sawhney
user
medha12
66d57e0c64c1e9b732b85e6c
Mridul Khurana
admin_assigned
2024-09-02T09:26:48.938Z
65147ce8873a37df0f6d0d9a
/avatars/2e07db81d49e39711c70e6a491ae5eee.svg
Mridul Khurana
user
mridulk
66d57e0c64c1e9b732b85e6d
James P. Balhoff
claimed_verified
2024-09-04T15:25:07.295Z
64db9b65ff83b3386a4840b4
/avatars/7bc34ee9bf20b7a1ddb4c462d7e90426.svg
Jim Balhoff
user
balhoff
66d57e0c64c1e9b732b85e6e
Yasin Bakis
66d57e0c64c1e9b732b85e6f
Bahadir Altintas
66d57e0c64c1e9b732b85e70
Matthew J. Thompson
claimed_verified
2025-07-17T18:43:48.313Z
63eaa5f65c837d9968eea66c
https://cdn-avatars.huggingface.co/v1/production/uploads/1676322271087-noauth.jpeg
Matthew Thompson
user
thompsonmj
66d57e0c64c1e9b732b85e71
Elizabeth G. Campolongo
admin_assigned
2024-09-02T09:27:41.298Z
642dbba7c3a53b9e1d6e8726
https://cdn-avatars.huggingface.co/v1/production/uploads/642dbba7c3a53b9e1d6e8726/9ifixpoG2cu91WPAdsg-_.jpeg
Elizabeth Campolongo
user
egrace479
66d57e0c64c1e9b732b85e72
Josef C. Uyeda
admin_assigned
2024-09-02T09:27:50.166Z
62a23da28de73752aa5d37e0
/avatars/3bd2ac717ef5a1aae5f7405352ab7f0c.svg
Josef Uyeda
user
juyeda
66d57e0c64c1e9b732b85e73
Hilmar Lapp
admin_assigned
2024-09-02T09:27:55.923Z
6330c479c375586dc3b2aa2c
https://cdn-avatars.huggingface.co/v1/production/uploads/1664141174743-6330c479c375586dc3b2aa2c.jpeg
Hilmar Lapp
user
hlapp
66d57e0c64c1e9b732b85e74
Henry L. Bart
66d57e0c64c1e9b732b85e75
Paula M. Mabee
66d57e0c64c1e9b732b85e76
Yu Su
claimed_verified
2024-10-14T19:21:56.313Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
66d57e0c64c1e9b732b85e77
Wei-Lun Chao
admin_assigned
2024-09-02T09:28:14.898Z
66a144328cce07a5e6db1b0b
/avatars/4338bc493ab854f5f1b62f2a1b27c764.svg
Wei-Lun Chao
user
chao209
66d57e0c64c1e9b732b85e78
Charles Stewart
admin_assigned
2024-09-02T09:28:20.534Z
66aa43e3894e79991e585328
/avatars/98012824666b184d80175f7094a0e14f.svg
Charles Stewart
user
charless969
66d57e0c64c1e9b732b85e79
Tanya Berger-Wolf
66d57e0c64c1e9b732b85e7a
Wasila Dahdul
admin_assigned
2024-09-02T09:28:33.940Z
6570fd3bb6f7eb7f3109f41c
/avatars/09060022745673862523d08b9978b9a6.svg
Wasila Dahdul
user
wdahdul
66d57e0c64c1e9b732b85e7b
Anuj Karpatne
VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language
  Models for Trait Discovery from Biological Images
Images are increasingly becoming the currency for documenting biodiversity on
the planet, providing novel opportunities for accelerating scientific
discoveries in the field of organismal biology, especially with the advent of
large vision-language models (VLMs). We ask if pre-trained VLMs can aid
scientists in answering a range of biologically relevant questions without any
additional fine-tuning. In this paper, we evaluate the effectiveness of 12
state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel
dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images
from three groups of organisms: fishes, birds, and butterflies, covering five
biologically relevant tasks. We also explore the effects of applying prompting
techniques and tests for reasoning hallucination on the performance of VLMs,
shedding new light on the capabilities of current SOTA VLMs in answering
biologically relevant questions using images. The code and datasets for running
all the analyses reported in this paper can be found at
https://github.com/sammarfy/VLM4Bio.
Pre-trained vision-language models are evaluated for their effectiveness in answering biologically relevant questions using a novel dataset without additional fine-tuning, exploring prompting techniques and reasoning hallucination.
vision-language models (VLMs) VLM4Bio prompting techniques reasoning hallucination