CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
ZeMing Gong Austin Wang Xiaoliang Huo Joakim Bruslund Haurum Scott C. Lowe Graham W. Taylor Angel X Chang
Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multi-modal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.
Multimodal Learning, Contrastive Learning, DNA Barcodes, Taxonomic Classification, Fine-grained Classification, Biodiversity Monitoring
CLIBD uses a multimodal machine learning approach with images, DNA, and text to enhance insect classification, outperforming traditional methods in identifying known and unknown species without specific fine-tuning.