Figuring out Figures: Using Textual References to Caption Scientific Figures
<|reference_start|>Figuring out Figures: Using Textual References to Caption Scientific Figures: Figures are essential channels for densely communicating complex ideas in scientific papers. Previous work in automatically generating figure captions has been largely unsuccessful and has defaulted to using single-layer LSTMs, which no longer achieve state-of-the-art performance. In our work, we use the SciCap datasets curated by Hsu et al. and use a variant of a CLIP+GPT-2 encoder-decoder model with cross-attention to generate captions conditioned on the image. Furthermore, we augment our training pipeline by creating a new dataset MetaSciCap that incorporates textual metadata from the original paper relevant to the figure, such as the title, abstract, and in-text references. We use SciBERT to encode the textual metadata and use this encoding alongside the figure embedding. In our experimentation with different models, we found that the CLIP+GPT-2 model performs better when it receives all textual metadata from the SciBERT encoder in addition to the figure, but employing a SciBERT+GPT2 model that uses only the textual metadata achieved optimal performance.<|reference_end|>
@article{cao2024figuring,
    title={Figuring out Figures: Using Textual References to Caption Scientific
  Figures},
    author={Stanley Cao and Kevin Liu},
    journal={arXiv preprint arXiv:2407.11008},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.11008},
    primaryClass={cs.CL cs.AI cs.CV}
}