FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones
<|reference_start|>FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones: Progress in machine understanding of sign languages has been slow and hampered by limited data. In this paper, we present FSboard, an American Sign Language fingerspelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments. Fingerspelling recognition is an incomplete solution that is only one small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers as more broadly capable technology develops. At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks: plausible optimizations to help models run on device in real time.<|reference_end|>
@article{georg2024fsboard:,
    title={FSboard: Over 3 million characters of ASL fingerspelling collected via
  smartphones},
    author={Manfred Georg, Garrett Tanzer, Saad Hassan, Maximus Shengelia, Esha
  Uboweja, Sam Sepah, Sean Forbes, Thad Starner},
    journal={arXiv preprint arXiv:2407.15806},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15806},
    primaryClass={cs.CV cs.CL}
}