AudioBERT: Audio Knowledge Augmented Language Model
Recent studies have identified that language models, pretrained on text-only
datasets, often lack elementary visual knowledge, e.g., colors of
everyday objects. Motivated by this observation, we ask whether a similar
shortcoming exists in terms of the auditory knowledge. To answer this
question, we construct a new dataset called AuditoryBench, which consists of
two novel tasks for evaluating auditory knowledge. Based on our analysis using
the benchmark, we find that language models also suffer from a severe lack of
auditory knowledge. To address this limitation, we propose AudioBERT, a novel
method to augment the auditory knowledge of BERT through a retrieval-based
approach. First, we detect auditory knowledge spans in prompts to query our
retrieval model efficiently. Then, we inject audio knowledge into BERT and
switch on low-rank adaptation for effective adaptation when audio knowledge is
required. Our experiments demonstrate that AudioBERT is quite effective,
achieving superior performance on the AuditoryBench. The dataset and code are
available at https://github.com/HJ-Ok/AudioBERT.
66e949bd1fd0c4f1a55cc24a
Hyunjong Ok
claimed_verified
2024-09-17T09:30:25.970Z
631974d51328b6caf9fe328f
/avatars/5d7b54d2798d9e42d2db66cdba24e085.svg
Hyunjong Ok
user
HJOK
66e949bd1fd0c4f1a55cc24b
Suho Yoo
admin_assigned
2024-09-17T09:39:29.808Z
651930780e3a5553d4a04c66
/avatars/35f1a91a2ed920253274c010abc6741d.svg
suho yoo
user
uso7d0
66e949bd1fd0c4f1a55cc24c
Jaeho Lee
AudioBERT: Audio Knowledge Augmented Language Model
Recent studies have identified that language models, pretrained on text-only
datasets, often lack elementary visual knowledge, e.g., colors of
everyday objects. Motivated by this observation, we ask whether a similar
shortcoming exists in terms of the auditory knowledge. To answer this
question, we construct a new dataset called AuditoryBench, which consists of
two novel tasks for evaluating auditory knowledge. Based on our analysis using
the benchmark, we find that language models also suffer from a severe lack of
auditory knowledge. To address this limitation, we propose AudioBERT, a novel
method to augment the auditory knowledge of BERT through a retrieval-based
approach. First, we detect auditory knowledge spans in prompts to query our
retrieval model efficiently. Then, we inject audio knowledge into BERT and
switch on low-rank adaptation for effective adaptation when audio knowledge is
required. Our experiments demonstrate that AudioBERT is quite effective,
achieving superior performance on the AuditoryBench. The dataset and code are
available at https://github.com/HJ-Ok/AudioBERT.
AudioBERT enhances BERT's auditory knowledge using a retrieval-based approach, significantly improving performance on a newly constructed AuditoryBench dataset.
language models auditory knowledge AuditoryBench AudioBERT retrieval-based approach auditory knowledge spans low-rank adaptation