title: AudioBERT: Audio Knowledge Augmented Language Model
summary: Recent studies have identified that language models, pretrained on text-only
  datasets, often lack elementary visual knowledge, e.g., colors of
  everyday objects. Motivated by this observation, we ask whether a similar
  shortcoming exists in terms of the auditory knowledge. To answer this
  question, we construct a new dataset called AuditoryBench, which consists of
  two novel tasks for evaluating auditory knowledge. Based on our analysis using
  the benchmark, we find that language models also suffer from a severe lack of
  auditory knowledge. To address this limitation, we propose AudioBERT, a novel
  method to augment the auditory knowledge of BERT through a retrieval-based
  approach. First, we detect auditory knowledge spans in prompts to query our
  retrieval model efficiently. Then, we inject audio knowledge into BERT and
  switch on low-rank adaptation for effective adaptation when audio knowledge is
  required. Our experiments demonstrate that AudioBERT is quite effective,
  achieving superior performance on the AuditoryBench. The dataset and code are
  available at https://github.com/HJ-Ok/AudioBERT.
paper_authors._id: 66e949bd1fd0c4f1a55cc24a
paper_authors.name: Hyunjong Ok
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T09:30:25.970Z
paper_authors._id: 631974d51328b6caf9fe328f
paper_authors.avatarUrl: /avatars/5d7b54d2798d9e42d2db66cdba24e085.svg
paper_authors.fullname: Hyunjong Ok
paper_authors.type: user
paper_authors.user: HJOK
paper_authors._id: 66e949bd1fd0c4f1a55cc24b
paper_authors.name: Suho Yoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T09:39:29.808Z
paper_authors._id: 651930780e3a5553d4a04c66
paper_authors.avatarUrl: /avatars/35f1a91a2ed920253274c010abc6741d.svg
paper_authors.fullname: suho yoo
paper_authors.type: user
paper_authors.user: uso7d0
paper_authors._id: 66e949bd1fd0c4f1a55cc24c
paper_authors.name: Jaeho Lee
paper_title: AudioBERT: Audio Knowledge Augmented Language Model
paper_summary: Recent studies have identified that language models, pretrained on text-only
  datasets, often lack elementary visual knowledge, e.g., colors of
  everyday objects. Motivated by this observation, we ask whether a similar
  shortcoming exists in terms of the auditory knowledge. To answer this
  question, we construct a new dataset called AuditoryBench, which consists of
  two novel tasks for evaluating auditory knowledge. Based on our analysis using
  the benchmark, we find that language models also suffer from a severe lack of
  auditory knowledge. To address this limitation, we propose AudioBERT, a novel
  method to augment the auditory knowledge of BERT through a retrieval-based
  approach. First, we detect auditory knowledge spans in prompts to query our
  retrieval model efficiently. Then, we inject audio knowledge into BERT and
  switch on low-rank adaptation for effective adaptation when audio knowledge is
  required. Our experiments demonstrate that AudioBERT is quite effective,
  achieving superior performance on the AuditoryBench. The dataset and code are
  available at https://github.com/HJ-Ok/AudioBERT.
paper_ai_summary: AudioBERT enhances BERT's auditory knowledge using a retrieval-based approach, significantly improving performance on a newly constructed AuditoryBench dataset.
paper_ai_keywords: language models auditory knowledge AuditoryBench AudioBERT retrieval-based approach auditory knowledge spans low-rank adaptation