title: Memory Augmented Language Models through Mixture of Word Experts
summary: Scaling up the number of parameters of language models has proven to be an
  effective approach to improve performance. For dense models, increasing model
  size proportionally increases the model's computation footprint. In this work,
  we seek to aggressively decouple learning capacity and FLOPs through
  Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
  based routing functions and experts. Our proposed approach, dubbed Mixture of
  Word Experts (MoWE), can be seen as a memory augmented model, where a large set
  of word-specific experts play the role of a sparse memory. We demonstrate that
  MoWE performs significantly better than the T5 family of models with similar
  number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
  regular MoE models on knowledge intensive tasks and has similar performance to
  more complex memory augmented approaches that often require to invoke custom
  mechanisms to search the sparse memory.
paper_authors._id: 655c1c3971ed242b42686669
paper_authors.name: Cicero Nogueira dos Santos
paper_authors._id: 655c1c3971ed242b4268666a
paper_authors.name: James Lee-Thorp
paper_authors._id: 655c1c3971ed242b4268666b
paper_authors.name: Isaac Noble
paper_authors._id: 655c1c3971ed242b4268666c
paper_authors.name: Chung-Ching Chang
paper_authors._id: 655c1c3971ed242b4268666d
paper_authors.name: David Uthus
paper_title: Memory Augmented Language Models through Mixture of Word Experts
paper_summary: Scaling up the number of parameters of language models has proven to be an
  effective approach to improve performance. For dense models, increasing model
  size proportionally increases the model's computation footprint. In this work,
  we seek to aggressively decouple learning capacity and FLOPs through
  Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
  based routing functions and experts. Our proposed approach, dubbed Mixture of
  Word Experts (MoWE), can be seen as a memory augmented model, where a large set
  of word-specific experts play the role of a sparse memory. We demonstrate that
  MoWE performs significantly better than the T5 family of models with similar
  number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
  regular MoE models on knowledge intensive tasks and has similar performance to
  more complex memory augmented approaches that often require to invoke custom
  mechanisms to search the sparse memory.
paper_ai_summary: A Mixture of Word Experts (MoWE) model demonstrates superior performance in NLP tasks compared to T5 models and traditional MoE models, using memory-augmented techniques and knowledge-rich routing functions.
paper_ai_keywords: Mixture-of-Experts MoE Mixture of Word Experts MoWE dense models FLOPs language models knowledge-intensive tasks memory augmented model sparse memory