title: Complexity of Symbolic Representation in Working Memory of Transformer
    Correlates with the Complexity of a Task
summary: Even though Transformers are extensively used for Natural Language Processing
  tasks, especially for machine translation, they lack an explicit memory to
  store key concepts of processed texts. This paper explores the properties of
  the content of symbolic working memory added to the Transformer model decoder.
  Such working memory enhances the quality of model predictions in machine
  translation task and works as a neural-symbolic representation of information
  that is important for the model to make correct translations. The study of
  memory content revealed that translated text keywords are stored in the working
  memory, pointing to the relevance of memory content to the processed text.
  Also, the diversity of tokens and parts of speech stored in memory correlates
  with the complexity of the corpora for machine translation task.
paper_authors._id: 66753855e2494e5447dd7a5e
paper_authors.name: Alsu Sagirova
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-21T08:23:35.153Z
paper_authors._id: 65c0db0fbda79a18292dfbb7
paper_authors.avatarUrl: /avatars/1201b8282664c2d8c18beaba2396c03b.svg
paper_authors.fullname: Alsu Sagirova
paper_authors.type: user
paper_authors.user: alsu-sagirova
paper_authors._id: 66753855e2494e5447dd7a5f
paper_authors.name: Mikhail Burtsev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:01:43.806Z
paper_authors._id: 639c6e978a34ed9a404c6a7b
paper_authors.avatarUrl: /avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
paper_authors.fullname: MIKHAIL BURTSEV
paper_authors.type: user
paper_authors.user: mbur
paper_title: Complexity of Symbolic Representation in Working Memory of Transformer
    Correlates with the Complexity of a Task
paper_summary: Even though Transformers are extensively used for Natural Language Processing
  tasks, especially for machine translation, they lack an explicit memory to
  store key concepts of processed texts. This paper explores the properties of
  the content of symbolic working memory added to the Transformer model decoder.
  Such working memory enhances the quality of model predictions in machine
  translation task and works as a neural-symbolic representation of information
  that is important for the model to make correct translations. The study of
  memory content revealed that translated text keywords are stored in the working
  memory, pointing to the relevance of memory content to the processed text.
  Also, the diversity of tokens and parts of speech stored in memory correlates
  with the complexity of the corpora for machine translation task.
paper_ai_summary: The addition of symbolic working memory to Transformer decoders improves machine translation by storing key concepts, with memory content correlating to text complexity.
paper_ai_keywords: Transformers machine translation symbolic working memory neural-symbolic representation translated text keywords token diversity parts of speech corpus complexity