title: Dolphin: Long Context as a New Modality for Energy-Efficient On-Device
    Language Models
summary: This paper presents Dolphin, a novel decoder-decoder architecture for
  energy-efficient processing of long contexts in language models. Our approach
  addresses the significant energy consumption and latency challenges inherent in
  on-device models. Dolphin employs a compact 0.5B parameter decoder to distill
  extensive contextual information into a memory embedding, substantially
  reducing the input length for the primary 7B parameter decoder model. Inspired
  by vision-language models, we repurpose the image embedding projector to encode
  long textual contexts, effectively treating extended context as a distinct
  modality. This innovative method enables processing of substantially longer
  contexts without the typical computational overhead associated with extended
  input sequences. Empirical evaluations demonstrate a 10-fold improvement in
  energy efficiency and a 5-fold reduction in latency compared to conventional
  full-length context processing methods without losing quality of the response.
  Our work contributes to the development of more sustainable and scalable
  language models for on-device applications, addressing the critical need for
  energy-efficient and responsive AI technologies in resource-constrained
  environments while maintaining the accuracy to understand long contexts. This
  research has implications for the broader field of natural language processing,
  particularly in the domain of efficient model design for resource-limited
  settings. By enabling more sophisticated AI capabilities on edge devices,
  Dolphin paves the way for advanced language processing in a wide range of
  applications where computational resources are at a premium. The Dolphin model
  is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
paper_authors._id: 66cfc67e9e405f72dd76797f
paper_authors.name: Wei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:18.559Z
paper_authors._id: 65c082c626c639b0009a3c82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c082c626c639b0009a3c82/w_Q_5iLOM-nh_drd9Tu4X.jpeg
paper_authors.fullname: Alex Chen
paper_authors.type: user
paper_authors.user: alexchen4ai
paper_authors._id: 66cfc67e9e405f72dd767980
paper_authors.name: Zhiyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:20.611Z
paper_authors._id: 644de55076c0ab1880b9cb4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644de55076c0ab1880b9cb4d/zKm7baOIzt_a4py8mYkck.jpeg
paper_authors.fullname: Zack Li
paper_authors.type: user
paper_authors.user: zackli4ai
paper_authors._id: 66cfc67e9e405f72dd767981
paper_authors.name: Shuo Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:33:25.418Z
paper_authors._id: 63fc3f240aa18292d5c830f3
paper_authors.avatarUrl: /avatars/66a10e1a40448cf923df7434244144f8.svg
paper_authors.fullname: Shuo Xin
paper_authors.type: user
paper_authors.user: xinshuo
paper_authors._id: 66cfc67e9e405f72dd767982
paper_authors.name: Yihao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:33:15.390Z
paper_authors._id: 63db7b7c9fb38bae84aa1706
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63db7b7c9fb38bae84aa1706/PMuIT_EzPLbQIUCTkr2pd.png
paper_authors.fullname: Yihao Wang
paper_authors.type: user
paper_authors.user: AgainstEntropy
paper_title: Dolphin: Long Context as a New Modality for Energy-Efficient On-Device
    Language Models
paper_summary: This paper presents Dolphin, a novel decoder-decoder architecture for
  energy-efficient processing of long contexts in language models. Our approach
  addresses the significant energy consumption and latency challenges inherent in
  on-device models. Dolphin employs a compact 0.5B parameter decoder to distill
  extensive contextual information into a memory embedding, substantially
  reducing the input length for the primary 7B parameter decoder model. Inspired
  by vision-language models, we repurpose the image embedding projector to encode
  long textual contexts, effectively treating extended context as a distinct
  modality. This innovative method enables processing of substantially longer
  contexts without the typical computational overhead associated with extended
  input sequences. Empirical evaluations demonstrate a 10-fold improvement in
  energy efficiency and a 5-fold reduction in latency compared to conventional
  full-length context processing methods without losing quality of the response.
  Our work contributes to the development of more sustainable and scalable
  language models for on-device applications, addressing the critical need for
  energy-efficient and responsive AI technologies in resource-constrained
  environments while maintaining the accuracy to understand long contexts. This
  research has implications for the broader field of natural language processing,
  particularly in the domain of efficient model design for resource-limited
  settings. By enabling more sophisticated AI capabilities on edge devices,
  Dolphin paves the way for advanced language processing in a wide range of
  applications where computational resources are at a premium. The Dolphin model
  is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
paper_ai_keywords: decoder-decoder architecture energy-efficient processing long contexts language models parameter decoder memory embedding image embedding projector computational overhead extended input sequences energy efficiency latency natural language processing efficient model design edge devices