Dolphin: Long Context as a New Modality for Energy-Efficient On-Device
  Language Models
This paper presents Dolphin, a novel decoder-decoder architecture for
energy-efficient processing of long contexts in language models. Our approach
addresses the significant energy consumption and latency challenges inherent in
on-device models. Dolphin employs a compact 0.5B parameter decoder to distill
extensive contextual information into a memory embedding, substantially
reducing the input length for the primary 7B parameter decoder model. Inspired
by vision-language models, we repurpose the image embedding projector to encode
long textual contexts, effectively treating extended context as a distinct
modality. This innovative method enables processing of substantially longer
contexts without the typical computational overhead associated with extended
input sequences. Empirical evaluations demonstrate a 10-fold improvement in
energy efficiency and a 5-fold reduction in latency compared to conventional
full-length context processing methods without losing quality of the response.
Our work contributes to the development of more sustainable and scalable
language models for on-device applications, addressing the critical need for
energy-efficient and responsive AI technologies in resource-constrained
environments while maintaining the accuracy to understand long contexts. This
research has implications for the broader field of natural language processing,
particularly in the domain of efficient model design for resource-limited
settings. By enabling more sophisticated AI capabilities on edge devices,
Dolphin paves the way for advanced language processing in a wide range of
applications where computational resources are at a premium. The Dolphin model
is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
66cfc67e9e405f72dd76797f
Wei Chen
claimed_verified
2024-08-29T07:23:18.559Z
65c082c626c639b0009a3c82
https://cdn-avatars.huggingface.co/v1/production/uploads/65c082c626c639b0009a3c82/w_Q_5iLOM-nh_drd9Tu4X.jpeg
Alex Chen
user
alexchen4ai
66cfc67e9e405f72dd767980
Zhiyuan Li
claimed_verified
2024-08-29T07:23:20.611Z
644de55076c0ab1880b9cb4d
https://cdn-avatars.huggingface.co/v1/production/uploads/644de55076c0ab1880b9cb4d/zKm7baOIzt_a4py8mYkck.jpeg
Zack Li
user
zackli4ai
66cfc67e9e405f72dd767981
Shuo Xin
admin_assigned
2024-08-29T07:33:25.418Z
63fc3f240aa18292d5c830f3
/avatars/66a10e1a40448cf923df7434244144f8.svg
Shuo Xin
user
xinshuo
66cfc67e9e405f72dd767982
Yihao Wang
admin_assigned
2024-08-29T07:33:15.390Z
63db7b7c9fb38bae84aa1706
https://cdn-avatars.huggingface.co/v1/production/uploads/63db7b7c9fb38bae84aa1706/PMuIT_EzPLbQIUCTkr2pd.png
Yihao Wang
user
AgainstEntropy
Dolphin: Long Context as a New Modality for Energy-Efficient On-Device
  Language Models
This paper presents Dolphin, a novel decoder-decoder architecture for
energy-efficient processing of long contexts in language models. Our approach
addresses the significant energy consumption and latency challenges inherent in
on-device models. Dolphin employs a compact 0.5B parameter decoder to distill
extensive contextual information into a memory embedding, substantially
reducing the input length for the primary 7B parameter decoder model. Inspired
by vision-language models, we repurpose the image embedding projector to encode
long textual contexts, effectively treating extended context as a distinct
modality. This innovative method enables processing of substantially longer
contexts without the typical computational overhead associated with extended
input sequences. Empirical evaluations demonstrate a 10-fold improvement in
energy efficiency and a 5-fold reduction in latency compared to conventional
full-length context processing methods without losing quality of the response.
Our work contributes to the development of more sustainable and scalable
language models for on-device applications, addressing the critical need for
energy-efficient and responsive AI technologies in resource-constrained
environments while maintaining the accuracy to understand long contexts. This
research has implications for the broader field of natural language processing,
particularly in the domain of efficient model design for resource-limited
settings. By enabling more sophisticated AI capabilities on edge devices,
Dolphin paves the way for advanced language processing in a wide range of
applications where computational resources are at a premium. The Dolphin model
is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
Dolphin, a novel decoder-decoder architecture, enables efficient processing of long contexts in language models by distilling contextual information into memory embeddings, thereby improving energy efficiency and latency without sacrificing accuracy.
decoder-decoder architecture energy-efficient processing long contexts language models memory embedding image embedding projector computational overhead empirical evaluations energy efficiency latency natural language processing efficient model design resource-limited settings edge devices