Resource eﬃciency in LLMs is a crucial and complex area that demands innovative
solutions to address signiﬁcant challenges. These challenges, more pronounced in LLMs
than in smaller neural networks like CNNs and MLPs, arise from the unique scale and
complexity of LLMs. We outline these challenges from various key perspectives:
• [Model] 1. Low parallelism in auto-regressive generation: Auto-regressive token
generation, the predominant method in LLMs, suﬀers from signiﬁcant latency due
to poor parallelism [10]. This is especially problematic for large model sizes or exten-
sive input lengths, hindering eﬃcient processing in both training and inference. 2.
Quadratic complexity in self-attention layers: The multi-head self-attention layer in
LLMs exhibits quadratic complexity with respect to the input sequence length [11].
This complexity creates a computational bottleneck as the input length increases,
limiting the practical input sequence length LLMs can handle eﬃciently.
Beyond Efficiency