10.1 Managing resource type disagreements

A complex challenge in optimizing Large Language Models (LLMs) is the reconciliation
of conﬂicting resource demands. Various optimization techniques introduce trade-oﬀs
between diﬀerent performance metrics [124, 133, 149, 156]. For instance, some meth-
ods might enhance computational eﬃciency at the cost of increasing the number of
model parameters or vice versa [106, 136]. Additionally, strategies that reduce com-
putational operations could inadvertently lead to higher memory requirements due
to the complexities involved in managing sparse data structures [117, 124]. The key
challenge, therefore, is to develop a holistic optimization strategy for LLMs. This strat-
egy should balance multiple objectives, including computational eﬃciency, parameter
count, and memory usage, ensuring that improvements in one area do not dispro-
portionately compromise performance in another. Achieving this balance is essential
Beyond Efficiency