Prompt Engineering and Fine-Tuning. While GPT-3 175B can adapt its behavior with just a
few additional training examples, the result depends heavily on the input prompt (Brown et al.,
2020). This necessitates an empirical art of composing and formatting the prompt to maximize a
model’s performance on a desired task, which is known as prompt engineering or prompt hacking.
Fine-tuning retrains a model pre-trained on general domains to a speciﬁc task Devlin et al. (2019b);
Radford et al. (a). Variants of it include learning just a subset of the parameters Devlin et al. (2019b);
Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream
performance. However, the enormity of GPT-3 175B makes it challenging to perform ﬁne-tuning in
the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it
has the same memory footprint as pre-training.
LORA