SwiGLU activation function [PaLM]. We re-
place the ReLU non-linearity by the SwiGLU ac-
tivation function, introduced by Shazeer (2020) to
improve the performance. We use a dimension of
3 4d instead of 4d as in PaLM.
2
Rotary Embeddings [GPTNeo]. We remove the
absolute positional embeddings, and instead, add
rotary positional embeddings (RoPE), introduced
by Su et al. (2021), at each layer of the network.

The details of the hyper-parameters for our dif-

ferent models are given in Table 2.

2.3 Optimizer
Our models are trained using the AdamW opti-
mizer (Loshchilov and Hutter, 2017), with the fol-
lowing hyper-parameters: β1 = 0.9, β2 = 0.95.
We use a cosine learning rate schedule, such that
the ﬁnal learning rate is equal to 10% of the maxi-
mal learning rate. We use a weight decay of 0.1 and
gradient clipping of 1.0. We use 2, 000 warmup
LLaMA- Open and Efficient Foundation Language Models