Value
6144
24576
8192
48
40
Multi-query
≈15.5B

Table 11: Model architecture of StarCoder.

FlashAttention (Dao et al., 2022) to speed up the attention computation and reduce its memory
footprint, allowing us to scale to context length 8K. To make FlashAttention work with MQA during
training, we simply expand the key and value before calling the attention kernel. The architecture
hyper-parameters are given in Table 11.

5 . 5 T R A I N I N G D E TA I L S
StarCoder_paper (1)