In preliminary explorations, we discovered that using Anthropic’s HHH prompt (Askell et al., 2021)
turned the model into a somewhat capable yet brittle technical assistant. We were surprised that,
without instruction-tuning, we were able to utilize the 8k context length of StarCoder and let the
model answer questions, follow instructions, and help with solving technical problems. We further
improved the HHH prompt by including more conversational examples related to the programming
domain. We used examples from various sources to create the prompt, including CoT (Wei et al.
2022b; Kojima et al. 2022) and Least-to-most prompting (Zhou et al., 2022) for reasoning questions.
The prompt’s examples originate from StackExchange, PAL (Gao et al., 2022), Anthropic’s HHH
prompt, and BigCode’s effort. We provide example responses in Table 28 and the prompt itself in
Appendix E. Note that the technical assistant has clear limitations: it sometimes proposes wrong
StarCoder_paper (1)