5.5 Ablation study

In the ablation study, we only compare nAcc@1 (or Rank@1) as the conclusions drawn from these
metrics are consistent with those from nAcc@2 and nAcc@3. The full results can be found in
Appendix C.
Study of retrieval. The ﬁrst question we are trying to answer is whether retrieving experience
and knowledge are necessary – what happens if either of them is missing from the prompt sent to
LLM? The results are shown in Table 5. While the absence of knowledge leads to a reduction in
performance, the absence of demonstrations leads to a complete collapse. Thus, we can conclude that
presenting a set of demonstrations is crucial to make the whole framework work.

3https://github.com/features/copilot
4https://aws.amazon.com/codewhisperer/

9

Discretization

On Best
On All

(cid:55)

HPO-B↑
81.59±0.94
70.82±0.01
74.09±0.44

PD1 ↑
1.48±0.06
1.41±0.02
1.45±0.05

HyperFD↓
59.74±1.89

–

84.96±7.16

Table 7: Ablation if discretization in canonicalization.

Knowledge
MLCopilot- Unleashing the Power of Large Language Models in Solving Machine Learning Tasks