C.6. APPS dataset settings
The CodeContests training set has a non-empty intersection with the APPS test set, and therefore
CodeContests cannot be used during training when evaluating on the APPS benchmark.

46

Competition-Level Code Generation with AlphaCode

Copyright 2022 DeepMind Technologies Limited .
SPDX - License - Identifier : Apache -2.0
def _extract_num_public_testcases_from_description ( desc : str) -> int:

""" Parses the description to extract the number of public testcases ."""
if ’--- Sample Input ’ in desc :

return desc . count (’--- Sample Input ’)

elif ’--- Example Input ’ in desc :

return desc . count (’--- Example Input ’)

else :

for section in desc . split (’\n-----’):

if section . startswith (’Example ’):

return n_tests

return 0

n_tests = max( section . count (’\ nInput ’), section . count (’\ nSample Input ’))
if n_tests :
alphacode