Super cool idea !
As Julien said picking the right tokenizer would be important.
And one could also consider fine-tuning GPTNeo on code data instead of pre-training from scratch, as GPTNeo is trained on Pile dataset 4 which already contains the github  (almost 95GB as stated in the paper) and Stack Exchange data.