Wrt tokenization of code, it may be useful to refer to section 4.3 of the TransCoder paper 6. This paper is on the unsupervised translation of programming languages. In this work javalang for Java, clang for C++ and the tokenize library for Python were used. The figure below shows how robust tokenize is to 2 versions of the same function:

image2248×1123 597 KB


Then the BPE codes are learnt on the tokenized code files using fastBPE 2

For data that includes both useful comments and code, we could look at code snippets at  GeeksforGeeks 3 and code samples such as those for TF and PyTorch available on the official websites

Just my 2 cents