Try to guess if code is real or GPT2-generated
Hi, author here! Some details on the model:<p>* Trained 17GB of code from the top 10,000 most popular Debian packages. The source files were deduplicated using a process similar to the OpenWebText preprocessing (basically a locality-sensitive hash to detect near-duplicates).<p>* I used the [Megatron-LM](<a href="https://github.com/NVIDIA/Megatron-LM" rel="nofollow">https://github.com/NVIDIA/Megatron-LM</a>) code for training. Training took about 1 month on 4x RTX8000 GPUs.<p>* You can download the trained model here: <a href="https://moyix.net/~moyix/csrc_final.zip" rel="nofollow">https://moyix.net/~moyix/csrc_final.zip</a> and the dataset/BPE vocab here: <a href="https://moyix.net/~moyix/csrc_dataset_large.json.gz" rel="nofollow">https://moyix.net/~moyix/csrc_dataset_large.json.gz</a> <a href="https://moyix.net/~moyix/csrc_vocab_large.zip" rel="nofollow">https://moyix.net/~moyix/csrc_vocab_large.zip</a><p>Happy to answer any questions!