Omg I am sooooo happy to see so much excitement for this project . We are gonna kill this y’all .
I agree with Julien as well, tokenized will be important. Character or even byte level may be the way to go, but I worry we will run into memory issues if we have the model predicting large amount of code similar to copilot. My research group tried regular old BPE but then added in the special keywords that exist to try and make it so that the BPE model didn’t have too many superfluous tokens, but it’s hard to say if that is the optimal.
I love the idea of fine-tuning the model and using the stack exchange, especially since the big part of copilot is how you can prompt it with comments to generate your code. So, having all sorts of data that has some mix of natural language and code would be the best. We will need to define some cleaning criteria as well, maybe we could run some static analyzer to check for certain known vulnerabilities or insecurities. GitHub has their codescanning tool that does this and i know a few research tools as well that we could look at
There are also a few people who were interested in twitter that haven’t commented here. I’ll msg them to also post here.