Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week
Some links:<p>- Repo: <a href="https://github.com/replit/ReplitLM/tree/main/replit-code-v1-3b">https://github.com/replit/ReplitLM/tree/main/replit-code-v1-...</a><p>- HuggingFace: <a href="https://huggingface.co/replit/replit-code-v1-3b" rel="nofollow">https://huggingface.co/replit/replit-code-v1-3b</a><p>- Demo: <a href="https://huggingface.co/spaces/replit/replit-code-v1-3b-demo" rel="nofollow">https://huggingface.co/spaces/replit/replit-code-v1-3b-demo</a><p>- Early benchmark results: <a href="https://twitter.com/amasad/status/1651019556423598081" rel="nofollow">https://twitter.com/amasad/status/1651019556423598081</a><p>A lot about this project was surprising. We knew it was going to be good, but didn't expect to be this good -- especially surprising was the finetuned performance boost, and the fact that the model is decent at language tasks and reasoning (in some cases much better than much larger general-purpose models).<p>It feels like there is a lot more to do with this model, and I have a suspicion you can even make a half-decent chatbot (at least one focused on code) by finetuning it on conversation (and/or instruction) datasets.<p>Will follow up with a more comprehensive technical report and the UL2R version (fill-in-the-middle support).