Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week
Some links:<p>- Repo: <a href="https:&#x2F;&#x2F;github.com&#x2F;replit&#x2F;ReplitLM&#x2F;tree&#x2F;main&#x2F;replit-code-v1-3b">https:&#x2F;&#x2F;github.com&#x2F;replit&#x2F;ReplitLM&#x2F;tree&#x2F;main&#x2F;replit-code-v1-...</a><p>- HuggingFace: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;replit&#x2F;replit-code-v1-3b" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;replit&#x2F;replit-code-v1-3b</a><p>- Demo: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;replit&#x2F;replit-code-v1-3b-demo" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;replit&#x2F;replit-code-v1-3b-demo</a><p>- Early benchmark results: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;amasad&#x2F;status&#x2F;1651019556423598081" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;amasad&#x2F;status&#x2F;1651019556423598081</a><p>A lot about this project was surprising. We knew it was going to be good, but didn&#x27;t expect to be this good -- especially surprising was the finetuned performance boost, and the fact that the model is decent at language tasks and reasoning (in some cases much better than much larger general-purpose models).<p>It feels like there is a lot more to do with this model, and I have a suspicion you can even make a half-decent chatbot (at least one focused on code) by finetuning it on conversation (and&#x2F;or instruction) datasets.<p>Will follow up with a more comprehensive technical report and the UL2R version (fill-in-the-middle support).