Replit's new AI Model now available on Hugging Face
I’ve been using mistral and code llama to generate large volumes of code recently, and I have to say…<p>These small models just suck compared to the larger ones.<p>It get it, they’re quick and cheap (ha! Relatively) to make and good for research and fine tuning…<p>…but can anyone here speak authoritatively on fine tuning and getting good results out?<p>I’ve been super disappointed by how bad even the q6 13B code llama model is at generating consistent code (Ie. It even compiles, forget doing what you asked) > about 30 lines in length.<p>These smaller model seem good for a line or two, maybe, but gosh… it’s an effort to do anything useful with them out of the box.<p>Carefully crafted prompt.<p>Tests, hand written.<p>Iterate: prompt, compile, run tests, generate code metrics. Accept code that passes the tests and beats the target threshold.<p>You’re looking at like 7-10 iterations per prompt to get anything for simple (< 30 lines) functions, and maybe no candidates after 30 iterations for longer complex requests.<p>Are people just using this for 5 line code snippets and autocomplete?<p>Or is there a way to get better results by fine tuning?