This is unbelievable! Bravo team 👏
That's all I have to say. I really appreciate your team's efforts. Thank you!
Happy to hear this!
Some updates.. our LCB scores has just been verified by a 3rd party, it will be officially on the leaderboard early this week ;)
Good job!
@michaelzhiluo what about merge your model ? have you looked ? (https://huggingface.co/spacematt/Qwen2.5-Recursive-Coder-14B-Instruct via transitive Qwen2.5-Channel-Coder-14B-Instruct)
so far this has been good. I'm surprised how well and fast it works and how useful it is (on a 3060) when used for coding. Thank you for the release.
@michaelzhiluo Your comment about the LCB scores being on the leaderboard soon were made 9 days ago. Maybe I am missing something, but I am unable to find them. Could you point me in the right direction, please?
Recently, the LCB leaderboard has been updated to v6. They should be there if the marker is set <=2/1/2025. We will contact the author leads to update the leaderboard with the new DeepCoder results!
I am still unable to find the model. What am I doing wrong? These are the steps I am taking:
- Go to https://livecodebench.github.io/leaderboard.html
- Drag the right slider to 2/1/2025 (I also tried 1/1/2025 with the same result)
- Control + F and search for DeepCoder -> No results found
I get 16 results (only 15 visible in the screenshot, the 16th is Claude-3-Haiku)
There were formerly 40+ results on the leaderboard before it got updated to v6. I do happen to have a screenshot here for v5 a week ago.
Also a Tweet from one of our collaborators: https://x.com/AlpayAriyak/status/1912171348409061468
Thanks for that! Unfortunate that they removed so many models all of a sudden, really weird. Is there any chance your model could also be evaluated on Aider's benchmark? It's quite a popular benchmark for testing the coding capabilities of LLMs. The leaderboard can be found here: https://aider.chat/docs/leaderboards/
How to run the benchmark can be found here: https://github.com/Aider-AI/aider/blob/main/benchmark/README.md
That's all I have to say. I really appreciate your team's efforts. Thank you!
still doing my own testing, but i tend to agree.
@Mushoz Can try later... we do note that it is trained on Competitive Coding problems, and we're currently working on training it for more general coding settings ;)
I just tried this model out and it had an existential crisis. "Please write me the shortest C# hello world program you can.". It got into an infinite loop and rambled on forever.
Okay, I need to figure out how to write the shortest possible "Hello World" program in C#. The user wants just that, so I should focus on brevity.
First, I know that in C#, a simple console application typically starts with creating a class and a Main method. So something like:
class Program {
static void Main() { }
}
But inside the Main method, I need to output "Hello World". The easiest way is using Console.WriteLine("Hello World");.
Putting it all together, that would be:
using System;
class Program {
static void Main() {
Console.WriteLine("Hello World");
}
}
Wait, but does the 'using System;' directive really be necessary? Because in some cases, if you're using certain namespaces, you might not need it. But for Console.WriteLine, I think you do need to include the System namespace or fully qualify it.
Alternatively, could I avoid the using statement by writing System.Console.WriteLine()? That way, I don't have to include 'using System;' which would save a line. Let me check that.
If I change the code to:
class Program {
static void Main() {
System.Console.WriteLine("Hello World");
}
}
Then I can remove the 'using System;' line. So now the program is shorter because it doesn't have that using statement anymore.
Wait, but in C#, when you're writing a console application, do you need to include any namespaces? If not, then perhaps even the namespace declarations could be omitted if we don't mind having everything in the global namespace. But for such a simple program, maybe it's acceptable.
Wait no, in C#, every code file must have at least one class, and that's all. So the minimal structure is:
class Program {
static void Main() {}
}
But I need to add the write line inside. So focusing on that.
So putting it all together, perhaps the absolute shortest version would be:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
Wait, can I remove some spaces? In C#, after commas and braces, spacing isn't required unless it's for operators or parameters separated by commas. So I think in the class declaration, the curly brace can come right after without space.
So:
class Program{static void Main(){Console.WriteLine("Hello World");}}
And then include the using System; directive. But wait, is that necessary? If I don't have the using statement, does it work?
If I write:
System.Console.WriteLine("Hello World");
without the 'using System;' directive, would that still compile? Let me think: yes, because you're fully qualifying the namespace.
So in that case, I can omit the 'using System;' line. That would make the code even shorter.
Wait no, wait. Wait, if I'm writing System.Console.WriteLine, does that require including the System namespace? Or is it allowed to write the full name?
No, you don't need a using directive for that. You can fully qualify the type. So in the Main method, I can just use System.Console.WriteLine without needing 'using System;'.
So then my code could be:
class Program {
static void Main() {
System.Console.WriteLine("Hello World");
}
}
But wait, can I make it even shorter by removing unnecessary braces or spaces? For example, in the class declaration, I can write it as:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
That's possible. So the entire code would be:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
Wait no, wait: if I don't have 'using System;', then I need to use System.Console.WriteLine.
So without the using statement, it's:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
But that requires including the namespace each time. Alternatively, perhaps adding 'using System;' allows me to write Console.WriteLine, which is shorter.
Wait no, wait: in C#, if I have a using directive for System, then I can use Console.WriteLine without the System prefix. So it's better to include the using statement and save some typing.
But wait, in this case, including 'using System;' adds an extra line, but allows me to write 'Console.WriteLine' instead of 'System.Console.WriteLine', which saves a bit more lines or characters.
Wait no, the code would be:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
So that's two lines: using and class. But can I make it all in one line?
No, because you have to declare the class and the method structure.
Alternatively, perhaps if I write everything on a single line? Wait, no, C# requires certain syntax elements like curly braces and semicolons, but you can't really put multiple statements on one line unless using statement separators, which aren't allowed in that context.
Wait, maybe I can have the entire code as:
using System;class Program{static void Main(){Console.WriteLine("Hello World");}}
But wait, does C# allow putting 'using System;' and then the class declaration on the same line without a space? Or is there a syntax error?
I think in C#, you can't put multiple top-level elements like that. Each using directive must be at the top before any namespace or class declarations, but they have to be separate lines.
So I can't write 'using System;class Program' on the same line. So perhaps I need to keep them as two separate lines.
Alternatively, perhaps I can omit the using statement if I fully qualify Console every time. But that would make each call longer.
Wait no, let me think: if I don't include 'using System;', then in the code, I have to write:
System.Console.WriteLine("Hello World");
So that's a bit more characters, but perhaps it's not worth it because the using statement allows shorter code.
But wait, the minimal code would be either:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
Or without the using:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
Which is shorter? Let's count characters.
Option 1:
Line 1: 'using System;' (12 chars)
Line 2: 'class Program{static void Main(){Console.WriteLine("Hello World");}}' (75 chars)
Total lines: 2, but if I can write the class in one line with minimal spacing.
Wait, perhaps even better:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
But without using System, so that's two lines:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
Alternatively, no using statement and fully qualify:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
So which is shorter? The code with the using statement has 12 + 75 = 87 characters (excluding newlines), while without it's just one line of class declaration with more in the method.
Wait, no: if I don't have the using statement, then the code would be:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
That's all in one line after the class declaration? Wait no, the using statement is a separate line. So without it, you only have the class line.
Wait no: if I don't include 'using System;', then the code would be:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
But wait, that's not correct because in C#, the using directives must come before any namespace or class declarations. So if I omit the using statement, and just have the class, it might not compile without it.
Wait no: in C#, you can write code without including certain namespaces as long as you fully qualify them. So System.Console.WriteLine is acceptable without a using directive for System.
So perhaps that's possible.
But let me think about whether I can get away with writing:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
That's all in one line, but wait no: the class declaration requires { after it. So it should be:
class Program {
static void Main() {
System.Console.WriteLine("Hello World");
}
}
But I can write this more compactly by removing spaces:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
So that's one line for the class and then the method.
Wait no, actually, in C#, you can have the entire class body on the same line if you use braces without space. So:
class Program { static void Main() { System.Console.WriteLine("Hello World"); } }
But wait, does this syntax work? Or do I need semicolons or other elements?
Alternatively, perhaps even shorter by combining lines.
Wait no, in C#, each member must be on its own line unless you're using certain formatting. But for minimal code, perhaps the shortest possible is:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
That's two lines: using and class.
Alternatively, if I don't use 'using System;', then it's just one line of code (the class) with the method inside. Wait no, because the class declaration must be followed by its body.
Wait, perhaps even better: can I write a program without any classes? No, every C# program must have at least one class, which is the entry point.
So the minimal structure is:
class Program {
static void Main() {}
}
But we need to add the WriteLine call inside.
So considering all this, the shortest code would be:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
Alternatively, without using:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
Which is shorter? The version with 'using' has two lines, but the second line is shorter because it uses Console.WriteLine instead of System.Console.WriteLine.
So counting characters:
Option 1:
Line 1: 'using System;' (12 chars)
Line 2: 'class Program{static void Main(){Console.WriteLine("Hello World");}}' (75 chars)
Total lines: 2
Option 2:
No using statement, so code is:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
That's one line for the class declaration and method.
Wait no, that's all in one line? Or does it require multiple lines?
Actually, no. The entire class can be written on a single line if you use the compact syntax:
class Program { static void Main() { System.Console.WriteLine("Hello World"); } }
But I think even more compactly:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
So that's one line for the using (if included) and then another line for the class.
Wait no, if I don't include 'using System;', it's all in two lines: the using statement is not needed, so the code would be:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
But wait, that's just one line of code. Wait no, the class declaration must be followed by a body.
Wait no, I think you can write it as:
class Program{static void Main(){System.Console.WriteLine("Hello World");}}
Yes, because in C#, after the class name comes the curly brace, and then inside, the static method.
So that's one line for the using (if included) or not. Wait no: if I include 'using System;', it's two lines:
Line 1: using System;
Line 2: class Program{static void Main(){Console.WriteLine("Hello World");}}
Alternatively, without using, it's just one line for the class declaration and method.
Wait no, the code must have both the using (if any) and the class. So if I include 'using System;', it's two lines:
using System;
class Program{static void Main(){Console.WriteLine("Hello World");}}
If I don't include the using statement, then it's just one line for the class.
Wait no, no: you can't have multiple elements on a single line like that. Using directives must be separate from class declarations. So in any case, whether you use 'using' or not, you need at least two lines:
If using 'using System;':
Line 1: using System;
Line 2: class Program{static void Main(){Console.WriteLine("Hello World");}}
Without using:
Line 1: class Program{static void Main(){System.Console.WriteLine("Hello World");}}
Wait no, the code can be written as a single line if you don't use 'using'?
No, wait, the entire program must have at least one class. So perhaps you can write it all in one file with just the class.
But regardless of that, I think the minimal code is:
class Program{static void Main(){Console.WriteLine("Hello World");}}
Wait no, because without the 'using System;' directive, Console might not be recognized unless fully qualified.
```...
In general, i have found high temps cause looping and garbage. YMMV of course
In general, i have found high temps cause looping and garbage. YMMV of course
Yeah, not gonna lie -- my AI environment isn't great by any means.
The model tends to overthink (see in the blogpost the average context length). This is a result of not applying a CoT penalty for the reward signal.
Suggest using temperature=0.6 like in the Deepseek models.
The model tends to overthink (see in the blogpost the average context length). This is a result of not applying a CoT penalty for the reward signal.
Suggest using temperature=0.6 like in the Deepseek models.
Thanks. I obviously didn't realize what was meant when he said "temperature".
I obviously didn't realize what was meant when he said "temperature".
lol
Tho, i tend to go lower than .6 myself. all the way down to .3 for anything code related.
There were formerly 40+ results on the leaderboard before it got updated to v6. I do happen to have a screenshot here for v5 a week ago.
Also a Tweet from one of our collaborators: https://x.com/AlpayAriyak/status/1912171348409061468
Did your team try to submit for the latest version Livecodebench?