A GPT-4 capability forecasting challenge
I stopped playing after the challenge of creating a line-based canvas drawing of the word "Hello". The site says "Wrong! You guessed that GPT-4 could solve the question, but it can't!", whereas while ChatGPT made a mistake there, it clearly had a good approach.I think that this challenge is entirely unfair, and the LLM's should not be expected to write perfect code on the first try, but rather to have something good enough to then run, test and iterate on. Essentially, it should be compared against the first version of code I would write myself before having had a chance to test it yet. From my experience, with ChatGPT and Copilot, when I approach it in this iterative way, it's a great boon to my productivity, and I don't end up blaming neither it nor myself in the times when it's not quite accurate, just like I wouldn't blame a student for making silly mistakes when writing code on a paper-based exam.