A Case Study of Web App Coding with OpenAI Reasoning Models
This paper presents a case study of coding tasks by the latest reasoning
models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other
frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task
benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling
number of tasks and test cases. The new benchmark causes the o1 model
performances to decline significantly, falling behind Claude 3.5. Moreover,
they consistently fail when confronted with atypical yet correct test cases, a
trap non-reasoning models occasionally avoid. We hypothesize that the
performance variability is due to instruction comprehension. Specifically, the
reasoning mechanism boosts performance when all expectations are captured,
meanwhile exacerbates errors when key expectations are missed, potentially
impacted by input lengths. As such, we argue that the coding success of
reasoning models hinges on the top-notch base model and SFT to ensure
meticulous adherence to instructions.
66f22c0e8ed0ca089b38ba3a
Yi Cui
claimed_verified
2024-09-24T07:52:45.933Z
669dbd709a4bf63e08f1ddc2
https://cdn-avatars.huggingface.co/v1/production/uploads/669dbd709a4bf63e08f1ddc2/aV10ZJPPzH5LbnHFZNqc7.png
Yi Cui
user
onekq
A Case Study of Web App Coding with OpenAI Reasoning Models
This paper presents a case study of coding tasks by the latest reasoning
models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other
frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task
benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling
number of tasks and test cases. The new benchmark causes the o1 model
performances to decline significantly, falling behind Claude 3.5. Moreover,
they consistently fail when confronted with atypical yet correct test cases, a
trap non-reasoning models occasionally avoid. We hypothesize that the
performance variability is due to instruction comprehension. Specifically, the
reasoning mechanism boosts performance when all expectations are captured,
meanwhile exacerbates errors when key expectations are missed, potentially
impacted by input lengths. As such, we argue that the coding success of
reasoning models hinges on the top-notch base model and SFT to ensure
meticulous adherence to instructions.
Latest OpenAI reasoning models perform well on a single-task benchmark but decline on a more challenging dual-task benchmark and fail with atypical test cases, suggesting that instruction comprehension and model design are crucial for coding success.
reasoning models SOTA WebApp1K WebApp1K-Duo Claude 3.5 instruction comprehension input lengths SFT coding success