title: A Case Study of Web App Coding with OpenAI Reasoning Models
summary: This paper presents a case study of coding tasks by the latest reasoning
  models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other
  frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task
  benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling
  number of tasks and test cases. The new benchmark causes the o1 model
  performances to decline significantly, falling behind Claude 3.5. Moreover,
  they consistently fail when confronted with atypical yet correct test cases, a
  trap non-reasoning models occasionally avoid. We hypothesize that the
  performance variability is due to instruction comprehension. Specifically, the
  reasoning mechanism boosts performance when all expectations are captured,
  meanwhile exacerbates errors when key expectations are missed, potentially
  impacted by input lengths. As such, we argue that the coding success of
  reasoning models hinges on the top-notch base model and SFT to ensure
  meticulous adherence to instructions.
paper_authors._id: 66f22c0e8ed0ca089b38ba3a
paper_authors.name: Yi Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:45.933Z
paper_authors._id: 669dbd709a4bf63e08f1ddc2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669dbd709a4bf63e08f1ddc2/aV10ZJPPzH5LbnHFZNqc7.png
paper_authors.fullname: Yi Cui
paper_authors.type: user
paper_authors.user: onekq
paper_title: A Case Study of Web App Coding with OpenAI Reasoning Models
paper_summary: This paper presents a case study of coding tasks by the latest reasoning
  models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other
  frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task
  benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling
  number of tasks and test cases. The new benchmark causes the o1 model
  performances to decline significantly, falling behind Claude 3.5. Moreover,
  they consistently fail when confronted with atypical yet correct test cases, a
  trap non-reasoning models occasionally avoid. We hypothesize that the
  performance variability is due to instruction comprehension. Specifically, the
  reasoning mechanism boosts performance when all expectations are captured,
  meanwhile exacerbates errors when key expectations are missed, potentially
  impacted by input lengths. As such, we argue that the coding success of
  reasoning models hinges on the top-notch base model and SFT to ensure
  meticulous adherence to instructions.