Answers.Body: <p>I have gathered a <a href="https://github.com/neoneye/arc-notes/tree/main/awesome" rel="nofollow noreferrer">few notes</a> about ARC, with links to projects that use LLMs. Too many links to post them all here.</p>
  <p>As of 7-may-2024, <a href="https://lab42.global/arc-interview/" rel="nofollow noreferrer">Jack Cole+Mohamed Osman</a> solves 34 of the 100 hidden tasks. This is SOTA with LLMs and the hidden dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T10:18:18.730Z
Answers.LastActivityDate: 2024-05-07T18:26:07.040Z
Answers.LastEditDate: 2024-05-07T18:26:07.040Z
Answers.UserDisplayName: neoneye
Body: <p>Q: <strong>I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs?</strong><br />
  <strong>Can LLMs that can process graphics input do the following?</strong></p>
  <blockquote>
  <p>(This question is <strong>not</strong> about the The AI2’s Reasoning Challenge (ARC) dataset, a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9.)</p>
  </blockquote>
  <p>Background:<br />
  I came across a paper from 2019 <a href="https://arxiv.org/abs/1911.01547" rel="nofollow noreferrer">&quot;On the Measure of Intelligence&quot;</a> by Francois Chollet who is a Researcher at Google, Creator of the Keras Deep Learning Library.</p>
  <p>In that paper he proposed the Abstraction and Reasoning Corpus, ARC. Actually you can view this as  a Benchmark that consists of solving 400 graphical reasoning puzzles,  to test AI systems' abstraction and analogy abilities.</p>
  <p>To illustrate, here is an example from the paper:</p>
  <p><a href="https://i.sstatic.net/NX9DP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NX9DP.png" alt="enter image description here" /></a></p>
  <p>(A machine must infer what to do (transforming a test picture without seeing a &quot;result&quot; picture on the right) from ~3 similar training imagepairs per task)-</p>
  <p>2020 state of the art:
  In 2020 there was a Kaggle Competition and the <a href="https://www.kaggle.com/code/icecuber/arc-1st-place-solution/notebook" rel="nofollow noreferrer">1st place solution</a> got 20% of the 400 task right. (<a href="https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/leaderboard" rel="nofollow noreferrer">Leaderboard</a> Score in competition was &quot;Percentage still wrong&quot; or something - so lowest score of 79.4% won)
  The  <a href="https://github.com/top-quarks/ARC-solution" rel="nofollow noreferrer">C++ code</a> contains special-purpose image manipulation and inference routines, carefully crafted and combined as described in a <a href="https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/154597" rel="nofollow noreferrer">write-up</a> by the author &quot;icecuber&quot;.</p>
  <p>Now it's 2023 and OpenAI is making GPT-4V multimodal generally available.</p>
  <p>In April 2023 someone asked a similar question on the <a href="https://community.openai.com/t/gpt-4-and-the-arc-challenge/168955" rel="nofollow noreferrer">OpenAI discussion forum</a>; no answers. That post was written independently from this one,  and I just discovered that post right before submitting this post.</p>
  <p><strong>Yesterday I have  given ChatGPT-4 Advanced Data Analysis a single example, and the LLM quickly <a href="https://chat.openai.com/share/02cb360e-8f42-4932-b764-8b57c36167b2" rel="nofollow noreferrer">solved</a> the puzzle shown below;</strong> sort of. Admittedly, my prompt was  in sloppy natural language, and I uploaded 3 badly cropped training images as input.</p>
  <p>Task f8c80d96:</p>
  <p><a href="https://i.sstatic.net/qYe7Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qYe7Y.png" alt="f8c80d96-demo-3" /></a></p>
  <p>With human intelligence I was not able to infer the solution to task f8c80d96 easily. Made 3 wrong guesses.<br />
  (Solution: Add 1 vertical yellow line but also replace black background with grey background)</p>
  <p>Clearly my attempt is not a systematic study, just practice.<br />
  To conclude, I suspect if GPT-4 can solve 1 task, it can solve many more than 20% of those 400 examples, e.g. when using the API with carefully designed prompts and instructions.</p>
  
Tags: <papers><agi><benchmarks>
Title: How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)
UserDisplayName: knb
ThreadText: # How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)
  
  (Asked by: username_0 on 2023-10-08)
  
  Q: **I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs?**  
  **Can LLMs that can process graphics input do the following?**
  
  > (This question is **not** about the The AI2’s Reasoning Challenge (ARC) dataset, a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9.)
  
  Background:  
  I came across a paper from 2019 ["On the Measure of Intelligence"](https://arxiv.org/abs/1911.01547) by Francois Chollet who is a Researcher at Google, Creator of the Keras Deep Learning Library.
  
  In that paper he proposed the Abstraction and Reasoning Corpus, ARC. Actually you can view this as a Benchmark that consists of solving 400 graphical reasoning puzzles, to test AI systems' abstraction and analogy abilities.
  
  To illustrate, here is an example from the paper:
  
  [![enter image description here](https://i.sstatic.net/NX9DP.png)](https://i.sstatic.net/NX9DP.png)
  
  (A machine must infer what to do (transforming a test picture without seeing a "result" picture on the right) from ~3 similar training imagepairs per task)-
  
  2020 state of the art:
  In 2020 there was a Kaggle Competition and the [1st place solution](https://www.kaggle.com/code/icecuber/arc-1st-place-solution/notebook) got 20% of the 400 task right. ([Leaderboard](https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/leaderboard) Score in competition was "Percentage still wrong" or something - so lowest score of 79.4% won)
  The [C++ code](https://github.com/top-quarks/ARC-solution) contains special-purpose image manipulation and inference routines, carefully crafted and combined as described in a [write-up](https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/154597) by the author "icecuber".
  
  Now it's 2023 and OpenAI is making GPT-4V multimodal generally available.
  
  In April 2023 someone asked a similar question on the [OpenAI discussion forum](https://community.openai.com/t/gpt-4-and-the-arc-challenge/168955); no answers. That post was written independently from this one, and I just discovered that post right before submitting this post.
  
  **Yesterday I have given ChatGPT-4 Advanced Data Analysis a single example, and the LLM quickly [solved](https://chat.openai.com/share/02cb360e-8f42-4932-b764-8b57c36167b2) the puzzle shown below;** sort of. Admittedly, my prompt was in sloppy natural language, and I uploaded 3 badly cropped training images as input.
  
  Task f8c80d96:
  
  [![f8c80d96-demo-3](https://i.sstatic.net/qYe7Y.png)](https://i.sstatic.net/qYe7Y.png)
  
  With human intelligence I was not able to infer the solution to task f8c80d96 easily. Made 3 wrong guesses.  
  (Solution: Add 1 vertical yellow line but also replace black background with grey background)
  
  Clearly my attempt is not a systematic study, just practice.  
  To conclude, I suspect if GPT-4 can solve 1 task, it can solve many more than 20% of those 400 examples, e.g. when using the API with carefully designed prompts and instructions.
  
  **Tags:** papers, agi, benchmarks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-16. Score: 1 | Accepted answer
  
  I have gathered a [few notes](https://github.com/username_1/arc-notes/tree/main/awesome) about ARC, with links to projects that use LLMs. Too many links to post them all here.
  
  As of 7-may-2024, [Jack Cole+Mohamed Osman](https://lab42.global/arc-interview/) solves 34 of the 100 hidden tasks. This is SOTA with LLMs and the hidden dataset.
  
  
  ---
  
  