<p>I have gathered a <a href="https://github.com/neoneye/arc-notes/tree/main/awesome" rel="nofollow noreferrer">few notes</a> about ARC, with links to projects that use LLMs. Too many links to post them all here.</p>
<p>As of 7-may-2024, <a href="https://lab42.global/arc-interview/" rel="nofollow noreferrer">Jack Cole+Mohamed Osman</a> solves 34 of the 100 hidden tasks. This is SOTA with LLMs and the hidden dataset.</p>

CC BY-SA 4.0
2023-10-16T10:18:18.730Z
2024-05-07T18:26:07.040Z
2024-05-07T18:26:07.040Z
neoneye
<p>Q: <strong>I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs?</strong><br />
<strong>Can LLMs that can process graphics input do the following?</strong></p>
<blockquote>
<p>(This question is <strong>not</strong> about the The AI2’s Reasoning Challenge (ARC) dataset, a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9.)</p>
</blockquote>
<p>Background:<br />
I came across a paper from 2019 <a href="https://arxiv.org/abs/1911.01547" rel="nofollow noreferrer">&quot;On the Measure of Intelligence&quot;</a> by Francois Chollet who is a Researcher at Google, Creator of the Keras Deep Learning Library.</p>
<p>In that paper he proposed the Abstraction and Reasoning Corpus, ARC. Actually you can view this as  a Benchmark that consists of solving 400 graphical reasoning puzzles,  to test AI systems' abstraction and analogy abilities.</p>
<p>To illustrate, here is an example from the paper:</p>
<p><a href="https://i.sstatic.net/NX9DP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NX9DP.png" alt="enter image description here" /></a></p>
<p>(A machine must infer what to do (transforming a test picture without seeing a &quot;result&quot; picture on the right) from ~3 similar training imagepairs per task)-</p>
<p>2020 state of the art:
In 2020 there was a Kaggle Competition and the <a href="https://www.kaggle.com/code/icecuber/arc-1st-place-solution/notebook" rel="nofollow noreferrer">1st place solution</a> got 20% of the 400 task right. (<a href="https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/leaderboard" rel="nofollow noreferrer">Leaderboard</a> Score in competition was &quot;Percentage still wrong&quot; or something - so lowest score of 79.4% won)
The  <a href="https://github.com/top-quarks/ARC-solution" rel="nofollow noreferrer">C++ code</a> contains special-purpose image manipulation and inference routines, carefully crafted and combined as described in a <a href="https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/154597" rel="nofollow noreferrer">write-up</a> by the author &quot;icecuber&quot;.</p>
<p>Now it's 2023 and OpenAI is making GPT-4V multimodal generally available.</p>
<p>In April 2023 someone asked a similar question on the <a href="https://community.openai.com/t/gpt-4-and-the-arc-challenge/168955" rel="nofollow noreferrer">OpenAI discussion forum</a>; no answers. That post was written independently from this one,  and I just discovered that post right before submitting this post.</p>
<p><strong>Yesterday I have  given ChatGPT-4 Advanced Data Analysis a single example, and the LLM quickly <a href="https://chat.openai.com/share/02cb360e-8f42-4932-b764-8b57c36167b2" rel="nofollow noreferrer">solved</a> the puzzle shown below;</strong> sort of. Admittedly, my prompt was  in sloppy natural language, and I uploaded 3 badly cropped training images as input.</p>
<p>Task f8c80d96:</p>
<p><a href="https://i.sstatic.net/qYe7Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qYe7Y.png" alt="f8c80d96-demo-3" /></a></p>
<p>With human intelligence I was not able to infer the solution to task f8c80d96 easily. Made 3 wrong guesses.<br />
(Solution: Add 1 vertical yellow line but also replace black background with grey background)</p>
<p>Clearly my attempt is not a systematic study, just practice.<br />
To conclude, I suspect if GPT-4 can solve 1 task, it can solve many more than 20% of those 400 examples, e.g. when using the API with carefully designed prompts and instructions.</p>

<papers><agi><benchmarks>
How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)
knb
# How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)

(Asked by: username_0 on 2023-10-08)

Q: **I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs?**  
**Can LLMs that can process graphics input do the following?**

> (This question is **not** about the The AI2’s Reasoning Challenge (ARC) dataset, a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9.)

Background:  
I came across a paper from 2019 ["On the Measure of Intelligence"](https://arxiv.org/abs/1911.01547) by Francois Chollet who is a Researcher at Google, Creator of the Keras Deep Learning Library.

In that paper he proposed the Abstraction and Reasoning Corpus, ARC. Actually you can view this as a Benchmark that consists of solving 400 graphical reasoning puzzles, to test AI systems' abstraction and analogy abilities.

To illustrate, here is an example from the paper:

[![enter image description here](https://i.sstatic.net/NX9DP.png)](https://i.sstatic.net/NX9DP.png)

(A machine must infer what to do (transforming a test picture without seeing a "result" picture on the right) from ~3 similar training imagepairs per task)-

2020 state of the art:
In 2020 there was a Kaggle Competition and the [1st place solution](https://www.kaggle.com/code/icecuber/arc-1st-place-solution/notebook) got 20% of the 400 task right. ([Leaderboard](https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/leaderboard) Score in competition was "Percentage still wrong" or something - so lowest score of 79.4% won)
The [C++ code](https://github.com/top-quarks/ARC-solution) contains special-purpose image manipulation and inference routines, carefully crafted and combined as described in a [write-up](https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/154597) by the author "icecuber".

Now it's 2023 and OpenAI is making GPT-4V multimodal generally available.

In April 2023 someone asked a similar question on the [OpenAI discussion forum](https://community.openai.com/t/gpt-4-and-the-arc-challenge/168955); no answers. That post was written independently from this one, and I just discovered that post right before submitting this post.

**Yesterday I have given ChatGPT-4 Advanced Data Analysis a single example, and the LLM quickly [solved](https://chat.openai.com/share/02cb360e-8f42-4932-b764-8b57c36167b2) the puzzle shown below;** sort of. Admittedly, my prompt was in sloppy natural language, and I uploaded 3 badly cropped training images as input.

Task f8c80d96:

[![f8c80d96-demo-3](https://i.sstatic.net/qYe7Y.png)](https://i.sstatic.net/qYe7Y.png)

With human intelligence I was not able to infer the solution to task f8c80d96 easily. Made 3 wrong guesses.  
(Solution: Add 1 vertical yellow line but also replace black background with grey background)

Clearly my attempt is not a systematic study, just practice.  
To conclude, I suspect if GPT-4 can solve 1 task, it can solve many more than 20% of those 400 examples, e.g. when using the API with carefully designed prompts and instructions.

**Tags:** papers, agi, benchmarks


---

## 1 answers

### Answer by username_1 on 2023-10-16. Score: 1 | Accepted answer

I have gathered a [few notes](https://github.com/username_1/arc-notes/tree/main/awesome) about ARC, with links to projects that use LLMs. Too many links to post them all here.

As of 7-may-2024, [Jack Cole+Mohamed Osman](https://lab42.global/arc-interview/) solves 34 of the 100 hidden tasks. This is SOTA with LLMs and the hidden dataset.


---

