content.raw: Can you beat an AI at Raven puzzles?
content.type: text
content.value: Can you beat an AI at Raven puzzles?
content.type: new_line
content.type: new_line
content.raw: https://huggingface.co/spaces/HuggingFaceM4/ai_raven
content.id: HuggingFaceM4/ai_raven
content.type: space
content.type: resource
content.url: https://huggingface.co/spaces/HuggingFaceM4/ai_raven
content.type: new_line
content.type: new_line
content.raw: The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
content.type: text
content.value: The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
content.label: How Far Are We from Intelligent Visual Deductive Reasoning? (2403.04732)
content.raw: https://huggingface.co/papers/2403.04732
content.id: 2403.04732
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2403.04732
content.raw: ).
content.type: text
content.value: ).
content.type: new_line
content.type: new_line
content.raw: But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
content.type: text
content.value: But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
content.type: new_line
content.type: new_line
content.raw: Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
content.type: text
content.value: Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
content.type: new_line
content.type: new_line
content.raw: Such puzzles can be procedurally generated at scale. 
content.type: text
content.value: Such puzzles can be procedurally generated at scale. 
content.raw: https://huggingface.co/datasets/HuggingFaceM4/RAVEN
content.id: HuggingFaceM4/RAVEN
content.type: dataset
content.type: resource
content.url: https://huggingface.co/datasets/HuggingFaceM4/RAVEN
content.raw:  is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
content.type: text
content.value:  is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
content.type: new_line
content.type: new_line
content.raw: We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
content.type: text
content.value: We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
content.type: new_line
content.type: new_line
content.raw: Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
content.type: text
content.value: Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
rawContent: Can you beat an AI at Raven puzzles?
  
  https://huggingface.co/spaces/HuggingFaceM4/ai_raven
  
  The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (https://huggingface.co/papers/2403.04732).
  
  But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
  
  Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
  
  Such puzzles can be procedurally generated at scale. https://huggingface.co/datasets/HuggingFaceM4/RAVEN is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
  
  We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
  
  Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
reactions.reaction: ❤️
reactions.users: samusenps osanseviero clefourrier
reactions.reaction: 🔥
reactions.users: yizheapple
commentators._id: 65f819cc01523ed703872006
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YurLUFRKxyg3R753mCF_V.jpeg
commentators.fullname: Kelvin Bruce 
commentators.name: Kelvin19
commentators.type: user