title: Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution
    Autonomous Driving VQA from Peru
summary: As multimodal foundational models start being deployed experimentally in
  Self-Driving cars, a reasonable question we ask ourselves is how similar to
  humans do these systems respond in certain driving situations -- especially
  those that are out-of-distribution? To study this, we create the Robusto-1
  dataset that uses dashcam video data from Peru, a country with one of the worst
  (aggressive) drivers in the world, a high traffic index, and a high ratio of
  bizarre to non-bizarre street objects likely never seen in training. In
  particular, to preliminarly test at a cognitive level how well Foundational
  Visual Language Models (VLMs) compare to Humans in Driving, we move away from
  bounding boxes, segmentation maps, occupancy maps or trajectory estimation to
  multi-modal Visual Question Answering (VQA) comparing both humans and machines
  through a popular method in systems neuroscience known as Representational
  Similarity Analysis (RSA). Depending on the type of questions we ask and the
  answers these systems give, we will show in what cases do VLMs and Humans
  converge or diverge allowing us to probe on their cognitive alignment. We find
  that the degree of alignment varies significantly depending on the type of
  questions asked to each type of system (Humans vs VLMs), highlighting a gap in
  their alignment.
paper_authors._id: 67cff2aa7f2291321730ebfe
paper_authors.name: Dunant Cusipuma
paper_authors._id: 67cff2aa7f2291321730ebff
paper_authors.name: David Ortega
paper_authors._id: 67cff2aa7f2291321730ec00
paper_authors.name: Victor Flores-Benites
paper_authors._id: 67cff2aa7f2291321730ec01
paper_authors.name: Arturo Deza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:42:25.026Z
paper_authors._id: 61fc521187117e8015dd1173
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643926004312-noauth.png
paper_authors.fullname: Arturo Deza
paper_authors.type: user
paper_authors.user: ArturoDeza
paper_title: Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution
    Autonomous Driving VQA from Peru
paper_summary: As multimodal foundational models start being deployed experimentally in
  Self-Driving cars, a reasonable question we ask ourselves is how similar to
  humans do these systems respond in certain driving situations -- especially
  those that are out-of-distribution? To study this, we create the Robusto-1
  dataset that uses dashcam video data from Peru, a country with one of the worst
  (aggressive) drivers in the world, a high traffic index, and a high ratio of
  bizarre to non-bizarre street objects likely never seen in training. In
  particular, to preliminarly test at a cognitive level how well Foundational
  Visual Language Models (VLMs) compare to Humans in Driving, we move away from
  bounding boxes, segmentation maps, occupancy maps or trajectory estimation to
  multi-modal Visual Question Answering (VQA) comparing both humans and machines
  through a popular method in systems neuroscience known as Representational
  Similarity Analysis (RSA). Depending on the type of questions we ask and the
  answers these systems give, we will show in what cases do VLMs and Humans
  converge or diverge allowing us to probe on their cognitive alignment. We find
  that the degree of alignment varies significantly depending on the type of
  questions asked to each type of system (Humans vs VLMs), highlighting a gap in
  their alignment.
paper_ai_summary: Robusto-1 dataset evaluates cognitive alignment between humans and foundational visual language models using multi-modal visual question answering and representational similarity analysis in diverse driving scenarios.
paper_ai_keywords: Foundational Visual Language Models VLMs multi-modal Visual Question Answering VQA Representational Similarity Analysis RSA