Lost in Time: Clock and Calendar Understanding Challenges in Multimodal
  LLMs
Understanding time from visual representations is a fundamental cognitive
skill, yet it remains a challenge for multimodal large language models (MLLMs).
In this work, we investigate the capabilities of MLLMs in interpreting time and
date through analogue clocks and yearly calendars. To facilitate this, we
curated a structured dataset comprising two subsets: 1) ClockQA,
which comprises various types of clock styles-standard, black-dial,
no-second-hand, Roman numeral, and arrow-hand clocks-paired with time related
questions; and 2) CalendarQA, which consists of yearly calendar
images with questions ranging from commonly known dates (e.g., Christmas, New
Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of
the year). We aim to analyse how MLLMs can perform visual recognition,
numerical reasoning, and temporal inference when presented with time-related
visual data. Our evaluations show that despite recent advancements, reliably
understanding time remains a significant challenge for MLLMs.
67aa05c5ffb9f6b5b2f658b2
Rohit Saxena
claimed_verified
2025-02-27T09:18:07.805Z
657ccbf2869d5bb0e53b482f
/avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
Rohit Saxena
user
rohitsaxena
67aa05c5ffb9f6b5b2f658b3
Aryo Pradipta Gema
claimed_verified
2025-02-13T08:26:03.971Z
644f895e23d7eb05ca695054
/avatars/3fb04dd8544b403262bf98507de05453.svg
Aryo Pradipta Gema
user
aryopg
67aa05c5ffb9f6b5b2f658b4
Pasquale Minervini
claimed_verified
2025-02-10T15:51:20.711Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
Lost in Time: Clock and Calendar Understanding Challenges in Multimodal
  LLMs
Understanding time from visual representations is a fundamental cognitive
skill, yet it remains a challenge for multimodal large language models (MLLMs).
In this work, we investigate the capabilities of MLLMs in interpreting time and
date through analogue clocks and yearly calendars. To facilitate this, we
curated a structured dataset comprising two subsets: 1) ClockQA,
which comprises various types of clock styles-standard, black-dial,
no-second-hand, Roman numeral, and arrow-hand clocks-paired with time related
questions; and 2) CalendarQA, which consists of yearly calendar
images with questions ranging from commonly known dates (e.g., Christmas, New
Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of
the year). We aim to analyse how MLLMs can perform visual recognition,
numerical reasoning, and temporal inference when presented with time-related
visual data. Our evaluations show that despite recent advancements, reliably
understanding time remains a significant challenge for MLLMs.
MLLMs face challenges in interpreting time and date from visual data, as demonstrated through evaluations using ClockQA and CalendarQA datasets focusing on clock styles and calendar images.
multimodal large language models MLLMs visual recognition numerical reasoning temporal inference