◆Scientific LLM Benchmarks
GitHub
← All benchmarks
General· robust-multimodal

MMMU-Pro

CMU / University of Waterloo / OSU · 2024

3,460 college-level multimodal questions in standard and vision-only settings, filtered to require visual evidence and expanded to ten answer options.

Paper ↗Dataset ↗Leaderboard ↗Homepage ↗
Task type
VQA
Modality
multimodal
Access
open
Size
3,460 items
License
Apache-2.0
Metrics
accuracy

Examples

Browse full dataset ↗
id
test_History_1
question
Which of the following best explains the overall trend shown in the <image 1>?
options
['Political instability leading to population decline', 'The spread of pathogens across the Silk Road', 'Development of new trade routes', 'Climate change affecting the Silk Road', 'Migrations to areas of Central Asia for resettlement', 'Technological advancements in transportation', 'Invasions by Mongol tribes', 'Large-scale famine due to crop failures', 'Economic prosperity and population growth', 'Rise of religious conflicts along the Silk Road']
explanation
?
image_1
<image/binary>
img_type
['Diagrams']
answer
B
topic_difficulty
Medium
subject
History
id
test_Art_113
question
<image 1> of Louis Black, believed to be a formerly enslaved man, was painted by which artist?
options
['James Irvine', 'John Duncan', 'Gavin Hamilton', 'David Gauld', 'Arthur Melville', 'Francis Cadell', 'Mary Cameron', 'John Lavery', 'Samuel Peploe', 'William York Macgregor']
explanation
?
image_1
<image/binary>
img_type
['Paintings']
answer
A
topic_difficulty
Easy
subject
Art

Real rows from the Hugging Face datasets server · long values truncated