Scientific LLM Benchmarks
GitHub
← All benchmarks
Math· multimodal

MathVista

UCLA / University of Washington / Microsoft · 2023

Mathematical reasoning benchmark combining visual contexts (figures, charts, diagrams) with problems.

Mathematics
GitHub stars
Task type
VQA
Modality
multimodal
Access
open
Size
6,141 items
License
CC-BY-SA-4.0
Metrics
accuracy
pid
1
question
When a spring does work on an object, we cannot find the work by simply multiplying the spring force by the object's displacement. The reason is that there is no one value for the force-it changes. However, we can split the displacement up into an infinite number of tiny parts and then approximate the force in each as being constant. Integration sums the work done in all those parts. Here we use the generic result of the integration. In Figure, a cumin canister of mass $m=0.40 \mathrm{~kg}$ slides across a horizontal frictionless counter with speed $v=0.50 \mathrm{~m} / \mathrm{s}$. It then …
image
images/1.jpg
decoded_image
<image/binary>
precision
1
answer
1.2
question_type
free_form
answer_type
float
metadata
{"category":"math-targeted-vqa","context":"scientific figure","grade":"college","img_height":720,"img_width":1514,"language":"english","skills":["scientific reasoning"],"source":"SciBench","split":"testmini","task":"textbook question answering"}
query
Hint: Please answer the question requiring a floating-point number with one decimal place and provide the final value, e.g., 1.2, 1.3, 1.4, at the end. Question: When a spring does work on an object, we cannot find the work by simply multiplying the spring force by the object's displacement. The reason is that there is no one value for the force-it changes. However, we can split the displacement up into an infinite number of tiny parts and then approximate the force in each as being constant. Integration sums the work done in all those parts. Here we use the generic result of the integration. …
pid
2
question
what is the total volume of the measuring cup?
image
images/2.jpg
decoded_image
<image/binary>
unit
g
answer
1000
question_type
free_form
answer_type
integer
metadata
{"category":"general-vqa","context":"natural image","grade":"daily life","img_height":768,"img_width":1024,"language":"english","skills":["numeric commonsense","arithmetic reasoning"],"source":"TextVQA","split":"testmini","task":"visual question answering"}
query
Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: what is the total volume of the measuring cup? (Unit: g)

Real rows from the Hugging Face datasets server · long values truncated