SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery

SQuID is a benchmark of 2,000 quantitative questions on satellite images. Numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers and SQuID-Train, 75,687 training question-answer pairs on separate images.

The best zero-shot end-to-end VLM reaches 28.1%. As a baseline, QASR has an LLM write a program that segments the image and measures the masks. It reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks raises accuracy from 42.2% to 82.6%. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%.

@misc{massih2026squid,
  title         = {{SQuID}: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery},
  author        = {Massih, Peter A. and Cosatto, Eric and Min, Martin Renqiang},
  year          = {2026},
  eprint        = {2601.13401},
  archivePrefix = {arXiv}
}