How well can AI measure a parking lot?

Aerial photo of an office parking lot in Justice, Illinois, with its paved area outlined in yellow and 30 landscape islands outlined in white. — Justice, Illinois. Cook County 2025 orthophoto, 15 cm per pixel.

Leaderboard

Score vs. cost

All setups

Method

Lot outline

Trace the paved area around a tapped point, islands cut out. Mean IoU with a hand trace; an invalid polygon scores 0.

Stall count

Count the marked stalls. Mean error against a hand count. Setups that put stalls in the wrong spots are left off the charts.

Striping

Submit a system

Get the cases

git clone https://github.com/DrillerDB/pave-bench
cd pave-bench && pip install -e .

Write predictions

{"caseId": "pb_us_il_justice_testa_001",
 "task": "click_connected_polygon", "track": "vlm_polygon",
 "boundary": [[412, 210], ...], "cutouts": [...],
 "latencyMs": 8200, "costUsd": 0.04,
 "metadata": {"model": "your-model"}}

Score them

python -m pavebench.cli score-manifest \
  --manifest dataset/v0/manifest.jsonl \
  --predictions predictions.jsonl --out score.json

Open a pull request
Add your row to results/runs.jsonl. Maintainers rerun the hidden cases before a result is official.

Changes

13 Oct 2026
Full-takeoff score. 23 setups from OpenAI, Google, Anthropic and Meta. Judged striping.
13 Oct 2026
Imagery importer fix: pixel-aligned Web Mercator. Justice case regenerated.
13 Oct 2026
Invalid polygons now score 0.
9 Jul 2026
First real-imagery case: Justice, Illinois.

Cite

@misc{pavebench2026,
  title  = {PaveBench: Accuracy, Cost and Speed of AI Pavement Takeoff},
  author = {ProPaving},
  year   = {2026},
  url    = {https://pavebench.org}
}