Zoe/GS-QA2

GS-QA2: A Benchmark for Question Answering over Raster–Vector Data

Release scope: The original canonical release contains 3,300 public questions, and its four existing configurations and public test splits remain unchanged. The new_v1 extension publishes 300 development questions: 100 each in new_vector, new_raster_only, and new_raster_vector. A separate 150-question challenge (50 per track) is retained privately on the maintainers' lab cluster. Its questions, SQL, answers, and geographic anchors are not part of this public release. The original description below refers to the canonical 3,300-question release. See the new extension and private evaluation workflow and development template examples. There is no automated hosted scoring service for the hidden challenge at present.

GS-QA2 is a benchmark for geospatial question answering over both vector and raster data. It extends GS-QA — a vector-only benchmark built on OpenStreetMap features in a PostGIS database — with a U.S. Digital Elevation Model (DEM) raster layer and 25 new question templates that require terrain reasoning: elevation lookups, slope, aspect, ruggedness, and questions that combine vector filtering with raster analysis.

Each record pairs a natural-language question with an executable PostGIS SQL query and its ground-truth answer, so every question is verifiable without manual annotation. This release contains the complete benchmark: the 2,800 vector-only pairs (V1–V28, 100 per template) inherited from GS-QA by Saeedan, Rashid, Eldawy, and Hristidis, plus the 500 raster-related pairs (25 templates × 20 questions) that GS-QA2 adds — 3,300 questions in total.

Code, ingestion scripts, generation pipeline, and baseline implementations live in the companion repository: https://github.com/ZhuochengShang/QARV

GS-QA2 resources: all resources collection · raw OSM + DEM source data · public leaderboard · code

Why this benchmark

Existing geospatial QA benchmarks evaluate vector data only (points, lines, polygons), while vision-oriented raster benchmarks target perception over imagery rather than structured reasoning over raster values. GS-QA2 fills the gap: it measures whether LLM-based systems (Text2SQL, RAG, multi-stage SQL agents, code-writing GIS agents) can reason over continuous terrain surfaces and their interaction with discrete vector features. In the paper's evaluation, systems answered simple point-elevation queries reasonably well but accuracy dropped sharply on terrain derivatives (slope, aspect) and combined raster–vector operations — zonal and global raster operations collapsed to near zero for every baseline.

Dataset structure

Four configurations, mirroring the paper's template groups of increasing raster-reasoning complexity:

All 3,300 questions form a single evaluation set, published as the test split of each config. All configs are stored as zstd-compressed Parquet. Questions were instantiated from templates by sampling real geographic entities (POIs, roads, parks, regions across the contiguous United States), executing the paired SQL against the reference PostGIS database (OSM + 30 m DEM, 265,950 raster tiles), and retaining only questions with valid, non-empty ground truth.

Data fields

Answer schema in the `vector` config (from the original GS-QA documentation): for entity-name templates (V1–V11 except V7) the answer attribute ends with the suffix _name (e.g. poi_name); for the multi-hop template V7 it is multihop_answer (with its type in multihop_attribute); for location templates (V12–V20) it is geometry (WKT — geocode it if evaluating against addresses rather than coordinates); for direction templates (V21–V22) it is angle; and for the remaining numeric templates it is count, distance, length, or area. For non-aggregate answers the full database record is stored alongside the answer attribute.

The untouched original template files (one JSONL per template, including 9 exploratory templates outside the canonical 500-question evaluation set) are available in the QARV repository under GS-QA/benchmark/qa2/.

Example record (raster_only, template elevation+poi)

{
  "question": "What is the elevation at Cousins Subs, Chicago, IL?",
  "sql": "SELECT ST_Value(rast, 1, ST_GeomFromText('POINT (-87.631702 41.88228)',4326)) AS elevation FROM public.dem_us WHERE ST_Intersects(rast, ST_GeomFromText('POINT (-87.631702 41.88228)',4326)) LIMIT 1;",
  "answers": [{"elevation": 286.0}],
  "answer_type": "elevation - point based"
}

Usage

import json
from datasets import load_dataset

ds = load_dataset("Zoe/GS-QA2", "raster_only", split="test")

example = ds[0]
print(example["question"])
print(example["sql"])
answers = json.loads(example["answers"])        # list of ground-truth rows
entities = json.loads(example["question_entities"])

To execute the SQL and reproduce ground truths or run the baselines, you need the reference PostGIS database (OSM vector tables + DEM raster tiles). Raw OSM extracts and DEM files are published separately in Zoe/GS-QA2-source; use them with the ingestion instructions in the QARV repository (GS-QA/ingestion/).

Evaluation

Answers are scored per output type, following the paper (Tables 9–10): token-level F1 ≥ 0.8 for entity names, geodesic distance error ≤ 5 m for locations, circular angular error ≤ 5° for directions and aspect, absolute error ≤ 10 m for elevations, ≤ 5° for slope, and relative error ≤ 0.05 for areas, lengths, distances, and counts. Compound outputs are correct only when every component passes. Evaluation scripts are in GS-QA/baselines/evaluation/ of the companion repository.

Benchmark construction

1. 1.Template selection — 25 raster-related templates organized by Tomlin's map-algebra operation types (local, focal, zonal, global) plus spatial predicates (range, nearest neighbor, intersects, …).
2. 2.Parameter sampling — placeholders ([ANCH_POI], [DISTANCE], [ELEV_COND], …) filled with real entities and values sampled from the database.
3. 3.NL generation — natural-language phrasing with grammar correction; duplicates removed via canonicalized SQL comparison.
4. 4.Ground-truth execution — each paired SQL runs against the PostGIS reference database (120 s timeout); empty/invalid results are discarded and regenerated. 20 questions are kept per template.

Source data and licensing

• —Vector features: © OpenStreetMap contributors, available under the Open Database License (ODbL). Question text, entity metadata, and ground-truth answers are derived from OSM data, so this dataset is shared under ODbL.
• —Terrain: elevation values derive from the ASTER Global Digital Elevation Model V3 (ASTGTM v003), 1 arc-second (≈ 30 m) resolution over the contiguous United States. ASTER GDEM is a product of METI and NASA.

Citation

If you use the vector config (V1–V28), please also cite the original GS-QA benchmark:

@article{saeedan2026gsqa,
  title   = {GS-QA: A Benchmark for Geospatial Question Answering},
  author  = {Saeedan, Majid and Shihab Rashid, Muhammad and Eldawy, Ahmed and Hristidis, Vagelis},
  journal = {arXiv preprint arXiv:2605.22811},
  year    = {2026}
}

@inproceedings{shang2026gsqa2,
  title     = {GS-QA2: A Benchmark for Question Answering over Raster--Vector Data},
  author    = {Shang, Zhuocheng and Elmahallawy, Shahd and Al Nazi, Zabir and Hristidis, Vagelis and Eldawy, Ahmed},
  year      = {2026},
  note      = {Benchmark and code: https://github.com/ZhuochengShang/QARV}
}

Contact

Zhuocheng Shang — zshan011@ucr.edu — University of California, Riverside