nativeport/web-access-api-benchmarks

NativePort Web-Access API Benchmarks

Measured quality, latency, cost and error-rate figures for 22 commercial web-access APIs — search, SERP, scraping, crawling, extraction, sourced answers, screenshots, document parsing, browser actions and change watching — scored per capability on a fixed task corpus. This is the 2026-08-05 run: 67 provider × capability scorecards across 13 capabilities, flattened into 297 metric rows.

It exists for one practical decision: when an AI agent needs to reach the live web, which API should the call go to, and what will that cost in latency and dollars? Composite scores alone rarely answer that. The raw per-metric values do, and they are all here.

Disclosure

This dataset is produced by NativePort from its own first-party benchmark runs. NativePort operates a commercial gateway that routes to many of the providers scored here. It is not an independent third-party evaluation, and it should not be cited as one. The measurement protocol, including what NativePort explicitly declines to claim, is published in full (linked under Methodology) so readers can weigh the numbers accordingly. Weak and last-place results are published unchanged — three scorecards in this snapshot sit below 2.0 out of 10, and two record a 100% error rate.

Dataset structure

Two configs, two views of the same 67 scorecards. Nothing is aggregated or re-scored between them.

Both configs are JSONL deliberately. The Hub resolves a single packaged loader for the whole repository from the declared config data files and then applies it to every config, so a repository that mixes formats across its configs ends up parsing one of them with the other's reader.

Two more files ship alongside them and back no config:

data/benchmarks.csv is a download rather than a config. load_dataset never reads it, but pd.read_csv("hf://datasets/nativeport/web-access-api-benchmarks/data/benchmarks.csv"), huggingface_hub.hf_hub_download and a plain browser download all do. Both tidy files are written in one pass from one row builder, so they cannot drift; the test suite compares them field by field across all 297 rows, including the exact text of every number. Every figure in data/summary.json is derived by the generator, never typed.

data/metric_rows.jsonl and data/benchmarks.csv — tidy metric rows

Both carry these 23 fields, in this order:

In data/metric_rows.jsonl the types above are the JSON types: metric_raw and composite_score are JSON numbers, metric_index / rank / rank_of / composite_scale_max are integers, and is_capability_top is a boolean. CSV has no types, so data/benchmarks.csv carries the same values as text — numbers written as the exact source token, booleans as true / false.

Provenance fields repeat on every row so that a filtered slice stays interpretable and verifiable on its own.

{
  "evaluation_id": "serper:serp",
  "provider_id": "serper",
  "provider_name": "Serper",
  "provider_group": "Search",
  "provider_category": "Google SERP scrape",
  "capability_id": "serp",
  "capability_label": "SERP verticals",
  "metric_key": "latency_p50_ms",
  "metric_label": "Latency p50",
  "metric_raw": 886.0,
  "metric_display": "886 ms",
  "metric_index": 1,
  "composite_score": 9.13,
  "composite_scale_max": 10,
  "rank": 1,
  "rank_of": 4,
  "is_capability_top": true,
  "measured_date": "2026-08-05",
  "note": "Lowest cost and latency; Google only.",
  "provider_page_url": "https://nativeport.ai/providers/serper/",
  "run_id": "2026-08-05",
  "source_url": "https://nativeport.ai/evals.json",
  "snapshot_sha256": "f46bf416803d5adf696c66ac14a6cbf06f4dfa39727cca164e8f3872abd6ed9b"
}

data/benchmarks.jsonl — one record per evaluation

{
  "evaluation_id": "serper:serp",
  "provider_id": "serper",
  "provider_name": "Serper",
  "provider_group": "Search",
  "provider_category": "Google SERP scrape",
  "capability_id": "serp",
  "capability_label": "SERP verticals",
  "capability_description": "Google SERP verticals (web, news, images, places, scholar) as structured JSON",
  "composite_score": 9.13,
  "composite_scale_max": 10,
  "rank": 1,
  "rank_of": 4,
  "is_capability_top": true,
  "measured_date": "2026-08-05",
  "note": "Lowest cost and latency; Google only.",
  "metric_count": 4,
  "metrics": [
    {"metric_index": 0, "metric_key": "quality", "metric_label": "Quality", "raw_value": 0.938, "display_value": "0.94"},
    {"metric_index": 1, "metric_key": "latency_p50_ms", "metric_label": "Latency p50", "raw_value": 886.0, "display_value": "886 ms"},
    {"metric_index": 2, "metric_key": "cost_per_call_usd", "metric_label": "Cost", "raw_value": 0.0003, "display_value": "$0.0003 / call"},
    {"metric_index": 3, "metric_key": "error_rate_pct", "metric_label": "Errors", "raw_value": 0.0, "display_value": "0%"}
  ],
  "provider_page_url": "https://nativeport.ai/providers/serper/",
  "run_id": "2026-08-05",
  "source_url": "https://nativeport.ai/evals.json",
  "source_schema_version": 1,
  "snapshot_sha256": "f46bf416803d5adf696c66ac14a6cbf06f4dfa39727cca164e8f3872abd6ed9b"
}

metrics is a list of structs, not a key-value map, because metric sets differ by capability — a map would force a sparse union column across the whole dataset.

Capabilities in this snapshot

Each capability is a separate leaderboard with its own task corpus, its own quality metric and its own cost denominator.

Metric sets are stable within a capability but not guaranteed uniform: one screenshot scorecard carries three metrics rather than five — its source note records that every capture errored, and no cost-per-call or full-page value is published for it. Consumers should key on metric_key rather than on metric position or count, and treat a missing metric as absent rather than as zero.

Metric keys

Units and direction are read off the source's own metric_display strings (816 ms, $0.0003 / call, 0%, 3.4 / 10); the machine-readable value always lives in metric_raw / raw_value.

Usage

from datasets import load_dataset

rows = load_dataset("nativeport/web-access-api-benchmarks", "metric_rows", split="train")
evals = load_dataset("nativeport/web-access-api-benchmarks", "evaluations", split="train")

Both configs load as JSON Lines. rows has the 23 flat fields listed above, evals the nested metrics list.

Cheapest search provider that clears a recall floor — the typical routing question, answered from the tidy view with pandas:

import pandas as pd

df = pd.read_json(
    "hf://datasets/nativeport/web-access-api-benchmarks/data/metric_rows.jsonl", lines=True
)
# The CSV mirror gives the same frame, for tools that prefer a spreadsheet:
# df = pd.read_csv("hf://datasets/nativeport/web-access-api-benchmarks/data/benchmarks.csv")

search = df[df.capability_id == "search"]

wide = search.pivot_table(
    index=["provider_id", "composite_score", "rank"],
    columns="metric_key",
    values="metric_raw",
).reset_index()

eligible = wide[(wide.recall_at_10 >= 0.55) & (wide.error_rate_pct == 0)]
print(eligible.sort_values("cost_per_useful_usd")[
    ["provider_id", "recall_at_10", "latency_p50_ms", "cost_per_useful_usd", "rank"]
])

The same query with no third-party dependencies, from the nested view:

import json

with open("data/benchmarks.jsonl", encoding="utf-8") as handle:
    evaluations = [json.loads(line) for line in handle]

def metric(evaluation, key):
    for entry in evaluation["metrics"]:
        if entry["metric_key"] == key:
            return entry["raw_value"]
    return None

search = [e for e in evaluations if e["capability_id"] == "search"]
eligible = [e for e in search if metric(e, "recall_at_10") >= 0.55]
for evaluation in sorted(eligible, key=lambda e: metric(e, "cost_per_useful_usd")):
    print(
        evaluation["provider_id"],
        metric(evaluation, "recall_at_10"),
        f'{metric(evaluation, "latency_p50_ms"):.0f} ms',
        f'${metric(evaluation, "cost_per_useful_usd"):.5f}/useful',
    )

Two shapes worth knowing before you write a query:

• —Latency and cost are not comparable across capabilities. A serp call and an act_agent call differ by three orders of magnitude in wall time by nature.
• —Cost denominators differ by capability. cost_per_call_usd charges every attempt; cost_per_useful_usd and cost_per_successful_page_usd divide by usable output, so failures inflate them. Do not mix the three in one ordering.

Methodology

Each capability has a versioned task corpus held fixed across every provider — the same URLs, queries, target schemas and pass criteria — and four dimensions are recorded per provider × capability pair: a capability-specific quality metric, median wall-clock latency measured from the runner, track-specific cost computed from the provider's real metered price, and error rate across the run. Where quality needs judgment rather than string comparison, grading is done by an LLM panel working from written rubrics. These fold into a composite out of 10, and providers are ranked within each capability.

The full protocol — corpus construction, the four measured dimensions, how grading works, and what NativePort explicitly does not claim (no uptime, SLA or throughput figures) — is documented in How we measure. Human-readable ranked tables per capability are at the leaderboards.

The source snapshot summarises its own protocol as:

One fixed task corpus per capability, identical for every provider; scorecards carry their run dates. Weak scores stay published, and the gateway's flat top-up fee means the ranking earns nothing from steering you toward pricier providers.

Provenance and reproducibility

Rebuild the data files from that snapshot:

curl -sSfL https://nativeport.ai/evals.json -o evals.json
sha256sum evals.json          # must match the SHA-256 above
python3 scripts/build_dataset.py --input evals.json --output-dir data

scripts/build_dataset.py is standard-library-only and deterministic: identical input bytes produce byte-identical outputs. No wall-clock timestamp is written anywhere, so a rebuild can be diffed directly against the published files. Numeric values are carried across as exact source tokens — the generator verifies that each emitted number serialises character-for-character back to the token it read, and aborts rather than emit a rounded stand-in. That same serialiser writes the CSV cell and the JSON number, which is why the two tidy files agree token for token. data/summary.json records the SHA-256 of all three data files, together with the config-to-file mapping the front matter declares.

Fields present in the source but excluded by design: gateway routing and authentication strings, list prices, latency prose, marketing summaries, choose_if / avoid_if guidance, and catalog tier labels. Only benchmark measurements and the provenance needed to interpret them are published here.

Limitations and scope

• —Coverage is partial. 22 of the 32 providers in the source catalog carry scorecards in this run; the other 10 have no eval entries and therefore no rows here. The source additionally flags 5 provider × capability pairs as unscored in this run. Absence from this dataset means not measured in the `2026-08-05` run — not "failed", and not "unavailable".
• —Composites are capability-local. A composite is only meaningful against other providers in the same capability. A 9.13 on serp and a 9.13 on scrape are not the same achievement, and averaging a provider's composites across capabilities produces a number with no defined meaning.
• —Thin capabilities. act_agent and watch contain a single scored provider each; extract_rules and parse contain three. A rank of 1 out of 1 is not evidence of superiority. Always read rank_of alongside rank.
• —Single point in time. Every row in this snapshot was measured on 2026-08-05. Provider behaviour, pricing and anti-bot posture change; these figures age.
• —First-party measurement. Runs are operated by NativePort, which has a commercial relationship with providers in the catalog. See Disclosure.
• —Judged metrics carry model bias. Quality metrics that require judgment are graded by an LLM panel against rubrics, not by human annotators.
• —Not measured at all: uptime, SLA conformance, throughput ceilings, regional performance, concurrency behaviour, and long-run stability. No row in this dataset speaks to any of them.
• —Cost is a measurement, not a quote. Figures are computed from metered prices at run time for the calls in the corpus. They are not an offer, a rate card, or a prediction of any particular workload's bill.

Update policy

• —The dataset tracks NativePort benchmark runs. A new run publishes as a new revision of this repository, with run_id, measured_date and snapshot_sha256 changing together.
• —Prior revisions stay reachable through the repository's commit history; superseded numbers are not silently rewritten in place.
• —Schema changes that are not backward compatible will be described in the commit that makes them and reflected in the tables above.
• —File layout, for anyone who loaded an earlier revision: the metric_rows config is backed by data/metric_rows.jsonl. It previously pointed at data/benchmarks.csv, which left the two configs in different formats and made the Hub read one of them with the wrong parser. No row, field or measured value changed — only the file the config resolves to — and data/benchmarks.csv still ships, unchanged, as a download.
• —No update cadence is promised here. measured_date and run_id are on every row precisely so a consumer can decide for itself whether the snapshot is still fresh enough to act on.

Licensing

This dataset is licensed by NativePort under the Creative Commons Attribution 4.0 International licence (CC BY 4.0).

What the licence covers. NativePort licenses what it is in a position to license: this dataset as a compilation — its selection, arrangement, schema, documentation and this card — together with the benchmark measurements NativePort itself produced and any database rights NativePort holds in them. The grant extends only to those rights and only to the extent NativePort holds them. Where a jurisdiction treats an individual measured figure as an unprotectable fact, the licence simply does not reach it: CC BY 4.0 places no conditions on a use that is lawful without permission (legal code § 2(a)(2) and § 8(a)). Where sui generis database rights do apply, § 4 of the legal code grants extraction and reuse of all or a substantial part of the contents, subject to the same attribution condition.

What the licence does not cover. CC BY 4.0 does not license patent or trademark rights (legal code § 2(b)(2)). The provider, product and company names and marks that appear in this dataset — including every mark listed under Trademark notice — remain the property of their respective owners. They are not licensed, sublicensed or otherwise granted to you here, by NativePort or by this licence; NativePort has no authority to grant rights in another party's marks and does not purport to. Reusing this dataset under CC BY 4.0 therefore gives you no right to use those marks beyond whatever nominative, descriptive or fair use your own jurisdiction independently allows. Nor does the licence permit you to assert or imply a connection with, sponsorship by, or endorsement from NativePort as licensor (legal code § 2(a)(6)); no trademark owner named here has endorsed, reviewed or sponsored these results.

Attribution. Credit NativePort, name the dataset and the run (2026-08-05), link to this repository or to https://nativeport.ai/evals.json, state that the material is under CC BY 4.0 with a link to the licence, and indicate whether you modified it. The Citation block below carries everything needed.

No warranty. The material is offered as-is and as-available, without warranties or conditions of any kind, and NativePort's liability is limited, as set out in § 5 of the legal code. Read it alongside Limitations and scope: these are measurements from one run on one date, not a guarantee of any provider's future behaviour.

Trademark notice

Provider names, product names and logos referenced here — including Serper, SerpApi, SearchAPI.io, Brave Search, You.com, DataForSEO, Tavily, Exa, Linkup, Parallel, Jina, ScraperAPI, Firecrawl, ScrapingBee, Scrapfly, ZenRows, Crawlbase, Oxylabs, Bright Data, Spider, Zyte and Apify — are trademarks of their respective owners. They are used here for identification and factual comparison only. Their appearance does not imply any affiliation with, sponsorship by, endorsement by, or review of these results by the trademark owners. NativePort is a trademark of its owner. Hugging Face is a trademark of Hugging Face, Inc.

The CC BY 4.0 licence described under Licensing grants no rights in any of these marks — trademark rights are outside what that licence conveys (legal code § 2(b)(2)) and outside what NativePort could convey in the first place.

Citation

@misc{nativeport_web_access_api_benchmarks_2026_08_05,
  title        = {NativePort Web-Access API Benchmarks},
  author       = {{NativePort}},
  year         = {2026},
  note         = {Benchmark run 2026-08-05; 67 provider-capability scorecards across 13 capabilities.
                  Source snapshot SHA-256 f46bf416803d5adf696c66ac14a6cbf06f4dfa39727cca164e8f3872abd6ed9b},
  license      = {CC BY 4.0, \url{https://creativecommons.org/licenses/by/4.0/}},
  howpublished = {\url{https://nativeport.ai/evals.json}}
}

Plain text: NativePort. NativePort Web-Access API Benchmarks, run 2026-08-05. Retrieved from https://nativeport.ai/evals.json. Licensed under CC BY 4.0 (<https://creativecommons.org/licenses/by/4.0/>); indicate if you modified it.