accesslint/2d-webmcp-browser-focus
2D WebMCP Browser Focus (Prerelease) What this is This is an early test of whether agents need useful tool results to complete an accessible browser task. The agent must add a Retry step to a workflow, connect it correctly, and move keyboard focus to that new step. The test checks the real browser, not just the agent's final answer. What happened We ran each version 20 times with gpt-5-mini using low reasoning effort. Tool result Verified… See the full description on the dataset page: https://huggingface.co/datasets/accesslint/2d-webmcp-browser-focus.
2D WebMCP Browser Focus (Prerelease)
What this is
This is an early test of whether agents need useful tool results to complete an accessible browser task.
The agent must add a Retry step to a workflow, connect it correctly, and move keyboard focus to that new step. The test checks the real browser, not just the agent's final answer.
What happened
We ran each version 20 times with gpt-5-mini using low reasoning effort.
The success-or-failure version often gave a confident answer, but never completed and verified the real task. A useful result made a big difference.
For this task, the compact Surface result did not beat the simpler structured result. It also used more input tokens.
Files
data/test.jsonlhas all 60 scored runs. It includes tool calls, final browser state, and scores. Private LangSmith IDs were removed.report.mdhas the full tables and comparisons.
Limits
- This is a prerelease. It covers one app, one task, and one model setting.
- It is an automated browser test, not a study with screen-reader users.
- The code was not committed when this run was made. A later release will point to a committed version.