gui-wm/transition_data_v1
GUI World Model — Branch Transitions (s, a, s') transitions taken beside a walked trajectory rather than along it: the same state, a different action, and the screen that actually followed. Every state is captured from a running Ubuntu desktop — a screenshot, the accessibility tree as XML, and the rendered element table the model reads. A spine records what an agent did. This set records what it did not do and what would have happened, which is the question an agent asks a world… See the full description on the dataset page: https://huggingface.co/datasets/gui-wm/transition_data_v1.
GUI World Model — Branch Transitions
(s, a, s') transitions taken beside a walked trajectory rather than along it: the same state, a different action, and the screen that actually followed. Every state is captured from a running Ubuntu desktop — a screenshot, the accessibility tree as XML, and the rendered element table the model reads.
A spine records what an agent did. This set records what it did not do and what would have happened, which is the question an agent asks a world model when it plans.
What is here
All 81 trajectories carry AgentNet instructions, nine from each of nine applications:
chrome gimp libreofficecalc libreofficeimpress libreofficewriter os thunderbird vlc vscode
Nine each is deliberate — the collector takes one task from each application in turn, so a run that stops early still covers every application rather than the beginning of the alphabet.
Several tasks appear more than once under different start states (__cfg0 … __cfg3), which is how the same instruction is walked on more than one arrangement of the desktop.
Spine and branch are both present, and are not the same thing
Sampled over 250 rows spread through the set:
The spine rows are the walked path the branches hang off. Do not add the two counts into a single figure — they answer different questions, and a total that mixes them cannot be attributed to either.
Every branch in this set is candidate_origin: policy-sampled at branch_depth: 1: one step away from a spine state, with the action drawn from the policy rather than uniformly from the clickable elements. The uniform arm lives elsewhere.
Columns that say what a row is
A known gap
instruction_source is filled on spine rows and empty on branch rows. The task's origin is still recoverable from the task_id prefix, which is agentnet_ for every row here, but a reader filtering on the column alone will find the branches unlabelled.
What this set does not tell you
Whether the branched trajectories succeeded at their tasks. These were collected before spine was narrowed to mean a trajectory a judge called successful, so a branch here is an alternative to an action that was taken, not necessarily an alternative to a correct one.
That matters for evaluation and not for training. Training on a transition uses only the fact that this screen, under this action, became that screen — which is observed, and true regardless of how the task ended. Evaluation that asks whether a branch was better or worse than the path needs the path to be known good, and here it is not recorded.
How the branch actions were chosen
The policy is asked once for the state with n=32 at temperature 1.0, and the completions are grouped by intention rather than by string: the verb plus the smallest accessibility element containing the point. On one Thunderbird state, 24 distinct strings were three intentions, six of them the same button five pixels apart. The most common intentions are executed, up to a budget of ten, because each execution costs a VM action and a snapshot.
draw_count — how many of the 32 produced each intention — is computed but not stored, so a nine-of-thirty-two action cannot be told from a one-of-thirty-two one after the fact.
Related
gui-wm/spine— 154,220 spine-only transitions, no branches