Team Ai
Modelpublic

torontodeveloper/mind2web-candidate-ranker

sourceHugging Facemitupdated 4mo agoView on Hugging Face
0likes30downloads
Model Card

mind2web-candidate-ranker

A DeBERTa-v3-base cross-encoder fine-tuned for candidate generation in web agents: given a natural-language task (plus action history) and a DOM element, it scores how likely that element is the next action target. It is stage 1 of a two-stage computer-use agent (rank DOM elements → LLM picks the action → Playwright executes).

Trained on Mind2Web (2,000+ tasks across 137 real websites).

Results

Evaluated on the three official Mind2Web generalization splits:

splitacc@1recall@3recall@5recall@10recall@20MRR
test_task (unseen tasks)60.2%83.5%89.4%93.0%93.8%0.725
test_website (unseen websites)50.9%76.1%85.3%92.6%95.3%0.656
test_domain (unseen domains)55.4%79.8%87.1%92.8%94.5%0.689

~93% recall@10 on entirely unseen domains. The recall curve plateaus at k≈20, making top-20 a good cost/recall operating point for the downstream LLM.

Usage

python
from sentence_transformers import CrossEncoder

model = CrossEncoder("torontodeveloper/mind2web-candidate-ranker", num_labels=1, max_length=512)

candidate = "tag: input | aria_label: Destination | placeholder: To?"
query = "task is: Search for a flight from Toronto to New York. Type NYC into the destination field.\nPrevious actions: "

score = model.predict([(candidate, query)])

Candidates are serialized DOM elements (tag, aria_label, placeholder, name, is_clickable, bounding box, etc.); the query is the task description plus the last few actions. Score all interactable elements on a page and keep the top-k.

Training

  • —Base model: microsoft/deberta-v3-base (86M params)
  • —Data: Mind2Web train split — 7,775 actions, 1 positive + 3 sampled negatives each (31,100 pairs)
  • —3 epochs, batch size 8, lr 3e-5, BCEWithLogitsLoss, linear warmup
  • —Single T4 GPU (~85 minutes), fp32

⚠️ Precision note

DeBERTa-v3's disentangled attention overflows fp16 (NaN logits). Load and run this model in fp32 (or bf16 on Ampere+). Newer transformers versions may default to fp16 on GPU — pass torch_dtype=torch.float32 explicitly if needed.

Pipeline context

This model is the candidate generator from mind2web-computer-use-agent: stage 2 (action prediction) uses FLAN-T5 / GPT-class models over the top-k candidates, and stage 3 executes via Playwright. Architecture follows MindAct (Deng et al., 2023).