ShaunSpark/laya-mind2web-browser-agent

Laya Fine-Tuned for Browser Automation (Mind2Web)

This is a fine-tuned version of the Laya decision model, specialized for browser automation tasks. It predicts the next browser action (operation and target element) from a web page's DOM structure and a user goal.

Model Details

• —Developed by: Shaun Andrade (@ShaunSpark)
• —Base model: convaiinnovations/laya (421M parameters)
• —Training data: Mind2Web (671 usable training examples)
• —Validation accuracy: 74.3% on a held-out set of 68 examples

Training Procedure

• —Hardware: Google Colab, single NVIDIA T4 GPU (16 GB VRAM)
• —Optimizer: 8-bit AdamW (bitsandbytes), learning rate 1e-5
• —Effective batch size: 8 (micro-batch 1, gradient accumulation 8)
• —Precision: FP16 mixed precision (torch.cuda.amp)
• —Memory optimizations: Gradient checkpointing, expandable_segments
• —Epochs: 2 (best checkpoint; further training overfits)

Intended Use

Decision engine for a browser automation agent. Given a goal and a list of interactive DOM elements, the model answers two questions in one forward pass:

• —Operation: CLICK, TYPE, or SELECT
• —Target: the correct element index from the candidate list

Limitations

• —Trained on a single dataset (Mind2Web); may not generalize to all websites.
• —Overfits beyond 2 epochs on 671 examples.
• —Requires the base Laya tokenizer and architecture.

How to Use

import torch
import laya

agent = laya.load("convaiinnovations/laya")
model = agent.model
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

Citation

• —Laya: https://huggingface.co/convaiinnovations/laya
• —Mind2Web: https://arxiv.org/abs/2306.06070