5551z/VC-Tooler-SFT

VC-Tooler-SFT

Supervised cold-start trajectories for VC-Tooler: Learning Compositional and Adaptive Visual Tool Use.

🔗 Links

• —📄 Paper: arXiv <!-- TODO: add paper link -->
• —🌐 Project Page: w1zheng.github.io/VC-Tooler
• —🤗 Hugging Face: VC-Tooler-SFT (this dataset) · VC-Tooler-RL
• —🧩 ModelScope: VC-Tooler-SFT (this dataset) · VC-Tooler-RL

This dataset is the Stage I (supervised fine-tuning) trajectory bank used to teach a vision–language model to use visual tools as a compositional and adaptive capability, rather than to memorize a fixed set of invocation patterns. Each example is a multi-step, ReAct-style trajectory in which language reasoning is interleaved with tool calls and tool-returned observations while solving a visual query.

<p align="center"> <img src="assets/data_composition.png" width="100%" alt="VC-Tooler SFT trajectory composition across Single-Tool, Multi-Tool, and Diverse-Tool-Context subsets."><br> <em>Composition of the SFT trajectory bank across the Single-Tool, Multi-Tool, and Diverse-Tool-Context capability levels.</em> </p>

What's in this dataset

VC-Tooler frames visual tool use around three core capabilities, and the trajectory bank is synthesized to provide supervision for each of them:

• —Single-Tool (ST) grounding — interpreting a tool schema, grounding a call in the visual input, and predicting valid arguments to execute one tool.
• —Multi-Tool (MT) composition — chaining several tool calls across steps, where each step reasons over the observations returned by earlier calls.
• —Diverse-Tool-Context (DTC) — the same underlying visual operation re-expressed through many different tool interfaces, so the model learns to adapt to novel schemas instead of memorizing tool identities.

A small number of no-tool reasoning samples are also included to preserve the base model's direct-answering ability.

How it was built

<p align="center"> <img src="assets/synthesis_pipeline.png" width="100%" alt="Hierarchical trajectory synthesis pipeline: plan-then-execute, then diverse tool-context reinstantiation, producing Single-Tool, Multi-Tool, and Diverse-Tool-Context supervision."><br> <em>Hierarchical synthesis pipeline: a planner proposes a tool sequence, an executor grounds it into a tool-grounded trajectory (M=1 Single-Tool, M&gt;1 Multi-Tool), and reinstantiation re-expresses steps under diverse tool schemas.</em> </p>

Trajectories are produced by a hierarchical synthesis pipeline with two complementary phases:

1. 1.Plan-then-Execute. A large thinking VLM first proposes an ordered tool-use plan for a query (conditioned on the answer annotation, which is used only during planning to avoid answer leakage into execution). A non-thinking model then follows the plan step by step inside a tool environment, grounding each call and recording the returned observation. Trajectories are labeled ST or MT according to the number of tool calls.
2. 2.Diverse Tool-Context Reinstantiation. For each executed step, a strong VLM inspects the pre-/post-action visual states and either matches the operation to an existing tool or defines a new schema, producing an alternative interface that realizes the same visual operation. This yields the DTC subset and greatly expands interface diversity.

Candidate source instances are selected for diversity, verifiability, and agentic relevance (favoring queries that genuinely benefit from acquiring, transforming, or revisiting visual evidence) via a hierarchical filtering pipeline. Public multimodal sources include, among others, LLaVA-OneVision, DeepVision, and VisualProbe for plan-then-execute synthesis; Zebra-CoT and Monet for the diverse-context trajectories; and ChartVerse for the no-tool reasoning samples. After synthesis, trajectories are post-filtered to remove answer-inconsistent, format-invalid, or inefficient rollouts, and distractor tools are introduced into the candidate pool for robustness.

The executable tools underlying the ST/MT subsets cover common visual operations (e.g., zoom, rotate, enhance, code execution, mark, and edit), while the DTC subset augments these with a large pool of mock-but-valid tool interfaces to drive adaptive tool-use learning.

<p align="center"> <img src="assets/tool_wordcloud.png" width="30%" alt="Word cloud of tool names appearing across the trajectory bank."><br> <em>Tool-name diversity across the trajectory bank, spanning perception, editing, computation, and search operations.</em> </p>

Intended use

• —Cold-start supervised fine-tuning of vision–language models for agentic, multi-step visual tool use.
• —Research on compositional and adaptive tool calling, schema generalization, and trajectory synthesis for multimodal agents.

This bank is the supervised precursor to reinforcement learning; the companion RL dataset is released separately as VC-Tooler-RL-Data.

Data format

The released columns are:

messages.role uses a tool-use protocol:

function_call / observation pairs repeat for multi-step trajectories. Images are referenced from the text by an <image> placeholder — in the user turn for query images and in observation turns for tool-produced ones — where the n-th placeholder corresponds to images[n].

License

Released under CC BY-NC 4.0 (non-commercial). Individual source datasets used during synthesis retain their own licenses and terms of use; please review and comply with the terms of each upstream source before use.

Citation

@inproceedings{vctooler2026,
  title     = {VC-Tooler: Learning Compositional and Adaptive Visual Tool Use},
  author    = {Wu, Yizheng and Hua, Jiashen and Deng, Bing and Ye, Jieping},
  booktitle = {arXiv},
  year      = {2026}
}