AIIRLab/MathDisAM

MathDisAm: Towards Math Query Disambiguation

Welcome to the official repository for the paper MathDisAm: Towards Math Query Disambiguation. This repository contains the dataset, query generation pipelines, and baseline classifier models designed to distinguish between ambiguous and unambiguous mathematical queries.

Dataset

The data directory contains the core datasets used for training and evaluating the disambiguation models:

• —`MathDisAM_train.tsv`: The training file containing MathSE IDs, mathematical questions, and their corresponding tags ("Ambiguous" or "Unambiguous").
• —`MathDisAM_test.tsv`: The untouched, held-out test file used for final model evaluation.

Query Generation

The query generation process leverages a two-agent Large Language Model pipeline (using Gemma) to process natural mathematical questions:

• —Ambiguous Query Generation:
• —Uses an "Injector" agent to rewrite a math question to introduce deliberate ambiguity.
• —Uses a "Verifier" agent to independently judge if the rewrite is genuinely ambiguous.
• —The script processes questions until a target of 1,000 verified ambiguous questions is produced.
• —Unambiguous Query Generation:
• —Uses a "Clarifier" agent to rewrite queries to remove ambiguity.
• —Uses a "Verifier" agent to evaluate if the query is fully well-posed and unambiguous.
• —This pipeline also targets 1,000 verified unambiguous questions.