ThanhLe0125/e5-math

E5-Math-Vietnamese: MRR-Optimized with Base Model Comparison

Model Overview

Fine-tuned E5-base model optimized with MRR (Mean Reciprocal Rank) for exact chunk retrieval in Vietnamese mathematics. Includes comprehensive comparison with base model.

Performance Comparison

Training vs Test Performance

• —Best Validation MRR: 0.8527777777777777 (avg rank: 1.1726384364820848)
• —Test MRR: 0.8439068100358421 (avg rank: 1.1849649607135275)
• —Training Epochs: 6

Fine-tuned vs Base Model Comparison

Detailed Recall@k Comparison

Key Improvements from Fine-tuning

✅ MRR Boost: +0.0743727598566305 improvement in Mean Reciprocal Rank ✅ Ranking Quality: Correct chunks moved up by avg 0.11452269648255986 positions ✅ Hit Rate: Better success rates across all Recall@k metrics ✅ Vietnamese Math: Specialized for Vietnamese mathematical content ✅ Hierarchy: Maintains Correct > Related > Irrelevant scoring

Why MRR Matters for Exact Retrieval

MRR optimization pushes correct chunks to top positions:

Before (Base Model):
Rank 1: Related chunk    (MRR contribution: 0.0)
Rank 2: Irrelevant      (MRR contribution: 0.0)  
Rank 3: CORRECT chunk   (MRR contribution: 0.33)

After (Fine-tuned):
Rank 1: CORRECT chunk   (MRR contribution: 1.0)  ⭐
Rank 2: Related chunk   (MRR contribution: 0.0)
Rank 3: Irrelevant     (MRR contribution: 0.0)

Result: 3x better MRR, users find answers immediately!

Usage

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Load MRR-optimized model
model = SentenceTransformer('ThanhLe0125/e5-math')

# ⚠️ CRITICAL: Must use E5 prefixes
query = "query: Định nghĩa hàm số đồng biến là gì?"
chunks = [
    "passage: Hàm số đồng biến trên khoảng (a;b) là...",  # CORRECT
    "passage: Ví dụ bài tập về hàm đồng biến...",        # RELATED
    "passage: Phương trình bậc hai có dạng..."           # IRRELEVANT
]

# Get MRR-optimized rankings
query_emb = model.encode([query])
chunk_embs = model.encode(chunks)
similarities = cosine_similarity(query_emb, chunk_embs)[0]

# With fine-tuning, correct chunk should be at rank #1
ranked_indices = similarities.argsort()[::-1]
print(f"Rank 1: {chunks[ranked_indices[0]][:50]}... (Score: {similarities[ranked_indices[0]]:.3f})")

# Expected: Correct chunk at rank #1 with high score

Inference Efficiency

With MRR optimization, you typically only need top 1-2 chunks:

# Efficient inference - high probability correct chunk is #1
top_chunk = chunks[similarities.argmax()]
confidence = similarities.max()

if confidence > 0.7:  # High confidence threshold
    return top_chunk  # Likely the correct answer
else:
    return chunks[similarities.argsort()[::-1][:3]]  # Return top 3 as fallback

Evaluation Methodology

• —Training: trainquestion + valquestion with MRR optimization
• —Validation: MRR for early stopping, Recall@3/5 monitoring
• —Test: test_question used once for final comparison
• —Comparison: Direct evaluation against base E5-multilingual model
• —Metrics: MRR, Recall@1,2,3,4,5, Hierarchy Rate

Perfect For

🎯 Educational Q&A: Exact answers at rank #1 consistently ⚡ Efficient Systems: Fewer chunks needed at inference 🇻🇳 Vietnamese Math: Specialized mathematical terminology 📊 Quality Ranking: Hierarchical relevance scoring 🚀 Production Ready: Proven improvement over base model

Technical Notes

• —Base Model: intfloat/multilingual-e5-base
• —Fine-tuning: Hierarchical contrastive learning with MRR optimization
• —Max Sequence: 256 tokens
• —Training Data: Vietnamese mathematical content with expert annotations
• —Validation: Proper train/validation/test split methodology

Fine-tuned on 25/06/2025 with comprehensive base model comparison.