Team Ai
Apppublic

build-small-hackathon/BudgetBuddy

sourceHugging Faceupdated 4mo agoView on Hugging Face
3likes
App README

🧾 BudgetBuddy

A spend tracker for real people — built entirely on small, open models. Snap a messy receipt or a UPI/card screenshot (or just type it), and BudgetBuddy reads it, fixes the totals, categorises it, saves it privately, and lets you chat with a tool-using agent about where your money went — all in a custom dashboard UI. No third-party AI APIs. Two MiniCPM models do everything.

Build Small hackathon · Backyard AI track. Built for the people around me (homemakers, parents, small-shop owners) who want to understand their spending without a spreadsheet — and without handing their receipts to a cloud AI API.

▶️ Demo & links

  • —Live app: https://huggingface.co/spaces/build-small-hackathon/BudgetBuddy
  • —Demo video: https://youtu.be/QbgY6HDbrxE
  • —Social post: https://x.com/KrishnaIsCoding/status/2066565121464541191
  • —Build write-up (Field Notes): https://huggingface.co/blog/KrishnaGarg/budget-buddy-field-notes (repo copy)
  • —Open agent traces (Sharing is Caring): AGENT_TRACES.md

Why it fits "Build Small"

  • —Small, open models — no third-party AI API. Vision/OCR is MiniCPM-V-4.6 (1.3B), running on the Space's ZeroGPU. The reasoning + agent brain is MiniCPM4.1-8B (8B), running on our own [Modal](https://modal.com) GPU. ~9.3B total — well under the 32B cap. We never call a hosted AI inference API (no OpenAI/Anthropic/Gemini) — only open weights we run ourselves.
  • —Real problem, honest fit. Real bills are messy: missing totals, taxes, service charges, round-offs, mixed items, weird date formats. BudgetBuddy reasons about them and reconciles the math, so editing is the exception.

What it does

  1. 1.Capture, three ways — a photographed receipt, a payment screenshot (UPI / GPay / PhonePe / card), or a quick manual entry.
  2. 2.Read & reconcile — the vision model extracts vendor, date (normalised to YYYY-MM-DD), line items, taxes/service/tip/discount/round-off, and total; computes a missing total; flags anything that doesn't add up.
  3. 3.Reason & categorise — the 8B reviews the extraction, fixes obvious errors, and assigns an overall + per-item category (fixed 23-category list).
  4. 4.Dashboard — monthly spend, vs-last-month, top category, spend-by-category donut, spend-over-time chart, a budget ring, a calendar heatmap, and a filterable transaction list that shows every line item and every tax/charge.
  5. 5.Agent chat — ask "how much did I spend on Groceries last month?" or "what's my biggest expense?" and the agent answers with your real numbers, showing which tools it used.

🤖 The agent (Best Agent)

The assistant is a real tool-using agent over core/analytics, not a chatbot that guesses. It exposes 11 tools — total_spend, category_spend, item_spend, vendor_spend, top_categories, biggest_expense, average_spend, count_transactions, budget_status, monthly_trend, recent — each scoped by a flexible period (this_month, last_month, this_year, a specific month like 2026-07, a year, or all).

The 8B plans every question: it reads the question, decides which tool to call (and with what period), reads the result, optionally chains another tool, then answers — a real ReAct loop. The tools are deterministic Python, so the numbers can never be hallucinated; an answer is only ever returned once it is grounded by an actual tool call (ungrounded model output is rejected). A deterministic router over the same tools acts as a reliability fallback if the model can't produce a valid plan. Every reply shows the trace of tools used, so the reasoning is auditable.

Custom UI (Off-Brand)

The frontend is a hand-built dark single-page app (frontend/) served by `gradio.Server` (Gradio 6): Python API endpoints on the Gradio backend (queue + ZeroGPU), our own HTML/CSS/JS + Chart.js on top. The default Gradio shell is gone entirely.

⚡ Modal (Best Use of Modal)

MiniCPM4.1-8B's trust_remote_code targets transformers ~4.56 and breaks on the 5.7 that MiniCPM-V-4.6 needs. Modal resolves the conflict cleanly: the 8B runs in its own container/env on an A10G, loaded once into a memory snapshot for fast cold starts and kept warm (scaledown_window), and the Space calls it through the Modal SDK (core/modal_backend.py, modal_app.py). That's what makes the agent quick.

Privacy

Sign in with a username + PIN (PIN stored salted+hashed, never in plaintext). Each user's transactions live in their own file in a private HF Dataset; the frontend holds a signed session token, so no one can read another user's data.

Badges / prizes targeted

  • —Backyard AI track — a practical, everyday-life spending app.
  • —Best MiniCPM Build — the whole app is two MiniCPM models (vision + 8B).
  • —Best Use of Modal — the 8B reasoning/agent model runs on Modal.
  • —Off-Brand (achievement) — a fully custom gradio.Server frontend.
  • —Sharing is Caring (achievement) — open agent traces on the Hub (AGENT_TRACES.md).
  • —Field Notes (achievement) — a build write-up / report (FIELD_NOTES.md).
  • —Best Agent · Best Demo · Bonus Quest Champion — judged across entries (multi-step agent, full demo package, most bonus criteria met).

Structure

core/extract.py       # MiniCPM-V-4.6: receipt + payment extraction, reconcile, date-normalise
core/categorize.py    # 8B: refine/repair + categorise (overall + per-item)
core/agent.py         # tool-using spending agent (router + ReAct loop, 11 tools)
core/chat.py          # grounded one-shot answer (agent fallback)
core/analytics.py     # pure-Python aggregations (summary, by-category, over-time, calendar)
core/inference.py     # one place that owns the models / routes vision + text generation
core/modal_backend.py # client for the Modal 8B service
core/storage.py       # per-user transactions + budget in a HF Dataset
core/auth.py          # username + PIN accounts, signed session tokens
core/hubio.py         # low-level dataset JSONL IO
app.py                # gradio.Server: API endpoints + serves the custom frontend
modal_app.py          # Modal service hosting MiniCPM4.1-8B (deploy: modal deploy modal_app.py)
frontend/             # custom dark SPA (index.html + assets/app.js, Chart.js)

Run locally

bash
python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python app.py        # open http://127.0.0.1:7860

A GPU isn't required to try the UI (first run downloads the vision weights). On the Space, set an HF_TOKEN secret (dataset persistence) and MODAL_TOKEN_ID / MODAL_TOKEN_SECRET (the 8B backend), and BB_INFERENCE=modal. Tip: log in to Hugging Face in your browser to use your own ZeroGPU quota for the vision model.