CodingBad02/chhaya-medgemma-lora-v2

Chhaya-MedGemma (LoRA)

A QLoRA fine-tune of `google/medgemma-1.5-4b-it` for Chhaya, a skin & heat-health companion for outdoor workers. It reads a skin photo and emits a structured findings JSON directly — no chain-of-thought preamble — with a concern level calibrated against real clinical labels.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"
proc = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, "CodingBad02/chhaya-medgemma-lora-v2").merge_and_unload()

messages = [{"role": "user", "content": [
    {"type": "image", "image": img},          # image BEFORE text
    {"type": "text", "text": "skin check"},
]}]
inputs = proc.apply_chat_template(messages, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(proc.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Output schema:

{"what_i_see","spot":{"type","color","borders","symmetry","texture"},
 "heat_sun_signals":[],"concern":"low|watch|see_doctor","concern_reason",
 "image_quality":"good|limited","summary"}

Training

• —Data: `CodingBad02/chhaya-skin-extract` (ISIC-2024 biopsy-anchored + SCIN real-photo, 1,406 rows). see_doctor class oversampled 3× to improve recall.
• —QLoRA (4-bit nf4, r=16, α=32), frozen vision tower (only the language model adapts), 2 epochs, A100. ~$6 of compute.

Eval (141-image held-out test set, vs base)

Base's higher recall is achieved by labelling 60% of cases "watch" (33% accuracy); this model gives a real triage at 5× fewer tokens. Residual under-warning risk is mitigated by a deterministic ABCDE backstop in the app.

Limitations

Not a medical device. Does not diagnose. A research/education demo built for a hackathon. Misses some malignant-type lesions (recall 0.83). Always pair with clinician review. No heat-rash/sunburn class in training (ISIC/SCIN gap).

Inspired by Sunny by Daniel Bourke.