ML-Intern-lab/citrus-disease-vlm-instruct

Citrus Disease VLM Instruct

An instruction-tuning dataset for training a small vision-language model (VLM) to look at a photo of a citrus leaf, fruit or shoot, name the disease, pest or nutrient deficiency, explain the cause and symptoms, and recommend both biological/organic and chemical management.

Every example pairs one image with a chat conversation in the format used by TRL's SFTTrainer for multimodal models (Qwen-VL, SmolVLM, Idefics, LLaVA and similar).

What is in it

• —Train examples: 3017
• —Test examples: 335

The images come from three CC BY 4.0 datasets published in Data in Brief and mirrored on the Hub by the AgML project. The text answers were generated from a curated knowledge base (knowledge_base.py in this repo) written at the level of an agricultural-extension leaflet, so wording varies across examples but the agronomic content for each class is consistent.

Classes (21)

Two vague source classes (foliage_damaged, yellow_leaves) were dropped.

Conversation types

Every treatment answer ends with a safety line asking the user to confirm with local extension services and follow the local pesticide label.

Format

{
  "images": [PIL.Image],                       # exactly one image
  "messages": [
    {"role": "user", "content": [{"type": "image", "text": None},
                                 {"type": "text", "text": "What is wrong with this citrus plant and how do I treat it?"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "This looks like Citrus canker, caused by ..."}]}
  ],
  "label": "canker",
  "category": "disease_bacterial",
  "source_dataset": "Project-AgML/orange_leaf_disease_classification",
  "source_label": "citrus_canker",
  "plant_part": "leaf",
  "conversation_type": "diagnose_treat"
}

Images are RGB JPEG, longest side 768 px.

Training with TRL

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

ds = load_dataset("ML-Intern-lab/citrus-disease-vlm-instruct")
trainer = SFTTrainer(
    model="Qwen/Qwen3.5-2B",          # or HuggingFaceTB/SmolVLM2-2.2B-Instruct
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    args=SFTConfig(output_dir="citrus-vlm", per_device_train_batch_size=2,
                   gradient_accumulation_steps=8, num_train_epochs=2,
                   learning_rate=1e-4, bf16=True, max_length=2048),
    peft_config=__import__("peft").LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
)
trainer.train()

A suggested recipe: freeze the vision encoder, LoRA rank 16 on the language model, 2 epochs on one A100 or L40S, evaluate label accuracy on test by string-matching the disease name in the first assistant turn.

Limitations

• —Treatment text is templated from a knowledge base, not written per image. The model will learn the diagnosis from the image and the advice from the class. Review the knowledge base with an agronomist before using outputs for real decisions.
• —Chemical recommendations name active ingredients only. Registration, dose and pre-harvest intervals differ by country.
• —melanose and scab have very few images. shot_hole and die_back are symptom classes that can have several causes.
• —Sources were photographed in Pakistan, Mexico and Bangladesh; expect a domain shift on other varieties, backgrounds and cameras.
• —Nutrient deficiency images come from a single orchard study and may encode background cues.

Sources and attribution (all CC BY 4.0)

• —Rauf, H. T. et al. (2019). A citrus fruits and leaves dataset for detection and classification of citrus diseases through machine learning. Data in Brief 26, 104340. Mendeley Data doi:10.17632/3f83gxmv57.2. Hub mirror: Project-AgML/citrus_fruit_leaf_disease_classification.
• —Gomez-Flores, W., Garza-Saldana, J. J., Varela-Fuentes, S. E. (2024). CitrusUAT: A dataset of orange Citrus sinensis leaves for abnormality detection using image analysis techniques. Data in Brief 52, 109908. Zenodo doi:10.5281/zenodo.8294078. Hub mirror: Project-AgML/citrusuat_disease_classification.
• —Emon, Y. R., Ahad, M. T., Rabbany, G. (2024). Multi-format open-source sweet orange leaf dataset for disease detection, classification, and analysis. Data in Brief 55, 110713. Mendeley Data doi:10.17632/f7cr74mwpj.2. Hub mirror: Project-AgML/orange_leaf_disease_classification.

This derived dataset is released under CC BY 4.0. Build script and knowledge base are included in the repo.