breakpointsoftware/document-parser

1# Multi-Tenant Document Parser - Configuration & Usage Guide2 3## Overview4 5The document parser now supports two operational modes:6 71. **Multi-Tenant Mode (NEW)** - Load all active tenants from Firebase and process their enabled rules82. **Legacy Mode** - Single tenant configuration via environment variables (backward compatible)9 10## Architecture11 12### Data Model13 14All tenant and rule configurations are stored in Firestore under the `tenants` collection:15 16```17TENANTS_COLLECTION (Firestore)18├── {tenant_id}19│   ├── tenant_id: unique tenant ID20│   ├── name: "Company/User Name"21│   ├── active: boolean (Account Status)22│   ├── created_at: timestamp23│   ├── updated_at: timestamp24│   ├── credentials (object, 1:1 relationship)25│   │   ├── openai_api_key: "OpenAI Key"26│   │   └── google_service_account_json: "Service account JSON"27│   └── rules (subcollection, 1:N relationship)28│       └── {rule_id}29│           ├── rule_id: "unique rule ID"30│           ├── rule_name: "Invoices to Finance"31│           ├── source_folder_id: "Google Drive Source Folder ID"32│           ├── target_folder_id: "Google Drive Destination Folder ID"33│           ├── target_sheet_id: "Google Sheets Spreadsheet ID"34│           ├── sheet_tab_name: "Tab inside Google Sheet"35│           ├── parsing_prompt: "Custom LLM Instructions"36│           └── is_enabled: boolean (Rule Toggle)37```38 39### Processing Flow40 41```42Multi-Tenant Mode:431. orchestrate_all_active_tenants()44   └─> For each active tenant:45       └─> For each enabled rule:46           └─> orchestrate_single_rule()47               ├─> Scan rule.source_folder_id48               ├─> Parse documents with rule.parsing_prompt49               ├─> Move to rule.target_folder_id50               └─> Send to rule.target_sheet_id51```52 53## Usage54 55### Quick Start (Using .env)56 57If you have existing `.env` configuration, multi-tenant mode will automatically create a default tenant:58 59```bash60# Just run it! No Firebase setup needed initially61python document_orchester.py --send62 63# The system will:64# 1. Check for active tenants in Firebase65# 2. If none found, create "default_tenant" from your .env vars66# 3. Process documents using the default configuration67```68 69**This is the easiest path if you're migrating from single-tenant mode.**70 71### Mode 1: Multi-Tenant Mode (Default)72 73```bash74# Process all active tenants and their enabled rules75python document_orchester.py --model gpt-4o --send76 77# Optional flags:78# --no-subfolders    Only scan direct files (don't recurse)79# --send             Send results to Google Sheets80# --output FILE      Save JSON summary to file81```82 83### Mode 2: Legacy Single-Tenant Mode84 85```bash86# Use env var configuration (backward compatible)87python document_orchester.py --legacy --model gpt-4o --send88```89 90### Gradio Interface91 92Start the web app:93```bash94python app.py95```96 97Two orchestrator endpoints are now available:98- **Single Tenant (Legacy)** - Uses env var configuration99- **Multi Tenant** - Loads all active tenants from Firebase100 101## Configuration102 103### Automatic Default Tenant (No Firebase Setup Required)104 105When you run multi-tenant mode without any tenants in Firebase:106 1071. The system checks for active tenants in Firestore1082. If none found, it creates a **default tenant** from your `.env` file1093. Maps env vars like this:110 111| .env Variable | Default Tenant |112|---|---|113| `OPENAI_API_KEY` | `credentials.openai_api_key` |114| `GOOGLE_SERVICE_ACCOUNT_JSON` | `credentials.google_service_account_json` |115| `GOOGLE_DRIVE_FOLDER_IDS` (first) | `rule.source_folder_id` |116| `GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID` | `rule.target_folder_id` |117| `GOOGLE_SHEETS_SPREADSHEET_ID` | `rule.target_sheet_id` |118 119**Example: Your existing `.env`**120```env121OPENAI_API_KEY=sk-abc123...122GOOGLE_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"my-project",...}123GOOGLE_DRIVE_FOLDER_IDS=1ABC_folder_id_here124GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID=1XYZ_destination_folder_id125GOOGLE_SHEETS_SPREADSHEET_ID=18qKL...126```127 128**Creates this default tenant (in memory, not saved to Firebase):**129```json130{131  "tenant_id": "default_tenant",132  "name": "Default Tenant (from .env)",133  "active": true,134  "credentials": {135    "openai_api_key": "sk-abc123...",136    "google_service_account_json": "{...}"137  },138  "rules": [139    {140      "rule_id": "default_rule",141      "rule_name": "Default Processing Rule",142      "source_folder_id": "1ABC_folder_id_here",143      "target_folder_id": "1XYZ_destination_folder_id",144      "target_sheet_id": "18qKL...",145      "sheet_tab_name": "Parsed",146      "is_enabled": true147    }148  ]149}150```151 152The default tenant provides automatic setup:153- ✅ Created automatically if no Firebase tenants exist154- ✅ Saved to Firebase **BEFORE** orchestration starts155- ✅ Future runs load from Firebase instead of recreating156- ✅ Try multi-tenant mode without any setup157- ✅ Migrate gradually from single-tenant to multi-tenant158- ✅ Keep existing workflows unchanged159 160**The flow:**1611. First run: No Firebase tenants → Create default_tenant from .env → **Save to Firebase** → Process documents1622. Second run: Firebase has default_tenant → Load it → Process documents1633. Add more tenants anytime in Firebase → They're all processed together164 165### Multi-Tenant Setup (Firebase)166 1671. Create a Firestore collection called `tenants` (or set via `FIREBASE_TENANTS_COLLECTION` env var)168 1692. Add tenant documents with this structure:170 171```json172{173  "name": "Acme Corp",174  "active": true,175  "created_at": "2024-01-01T00:00:00Z",176  "updated_at": "2024-01-01T00:00:00Z",177  "credentials": {178    "openai_api_key": "sk-...",179    "google_service_account_json": "{...full service account json...}"180  }181}182```183 1843. Create a **rules** subcollection inside the tenant document, and add rules with this structure:185 186```json187{188  "rule_id": "invoices_2024",189  "rule_name": "Invoices to Finance",190  "source_folder_id": "1ABC...",191  "target_folder_id": "1XYZ...",192  "target_sheet_id": "18qKL...",193  "sheet_tab_name": "Invoices",194  "parsing_prompt": "Extract invoice details following these rules...",195  "is_enabled": true196}197```198 199### Environment Variables200 201**Required for both modes:**202```203FIREBASE_SERVICE_ACCOUNT_JSON    # Firebase credentials (JSON string or use GOOGLE_SERVICE_ACCOUNT_JSON)204OPENAI_API_KEY                   # (only for legacy mode, multi-tenant uses tenant credentials)205```206 207**Optional:**208```209FIREBASE_TENANTS_COLLECTION      # Default: "tenants"210FIREBASE_DOCUMENTS_COLLECTION    # Default: "processed_documents"211FIREBASE_TRACKER_COLLECTION      # Default: "processed_files"212FIREBASE_TRACK_PROCESSED         # Default: "true"213OPENAI_MODEL                     # Default: "gpt-4o"214GOOGLE_DRIVE_INVOICES_BASE_PATH  # Default: "Facturas"215ORCHESTRATOR_API_KEY             # For API authentication216```217 218## Firebase Storage Structure219 220### Document Tracking (Multi-Tenant)221 222Each tenant's processed files are tracked in nested collections:223 224```225tenants/{tenant_id}/processed_files/{file_hash}226└─ file_hash: "sha256 hash"227└─ source_file: "filename"228└─ processed_at: "timestamp"229 230tenants/{tenant_id}/processed_documents/{document_id}231└─ document_id: "drive file id"232└─ parsed_data: {...}233└─ status: "Parsed|Modified|Sent"234```235 236### Creating Tenants Programmatically237 238```python239from firebase_tenant_config import (240    FirebaseTenantConfigManager,241    TenantConfig,242    CredentialsObject,243    RuleObject,244)245 246manager = FirebaseTenantConfigManager()247 248# Create credentials249credentials = CredentialsObject(250    openai_api_key="sk-...",251    google_service_account_json="..." # full JSON as string252)253 254# Create a rule255rule = RuleObject(256    rule_id="invoices_2024",257    rule_name="Invoices to Finance",258    source_folder_id="1ABC...",259    target_folder_id="1XYZ...",260    target_sheet_id="18qKL...",261    sheet_tab_name="Invoices",262    parsing_prompt="Extract...",263    is_enabled=True,264)265 266# Create tenant267tenant = TenantConfig(268    tenant_id="acme-corp",269    name="Acme Corp",270    active=True,271    credentials=credentials,272    rules=[rule],273)274 275# Save to Firebase276manager.save_tenant(tenant)277 278# List all active tenants279active_tenants = manager.list_active_tenants()280 281# Get enabled rules for a tenant282enabled_rules = manager.get_enabled_rules("acme-corp")283 284# Add a rule285manager.add_rule("acme-corp", new_rule)286 287# Remove a rule288manager.remove_rule("acme-corp", "invoices_2024")289```290 291## Output Format292 293### Multi-Tenant Orchestration Result294 295```json296{297  "ok": true,298  "tenants_processed": 2,299  "total_parsed": 15,300  "total_modified": 3,301  "total_sent": 18,302  "total_moved": 18,303  "total_corrupted": 2,304  "total_errors": 1,305  "tenants_results": [306    {307      "tenant_id": "tenant_123",308      "tenant_name": "Acme Corp",309      "ok": true,310      "rules_processed": 2,311      "parsed": 10,312      "modified": 2,313      "sent": 12,314      "moved": 12,315      "corrupted": 1,316      "errors": 0,317      "rules_results": [318        {319          "ok": true,320          "tenant_id": "tenant_123",321          "rule_id": "invoices_2024",322          "rule_name": "Invoices to Finance",323          "scanned": 20,324          "to_process": 15,325          "skipped": 5,326          "parsed": 10,327          "modified": 2,328          "sent": 12,329          "moved": 12,330          "corrupted": 1,331          "errors": 0,332          "processed_items": [...],333          "skipped_items": [...]334        }335      ]336    }337  ]338}339```340 341## Migration from Legacy Mode342 343**No migration needed!** The system maintains full backward compatibility:344 3451. Existing env var configuration continues to work with `--legacy` flag3462. New multi-tenant mode coexists without affecting legacy deployments3473. To enable multi-tenant: Set up tenants in Firebase and run without `--legacy`348 349## Workflow Examples350 351### Simple: Use Existing .env (No Firebase Setup)352 353The **easiest** way to get started with multi-tenant mode:354 355```bash356# 1. Ensure your .env has these vars (you probably already do):357#    OPENAI_API_KEY358#    GOOGLE_SERVICE_ACCOUNT_JSON359#    GOOGLE_DRIVE_FOLDER_IDS360#    GOOGLE_DRIVE_INVOICES_ROOT_FOLDER_ID361#    GOOGLE_SHEETS_SPREADSHEET_ID362 363# 2. Just run multi-tenant orchestration:364python document_orchester.py --send365 366# That's it! The system will:367# - Check for tenants in Firebase368# - Find none, create default_tenant from .env369# - Process documents exactly as before370# - Results go to Google Sheets and Drive371```372 373**No code changes, no Firebase setup, just works!**374 375### Advanced: Multiple Tenants in Firebase376 377For multiple tenants with different configurations:378 379### Step 1: Set Up Tenant in Firebase380 381```python382# Run this once to initialize a tenant383from firebase_tenant_config import FirebaseTenantConfigManager, TenantConfig, CredentialsObject, RuleObject384 385manager = FirebaseTenantConfigManager()386 387credentials = CredentialsObject(388    openai_api_key="sk-xxx",389    google_service_account_json="json_string_here"390)391 392rule = RuleObject(393    rule_id="rule_1",394    rule_name="Invoice Processing",395    source_folder_id="drive_folder_id",396    target_folder_id="drive_dest_folder_id",397    target_sheet_id="sheets_id",398    sheet_tab_name="Invoices",399    is_enabled=True,400)401 402tenant = TenantConfig(403    tenant_id="my_tenant",404    name="My Company",405    active=True,406    credentials=credentials,407)408 409manager.save_tenant(tenant)410manager.add_rule("my_tenant", rule)411```412 413### Step 2: Run Orchestration414 415```bash416# This will automatically:417# 1. Load all active tenants from Firebase (not just one!)418# 2. For each tenant, process all enabled rules419# 3. Track results in tenant-specific collections420python document_orchester.py --model gpt-4o --send421```422 423### Step 3: Review Results424 425Results are stored in:426- Google Drive: Documents moved to `{target_folder_id}/{YYYY}{MM}/`427- Google Sheets: Parsed data appended to `{target_sheet_id}` sheet428- Firebase: Document metadata in `tenants/{tenant_id}/processed_documents/`429 430## Troubleshooting431 432### "No active tenants found"433- Check that `active: true` in Firebase tenant document434- Verify `FIREBASE_TENANTS_COLLECTION` env var is correct435- Confirm Firebase credentials are valid436 437### "No enabled rules"438- Check that `is_enabled: true` for at least one rule439- Verify rule has valid `source_folder_id`440 441### Credentials error442- Ensure `credentials.openai_api_key` and `credentials.google_service_account_json` are set443- Verify JSON strings are properly formatted444 445### Mixing modes446- Don't mix multi-tenant and legacy configurations447- Use either all Firebase-based tenants OR all env vars, not both448- When switching modes, ensure all required env vars/Firebase docs exist449 450## API Endpoints451 452### Gradio Interface453 454Two REST-compatible endpoints are exposed:455 456**Legacy Single-Tenant:**457```458POST /api/orchestrate_drive_documents459{460  "model": "gpt-4o",461  "include_subfolders": true,462  "send_to_sheet": true,463  "api_key": "your_orchestrator_api_key"464}465```466 467**Multi-Tenant:**468```469POST /api/orchestrate_all_tenants470{471  "model": "gpt-4o",472  "include_subfolders": true,473  "send_to_sheet": true,474  "api_key": "your_orchestrator_api_key"475}476```477 478## Performance Considerations479 480- Multi-tenant processing is sequential (one tenant at a time, then one rule at a time)481- Each tenant/rule maintains separate Firebase document tracking482- Large scale deployments may benefit from rule parallelization (future enhancement)483 484## Managing Rules485 486### Adding a Rule to an Existing Tenant487 488You can programmatically add rules to any tenant:489 490```python491from firebase_tenant_config import (492    FirebaseTenantConfigManager,493    RuleObject,494)495 496manager = FirebaseTenantConfigManager()497 498# Create a new rule499new_rule = RuleObject(500    rule_id="new_rule_123",501    rule_name="Additional Processing Rule",502    source_folder_id="1NEW_folder_id",503    target_folder_id="1NEW_destination_id",504    target_sheet_id="1NEW_sheet_id",505    sheet_tab_name="Sheet Tab Name",506    parsing_prompt="Custom parsing instructions...",507    is_enabled=True,508)509 510# Add to tenant511manager.add_rule("my_tenant", new_rule)512```513 514### Cloning a Rule515 516Use the provided script to clone the default rule:517 518```bash519python add_rule_from_default.py new_rule_id "New Rule Name"520```521 522This creates a new rule in the **default_tenant** with the same configuration (folder IDs, sheet, etc.) as the default rule. Perfect for creating multiple processing pipelines with identical settings.523 524### Removing a Rule525 526```python527from firebase_tenant_config import FirebaseTenantConfigManager528 529manager = FirebaseTenantConfigManager()530manager.remove_rule("my_tenant", "rule_id_to_remove")531```532 533## Future Enhancements534 535Potential improvements:536- Parallel rule processing within a tenant537- Rule scheduling (cron-like execution)538- Audit logging for rule changes539- Rule versioning and rollback540- Web UI for rule management541