Dattu005/webhook-processor

1---2title: Webhook Processor API3emoji: 🚀4colorFrom: indigo5colorTo: blue6sdk: docker7pinned: false8app_port: 80009---10 11# Webhook Processor Service (FastAPI + MongoDB + Redis/RQ)12 13## Overview14 15This service receives transaction webhooks from an external payment provider (e.g. Razorpay), acknowledges them immediately with HTTP 202, and processes them asynchronously in a background worker.16 17Why this matters:18 19- Payment gateways retry webhooks and expect a super-fast 2xx response.20- You _must not_ block that response while doing heavy work like settlement, reconciliation, downstream API calls, etc.21- You _must_ handle duplicate webhooks safely (idempotency).22 23This backend solves that.24 25---26 27## Features28 29### Endpoints30 31- `GET /`32  Health check. Returns `{ "status": "HEALTHY", "current_time": "<UTC_ISO8601>" }`33- `POST /v1/webhooks/transactions`34  Primary webhook receiver.35 36  - Validates and persists the transaction in MongoDB with status `"RECEIVED"`.37  - Enqueues a background job in Redis/RQ.38  - Returns **HTTP 202 Accepted** in under ~500ms.39 40- `GET /v1/transactions/{transaction_id}`41  Returns the live status:42 43  ```json44  {45    "transaction_id": "txn_abc123",46    "source_account": "acc_user_789",47    "destination_account": "acc_merchant_456",48    "amount": 1500,49    "currency": "INR",50    "status": "PROCESSED",51    "created_at": "2025-10-29T10:30:00Z",52    "processed_at": "2025-10-29T10:30:30Z"53  }54  ```55 56Possible `status` values:57 58- `"RECEIVED"` → saved, waiting to process59- `"PROCESSING"` → worker claimed it60- `"PROCESSED"` → finished successfully61 62### Background worker63 64- We run an RQ worker connected to Redis.65- The worker:66 67  1. Atomically flips status from `RECEIVED` → `PROCESSING` in MongoDB.68  2. Sleeps 30 seconds to simulate expensive external calls (like confirming with a payment gateway).69  3. Updates status to `PROCESSED` and sets `processed_at`.70 71### Idempotency72 73We guarantee that repeated webhooks with the same `transaction_id` do _not_ create duplicates or re-run the job:74 751. **MongoDB unique index** on `transaction_id` means first insert wins, subsequent inserts with the same ID are ignored.762. We enqueue the RQ job with `job_id = transaction_id`. RQ will refuse to enqueue a second job with the same ID.773. The worker uses a conditional update (`status == "RECEIVED"`) to take ownership. If it’s already `"PROCESSING"` or `"PROCESSED"`, it no-ops.78 79This matches real-world payment gateway behavior (gateways often retry webhooks; you must handle duplicates safely).80 81---82 83## Tech Choices84 85- **FastAPI**: async-friendly, fast, great request validation + auto docs.86- **MongoDB Atlas (M0 Free Tier)**: persistent transaction store.87  Atlas’ M0 tier is “Free Forever,” ~512MB storage, shared RAM/CPU, meant for prototyping and dev, and runs in the cloud. (Refs: MongoDB Atlas “Free Forever” / M0 sandbox, ~512MB storage and shared compute, marketed for learning and prototyping.) [Sources: MongoDB pricing pages and docs as of Oct 29, 2025, which describe M0 as free forever with ~512MB storage and shared resources.]88- **Redis + RQ**: queue system for async processing.89 90  - Redis stores jobs91  - RQ worker pulls jobs and runs long-running work outside the request/response path92 93---94 95## Project Structure96 97```text98app/99  main.py                     # FastAPI app100  api/routes.py               # All routes (/, /v1/webhooks/transactions, /v1/transactions/{id})101  core/config.py              # Env config (Mongo URL, Redis URL, delay seconds)102  db/mongo.py                 # Mongo client + unique index on transaction_id103  repository/transactions.py  # DB logic (create/get/update with idempotency)104  queues/redis_conn.py        # Redis connection + RQ Queue105  queues/tasks.py             # enqueue helper (q.enqueue(...))106  workers/processor.py        # the background job (30s delay, status transitions)107  schemas/transaction.py      # Pydantic models108 109Dockerfile110docker-compose.yml111requirements.txt112.env.example113README.md114```115 116---117 118## How To Run Locally (Docker)119 120Requirements:121 122- Docker Desktop running123 124Then:125 126```bash127docker compose up --build128# or: docker-compose up --build129```130 131This will start 4 containers:132 133- `api` → FastAPI app on [http://localhost:8000](http://localhost:8000)134- `worker` → RQ background worker135- `mongo` → MongoDB136- `redis` → Redis137 138Environment values inside those containers are already configured so `api` talks to `mongo` and `redis`, and `worker` talks to both.139 140### Test locally141 1421. Health:143 144   ```bash145   curl http://localhost:8000/146   ```147 1482. Send a fake webhook:149 150   ```bash151   curl -X POST http://localhost:8000/v1/webhooks/transactions \152     -H "Content-Type: application/json" \153     -d '{154       "transaction_id": "txn_demo_123",155       "source_account": "acc_user_789",156       "destination_account": "acc_merchant_456",157       "amount": 1500,158       "currency": "INR"159     }'160   ```161 162   Expected `202 Accepted` and body like:163 164   ```json165   {166     "accepted": true,167     "transaction_id": "txn_demo_123",168     "status": "RECEIVED"169   }170   ```171 1723. Immediately check:173 174   ```bash175   curl http://localhost:8000/v1/transactions/txn_demo_123176   ```177 178   You’ll see `"status": "RECEIVED"`.179 1804. After ~30 seconds:181 182   ```bash183   curl http://localhost:8000/v1/transactions/txn_demo_123184   ```185 186   You’ll now see `"status": "PROCESSED"` and a `processed_at` timestamp.187 1885. Re-send the same webhook again (same `transaction_id`).189   You’ll still get 202, but DB and processing won’t duplicate, proving idempotency.190 191---192 193## How To Run Locally (manual / venv dev mode)194 1951. Create venv and install deps:196 197   ```bash198   python3.11 -m venv .venv199   source .venv/Scripts/activate # for windows200   pip install --upgrade pip201   pip install -r requirements.txt202   ```203 2042. Run Redis + Mongo via Docker:205 206   ```bash207   docker run --name redis-local -p 6379:6379 -d redis:7208   docker run --name mongo-local -p 27017:27017 -d mongo:7209   ```210 2113. Export env vars:212 213   ```bash214   export MONGO_URL="mongodb://localhost:27017"215   export MONGO_DB="webhooks_db"216   export REDIS_URL="redis://localhost:6379/0"217   export PROCESS_DELAY_SECONDS=30218   ```219 2204. Run API:221 222   ```bash223   uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload224   ```225 2265. In a second terminal (same venv & env vars):227 228   ```bash229   rq worker -u redis://localhost:6379/0 transactions230   ```231 232---233 234## Contact / Notes235 236If this is being reviewed:237 238- Please start with `docker compose up --build` and test `/v1/webhooks/transactions`.239- Then review `app/workers/processor.py` + `app/repository/transactions.py` to see the idempotency logic.240