HuggingFaceBio/carbon-tokenization

1# Collab Editor - Project Specification2 3## 1. Overview4 5A collaborative, real-time scientific article editor deployed as a Hugging Face Space. Users write rich content (math, citations, custom components) in a TipTap-based editor synced via Yjs/Hocuspocus, then publish a self-contained static HTML article. An AI assistant helps with writing and editing.6 7**Stack**: React 18 + TipTap 3 + Yjs (frontend) / Express + Hocuspocus + Node 20 (backend) / Docker on HF Spaces. No CSS-in-JS - all styling via vanilla CSS custom properties.8 9**Relationship to `research-article-template`**: the CSS foundation, design tokens, and visual language come from the [research-article-template](https://huggingface.co/spaces/tfrere/research-article-template) project. The editor imports the template's CSS files (`_variables.css`, `_reset.css`, `_base.css`, `_layout.css`, component partials) and the publisher injects them inline into published HTML. The published output is designed to look identical to articles built with the Astro-based template.10 11---12 13## 2. Architecture14 15```mermaid16graph TB17  subgraph browser [Browser]18    SPA["React SPA<br/>TipTap + Yjs"]19  end20 21  subgraph server [Node Backend - port 8080]22    Express["Express HTTP"]23    Hocuspocus["Hocuspocus<br/>WebSocket"]24    Publisher["Publisher Pipeline<br/>HTML + PDF"]25    Agent["AI Agent<br/>HF Inference"]26    Auth["HF OAuth"]27  end28 29  subgraph storage [Persistence]30    LocalFS["Local FS<br/>data/*.yjs"]31    HFDataset["HF Dataset<br/>articles/ published/"]32  end33 34  SPA -->|"WebSocket /collab"| Hocuspocus35  SPA -->|"REST /api/*"| Express36  Hocuspocus -->|"Database ext"| LocalFS37  LocalFS -->|"schedulePush"| HFDataset38  HFDataset -->|"pullDocument"| LocalFS39  Express --> Publisher40  Express --> Agent41  Express --> Auth42  Publisher --> LocalFS43  Publisher -->|"uploadPublishedAssets"| HFDataset44```45 46**Single process in production**: the backend serves the Vite-built frontend, all REST APIs, the WebSocket collab channel, and static published articles. No reverse proxy needed.47 48---49 50## 3. Data Model (Yjs Shared Types)51 52The entire collaborative state lives in a single `Y.Doc`:53 54- **`Y.XmlFragment("default")`** - TipTap document content (ProseMirror nodes synced via Collaboration extension)55- **`Y.Map("frontmatter")`** - scalar metadata: `title`, `subtitle`, `description`, `published`, `doi`, `template`, `licence`56- **`Y.Array("frontmatter.authors")`** - `{ name, url?, affiliations: number[] }[]`57- **`Y.Array("frontmatter.affiliations")`** - `{ name, url? }[]`58- **`Y.Map("citations")`** - CSL-JSON entries keyed by citation ID59- **`Y.Map("settings")`** - `citationStyle`, `primaryHue`, and future editor preferences60- **`Y.Map("comments")`** - comment threads keyed by `commentId`, each with author/text/resolved61 62All types are concurrently editable by multiple users and persist to `data/default.yjs`.63 64---65 66## 4. Backend Components67 68### 4.1 HTTP Routes69 70| Method | Path | Auth | Purpose |71|--------|------|------|---------|72| `GET` | `/oauth/authorize` | Public | Redirect to HF OAuth |73| `GET` | `/auth/callback` | Public (CSRF state) | Exchange code, set cookie, redirect to `/editor` |74| `GET` | `/api/auth/status` | Cookie | Return `{ authenticated, canEdit, user }` |75| `POST` | `/api/chat` | OAuth (optional) | Stream AI agent responses (HF Inference Providers) |76| `POST` | `/api/publish` | OAuth (canEdit) | Run publish pipeline, generate HTML/PDF |77| `POST` | `/api/admin/reset-document` | OAuth (canEdit) | Delete local `.yjs`, close connections |78| `POST` | `/api/upload` | None (uses cookie for HF) | Upload image (multipart, max 10MB) |79| `POST` | `/api/citations/resolve` | None | Resolve DOI/URL to CSL-JSON |80| `POST` | `/api/citations/format` | None | Format entries to HTML bibliography |81| `POST` | `/api/citations/import-bib` | None | Parse BibTeX to CSL-JSON |82| `GET` | `/editor` | OAuth (canEdit) | Serve SPA (or login page) |83| `GET` | `*` | Public | Serve published article (or login page) |84 85### 4.2 WebSocket Collaboration86 87- Upgrade on `/collab` only; all other paths rejected88- Single document: `DEFAULT_DOC_NAME = "default"`89- Hocuspocus `onAuthenticate`: validates OAuth token if enabled, checks `canEdit`90- `Database` extension: `fetch` reads local `.yjs` or pulls from HF; `store` writes local + schedules HF push (10s debounce)91 92### 4.3 HF Storage93 94- Dataset ID: `HF_DATASET_ID` or `{SPACE_ID}-data`95- Dataset is created **private by default** (`createRepo({ private: true })`). The OAuth grant needs `manage-repos` on the user's first write; subsequent containers reuse the cached token.96- Token: `HF_TOKEN` (env) or cached OAuth token from last authenticated user97- **Documents**: `articles/<name>.yjs` - debounced push on every Hocuspocus store98- **Published assets**: `published/<name>/{index.html, article.pdf, thumb.jpg, meta.json, llms.txt}`99- **Images**: `images/<uuid-filename>` referenced from articles via `/d/images/...` proxy URLs100- `flushAll()` on `SIGTERM`/`SIGINT` to push pending changes101 102### 4.3.1 Storage Status & Recovery103 104The persistence pipeline used to fail silently in multiple places (`createRepo` 403 on a missing scope, `uploadFile` 5xx mid-debounce, `writeFileSync` on a readonly FS, ...) and the editor would happily keep showing "Saved". To make data first-class:105 106- **In-memory tracker** in `hf-storage.ts` records `datasetReady`, `lastLocalSaveAt`, `lastCloudPushAt`, `pendingPush`, `lastError {stage, message, statusCode, at, docName}`. Every write path updates it; every error path records the failure.107- **`GET /api/storage/status`** exposes the tracker (canEdit-gated). The frontend `SyncIndicator` polls it every 5s and displays a three-state badge: green "Saved" / amber "Saving..." / **red "Storage error"** (pulsing, with the exact reason in the tooltip + actionable hint for the 403 / missing-scope case).108- **Eager `ensureDatasetExists`** on first `/api/auth/status` for a canEdit user. A misconfigured fork now surfaces its error within ~10s of login instead of waiting for an edit + 12s debounce cycle.109- **`beforeunload` guard** on the editor: if a local edit is in flight, a push is armed, the WS is offline, or the tracker reports an error, the browser pops the standard "Leave site?" confirm.110- **`GET /api/admin/export-doc`** (canEdit-gated) streams the on-disk `.yjs` snapshot as a download. The escape hatch for disaster recovery: when the cloud push has been failing and the container is about to rebuild, an admin can grab the doc bytes manually.111 112### 4.3.2 Dataset Reverse Proxy (`/d/*`)113 114Since the dataset is private, anonymous viewers of a published article can't fetch its images / PDF / og:image directly from `huggingface.co/datasets/...`. The editor server exposes `GET /d/:path*` as an authenticated forward-proxy:115 116- **Whitelist**: only `images/` and `published/` are reachable; `articles/` (raw `.yjs` drafts) is **always 404** regardless of caller.117- **Token cascade**: request cookie → cached user token → `HF_TOKEN` env → anonymous fetch. The cookie token is also promoted into the cache opportunistically, so the first signed-in viewer warms the proxy for subsequent anonymous viewers within the same container lifetime.118- **Streaming**: WHATWG body piped straight to the Express response - no buffering of full PDFs in Node memory.119- **Caching**: `images/*` is served as `immutable, max-age=1y` (UUID names, never overwritten); `published/*` as `max-age=300, stale-while-revalidate=60` (re-published in place).120- **Error mapping**: upstream 401/403 collapse to 502 so the browser never gets prompted for credentials it can't supply; upstream 404 passes through.121 122### 4.4 Publisher Pipeline123 124```mermaid125flowchart LR126  YDoc["Y.Doc (.yjs)"] --> Extract["extractFromYDoc<br/>frontmatter + JSON"]127  Extract --> GenHTML["generateHTML<br/>@tiptap/html"]128  GenHTML --> PostProc["postProcess<br/>accordion, biblio,<br/>mermaid, htmlEmbed"]129  PostProc --> Render["renderArticleHTML<br/>full HTML page"]130  CSS["loadCSS<br/>template styles"] --> Render131  Render --> LocalWrite["Write local<br/>index.html"]132  Render --> PDF["Playwright<br/>PDF + thumbnail"]133  LocalWrite --> HFUpload["uploadPublishedAssets<br/>HF dataset"]134```135 136- **CSS loading**: reads template CSS files, resolves `@custom-media` queries via `resolveCustomMedia()`, splits into variables/reset/base/layout/components/article/print137- **Post-processing**: accordion divs to `<details>`, bibliography injection, mermaid to `<pre>`, htmlEmbed to `<iframe>`138- **HTML output**: self-contained page with inline CSS, CDN assets (KaTeX, highlight.js, Mermaid), theme toggle (SVG sun/moon), TOC generation (scroll-based, collapsible), lightbox, footer with citation/BibTeX/DOI139- **PDF**: optional Playwright Chromium headless (1200x630 thumbnail + full PDF)140- **Server extensions**: mirror of frontend TipTap extensions for server-side HTML generation141 142### 4.5 Auth143 144- Enabled when `SPACE_ID` + `OAUTH_CLIENT_ID` are set145- OAuth 2.0 flow with HF as provider; cookie `hf_access_token` (httpOnly, secure, sameSite: none)146- `resolveUser`: `whoAmI` via `@huggingface/hub`, then `checkWriteAccess` (Space owner or org member with write/admin role)147- In-memory state map with 10-min TTL for CSRF protection148 149### 4.6 AI Agent150 151- Provider: Hugging Face Inference Providers (`https://router.huggingface.co/v1`), default model `openai/gpt-oss-120b`. Model ids may be suffixed with `:<provider>` (e.g. `meta-llama/Llama-3.3-70B-Instruct:together`) to bypass providers that enforce overly strict tool-call validation (notably Groq) or that don't support the `tools` parameter (Nscale, etc.).152- Auth: per-request bearer token resolved from the editor's OAuth cookie when available, falling back to the server-side `HF_TOKEN`. On a HF Space with `inference-api` scope, no extra secret is needed - the logged-in user pays for their own inference under their HF quota.153- Streaming via Vercel AI SDK `streamText` over `@ai-sdk/openai-compatible`154- Reasoning parts from prior assistant turns are stripped before re-sending the history: providers like Cerebras reject `reasoning_content` on round-trip, and the model doesn't need to see its own past reasoning to continue the conversation.155- **Context**: document text, current selection, frontmatter (sent by frontend with each message)156- **Tools** (declarative, executed client-side by the frontend):157  - `replaceSelection` - replace selected text158  - `insertAtCursor` - insert at cursor position159  - `applyDiff` - search/replace in document160  - `updateFrontmatter` - modify metadata fields161  - `addAuthor` / `removeAuthor` - manage author list162- Agent edits are grouped in a single Yjs `UndoManager` batch for Cmd+Z163 164### 4.7 Citations165 166- Uses `@citation-js/core` with bibtex, doi, csl plugins167- Resolve: DOI URL or identifier to CSL-JSON entries168- Format: entries + style + locale to HTML bibliography169- Import: BibTeX string to CSL-JSON170 171---172 173## 5. Frontend Components174 175### 5.1 App Shell176 177- **No router** - single view with conditional rendering178- **Theme**: CSS custom properties with dynamic primary color from `settings.primaryHue` (OKLCH color model, synced via Yjs settings)179- **Layout**: top bar (undo/redo, settings, publish, user chip) + 3-column CSS grid (TOC / editor / comments)180- **Chat**: floating button bottom-left, `ChatPanel` overlay181- **Modals**: comment dialog, settings drawer, publish confirmation182 183### 5.2 Editor184 185- Creates `Y.Doc` + `HocuspocusProvider` (WebSocket to `/collab`)186- **Seeding**: after provider `synced` event only, if `Y.XmlFragment("default")` is empty, inserts `DEFAULT_CONTENT` + `seedFrontmatter` + `SEED_CITATIONS`187- **Yjs Maps**: `citations`, `settings`, `comments`, `frontmatter` (via dedicated stores)188- **Image handling**: paste/drop with upload to `/api/upload`189 190### 5.3 TipTap Extensions191 192**Built-in (configured)**:193- StarterKit (no codeBlock, no undo), CodeBlockLowlight (all languages), Placeholder, Collaboration, CollaborationCursorV3, Mathematics (KaTeX), Image, Table/Row/Cell/Header194 195**Custom**:196- `CollaborationUndo` - bridges Yjs UndoManager for agent batch edits197- `Comment` - inline mark with `commentId` + `resolved`198- `SlashCommands` - `/` trigger with suggestion popup199- `ImageUpload` - drag-drop upload node with progress200- `Citation` - inline atomic node (key + label), links to `citationsMap`201- `Bibliography` - block node with rendered HTML from citations202- `Glossary` - inline atomic (term + definition tooltip)203- `Footnote` - inline atomic (content shown in footer)204- `Stack` + `StackColumn` - multi-column layout (2/3/4 cols)205 206### 5.4 Component System207 208Registry-based system for MDX-like custom components:209 210| Component | Kind | Purpose |211|-----------|------|---------|212| `accordion` | wrapper | Collapsible section (details/summary) |213| `note` | wrapper | Info/warning/danger/success callout |214| `quoteBlock` | wrapper | Styled blockquote |215| `wide` | wrapper | Content wider than column |216| `fullWidth` | wrapper | Full viewport width |217| `sidenote` | wrapper | Marginal note |218| `reference` | wrapper | Reference container |219| `htmlEmbed` | atomic | External HTML embed (iframe) |220| `hfUser` | atomic | HF user card |221| `rawHtml` | atomic | Raw HTML injection |222| `mermaid` | atomic | Mermaid diagram (live preview) |223 224- **Factory**: `createComponentExtension(def)` generates TipTap nodes from registry definitions (handles both wrapper and atomic kinds)225- **NodeViews**: `WrapperView` (editable content area + chrome), `AtomicView` (placeholder + field editor), `MermaidView` (textarea + SVG preview)226- **Slash menu integration**: each component generates a slash menu item via `getComponentSlashItems()`227 228### 5.5 Frontmatter System229 230- `FrontmatterStore`: wraps `Y.Map` + `Y.Array` for real-time collaborative metadata editing231- `useFrontmatter` hook: React state synced with Yjs observations232- `FrontmatterHero`: WYSIWYG editable hero section (title, subtitle, authors, affiliations, date, DOI)233- `SettingsDrawer`: template variant, SEO, banner, citation style, primary color hue slider, PDF/TOC/licence toggles234- `HueSlider`: OKLCH hue picker (0-360) with live preview, synced to `settingsMap.primaryHue`235 236### 5.6 Other UI237 238- **`TableOfContents`**: extracts headings from TipTap doc, scroll-based active state, collapsible sub-sections239- **`ChatPanel`**: message list + quick actions on selection + input with streaming240- **`CommentPopover`**: positioned comment popover anchored to the active thread (resolve/delete inline)241- **`BubbleToolbar`**: floating toolbar on text selection (bold, italic, link, comment, etc.)242- **`BlockHandle`**: drag handle for block-level nodes243 244### 5.7 CSS Architecture245 246```247styles/248  _variables.css       # Template tokens: --primary-color, breakpoints, @custom-media249  _reset.css           # Scoped reset for article content250  _base.css            # Typography, scoped to article content251  _layout.css          # 3-column grid, .wide/.full-width helpers252  _print.css           # Print styles253  _ui.css              # Editor chrome: buttons, dialogs, drawers, spinner254  tokens.css           # Design tokens (light/dark): text, bg, accent, code, danger, shadows255  article.css          # .tiptap content styles (shared editor/published)256  toc.css              # Editor TOC overrides257  editing.css          # Editor-only: layout, cursors, slash menu258  _publisher.css       # Published-only: theme toggle, wide/fullWidth, footer, lightbox259  components/260    _code.css          # Code blocks + syntax highlighting261    _table.css         # Tables262    _tag.css           # Tags263    _card.css          # Cards264    _mermaid.css       # Mermaid diagrams265    _embed.css         # Embed containers266    _embed-studio.css  # Embed studio overlay267    _hero.css          # Hero section (from template)268    _toc.css           # Base TOC styles (from template)269    _button.css        # Buttons (template)270    _form.css          # Form elements (template)271    _footer.css        # Footer (template)272```273 274The publisher reads these same CSS files server-side and injects them inline into published HTML, using `resolveCustomMedia()` to expand `@custom-media` queries into standard `@media` rules.275 276---277 278## 6. Deployment279 280### 6.1 Docker Build (3-stage)281 2821. **frontend-build**: `npm install` + `npm run build` (Vite)2832. **backend-build**: `npm install` + `npx tsc`2843. **runtime**: `node:20-slim` + Chromium system deps + `npm install --omit=dev` + Playwright Chromium + copy `frontend-dist/` + copy `frontend/src/styles/` to `frontend-styles/`285 286**CMD**: `node dist/server.js` on port 8080.287 288### 6.2 HF Space Configuration (README.md frontmatter)289 290- SDK: `docker`, port `8080`291- OAuth: `hf_oauth: true`, scopes: `manage-repos`, `inference-api`292- Two git remotes: `space` (tfrere/collab-editor, dev) and `prod` (tfrere/research-article-template-editor, production)293 294### 6.3 Environment Variables295 296| Variable | Required | Purpose |297|----------|----------|---------|298| `PORT` | No (default 8080) | HTTP listen port |299| `NODE_ENV` | No | `production` switches to `frontend-dist` path |300| `SPACE_ID` | For OAuth/HF | HF Space identifier, enables OAuth + dataset |301| `SPACE_HOST` | For OAuth | HTTPS callback URL host |302| `OAUTH_CLIENT_ID` | For OAuth | HF OAuth client |303| `OAUTH_CLIENT_SECRET` | For OAuth | HF OAuth secret |304| `OAUTH_SCOPES` | No (default `openid profile`) | OAuth scopes. Add `manage-repos` for dataset persistence and `inference-api` to power AI features with the user's token |305| `HF_DATASET_ID` | No | Override dataset name (default: `{SPACE_ID}-data`) |306| `HF_TOKEN` | For AI chat in local dev | Fallback Hub token for HF API + Inference Providers. Needs the "Make calls to Inference Providers" permission |307| `HF_INFERENCE_MODEL` | No (default `openai/gpt-oss-120b`) | Default chat-completion model id served by HF Inference Providers. May be suffixed with `:<provider>` to pin a specific routing |308| `ENABLE_PDF` | No (default true) | Toggle PDF/thumbnail generation |309 310### 6.4 Local Development311 312```bash313# Terminal 1 - Backend314cd backend && npm install && npm run dev315# Starts on http://localhost:8080316 317# Terminal 2 - Frontend318cd frontend && npm install && npm run dev319# Starts on http://localhost:5678 (proxies /api and /collab to :8080)320```321 322Create a `.env` file in `backend/` with at minimum `HF_TOKEN` for AI chat (must have the "Make calls to Inference Providers" permission). Without `SPACE_ID`, OAuth is disabled and all users can edit.323 324---325 326## 7. Key Data Flows327 328### 7.1 Collaborative Editing329 330```mermaid331sequenceDiagram332  participant ClientA as Client A333  participant Server as Hocuspocus334  participant ClientB as Client B335  participant Disk as Local FS336  participant HF as HF Dataset337 338  ClientA->>Server: WebSocket connect /collab339  Server->>Disk: Database.fetch (load .yjs)340  Server-->>ClientA: sync Y.Doc state341  ClientA->>Server: Y.Doc update (edit)342  Server->>ClientB: broadcast update343  Server->>Disk: Database.store (write .yjs)344  Disk-->>HF: schedulePush (10s debounce)345```346 347### 7.2 Publish Flow348 349```mermaid350sequenceDiagram351  participant User as Editor UI352  participant API as POST /api/publish353  participant HP as Hocuspocus354  participant Pub as Publisher355  participant FS as Local FS356  participant HF as HF Dataset357 358  User->>API: Click Publish359  API->>HP: openDirectConnection360  HP-->>API: Y.Doc snapshot361  API->>FS: Write .yjs snapshot362  API->>Pub: publishDocument()363  Pub->>Pub: extractFromYDoc + loadCSS364  Pub->>Pub: renderArticleHTML + PDF365  Pub->>FS: Write index.html locally366  Pub->>HF: uploadPublishedAssets367  Pub-->>API: { htmlUrl, pdfUrl, success }368  API-->>User: Publish result369```370 371### 7.3 Published Article Lifecycle (Container Restarts)372 373HF Spaces containers are ephemeral. The local filesystem is wiped on every restart (git push, Space rebuild, idle timeout). The published article survives via this restore flow:374 375```mermaid376sequenceDiagram377  participant Container as New Container378  participant FS as Local FS379  participant HF as HF Dataset380  participant Visitor as GET /381 382  Container->>Container: Server starts383  Container->>HF: ensurePublishedRestored()384  HF-->>FS: Pull index.html, PDF, meta.json385  Note over FS: data/published/default/index.html386 387  Visitor->>Container: GET /388  Container->>FS: Check published path389  FS-->>Container: index.html exists390  Container-->>Visitor: Serve published article391```392 393On publish, HTML is **always written locally first** (so `GET /` serves the new version immediately), then uploaded to HF dataset for persistence across restarts.394 395### 7.4 AI Agent Chat396 397```mermaid398sequenceDiagram399  participant User as Chat Panel400  participant Hook as useAgentChat401  participant API as POST /api/chat402  participant LLM as HF Inference403 404  User->>Hook: sendMessage(text)405  Hook->>Hook: Build context (doc, selection, frontmatter)406  Hook->>API: { messages, context }407  API->>LLM: streamText (system prompt + tools)408  LLM-->>API: Stream (text + tool_calls)409  API-->>Hook: SSE stream410  Hook->>Hook: Execute tool calls client-side411  Note over Hook: replaceSelection, applyDiff,<br/>updateFrontmatter, etc.412  Hook->>Hook: UndoManager batch for Cmd+Z413```414 415---416 417## 8. Current Limitations and Known Issues418 419- **Test suite in progress**: P0 tests being added (see `docs/TESTS.md`)420- **Single document**: only `"default"` document supported; no multi-doc421- **Single-user token**: last OAuth token cached globally for all HF API calls422- **No rate limiting** on `/api/chat` or `/api/citations/*`423- **XSS surface**: `meta.licence` and `biblioHtml` not escaped in published HTML424- **WS debug logging**: every WebSocket message logged in production425- **No `.env.example`**: environment variables documented only in code426