HuggingFaceBio/carbon-tokenization

1# Architecture2 3## Overview4 5The collab-editor is a collaborative article editor built for Hugging Face Spaces. It runs as a single Docker container serving both the backend (Express + Hocuspocus) and the frontend (React + TipTap).6 7```8┌─────────────────────────────────────────────────────────┐9│  Docker Container (port 8080)                           │10│                                                         │11│  ┌──────────────────┐   ┌────────────────────────────┐  │12│  │  Express Server   │   │  Hocuspocus (Y.js collab) │  │13│  │                   │   │                            │  │14│  │  /api/*           │   │  /collab (WebSocket)       │  │15│  │  /published/*     │   │                            │  │16│  │  /editor          │   │                            │  │17│  │  / (published)    │   │                            │  │18│  └──────────────────┘   └────────────────────────────┘  │19│                                                         │20│  ┌──────────────────┐   ┌────────────────────────────┐  │21│  │  Publisher        │   │  Frontend (static)         │  │22│  │  (HTML renderer)  │   │  /editor -> SPA            │  │23│  │  (PDF generator)  │   │  React + TipTap            │  │24│  └──────────────────┘   └────────────────────────────┘  │25└─────────────────────────────────────────────────────────┘26```27 28## Key directories29 30| Path | Description |31|------|-------------|32| `backend/src/server.ts` | Entry point: imports `createApp()`, starts listener, signal handlers |33| `backend/src/create-app.ts` | Express app factory: routes, Hocuspocus, WebSocket, middleware |34| `backend/src/publisher/` | HTML rendering, PDF generation, bibliography formatting |35| `backend/src/publisher/html-renderer.ts` | Converts TipTap JSON to static HTML page |36| `backend/src/publisher/extensions.ts` | Server-side TipTap extensions (mirrors frontend) |37| `backend/src/shared/component-defs.ts` | Shared component definitions (single source of truth) |38| `backend/src/utils.ts` | Shared utilities: `docPath()`, `sanitizeName()`, injectable `DATA_DIR` |39| `backend/src/auth.ts` | OAuth flow, token extraction, user resolution |40| `backend/src/hf-storage.ts` | HF dataset sync (push/pull documents and assets) |41| `frontend/src/editor/` | TipTap editor, toolbars, components |42| `frontend/src/editor/components/registry.ts` | Component registry (imports from shared defs) |43| `frontend/src/styles/` | All CSS files |44 45## Styling architecture46 47### CSS layers48 49The project uses five CSS layers, loaded in this order by `main.tsx`:50 511. **Template foundation** (`_variables.css`, `_reset.css`, `_base.css`, `_layout.css`, `_print.css`, `components/*`)52   - Defines the article's visual identity (typography, grid, components)53   - Uses CSS custom properties (`--text-color`, `--surface-bg`, `--primary-color`, etc.)54   - Layout tokens centralize grid math (`--layout-toc-width`, `--layout-content-width`, `--layout-gap`, breakpoints)55   - Shared between the editor preview and the published output56 572. **Editor chrome** (`_ui.css`)58   - Styles the editor UI: top-bar, sidebars, dialogs, chat panel, embed studio59   - Uses `--ed-*` custom properties for the dark editor theme60   - Only loaded in the editor, never in published output61 623. **Design tokens** (`tokens.css`)63   - Light/dark theming tokens for both editor and article64   - Text, background, border, accent, code highlighting, danger, shadows65   - Supports `data-theme` attribute and `prefers-color-scheme` media query66   - Shared between editor and published output (injected by publisher)67 684. **Shared article styles** (`article.css`, `toc.css`)69   - Article content styles shared between editor preview and published output70   - `article.css` includes wrapper components (Note, Stack, Quote, Sidenote...), with editor-specific variants scoped by `.editor-app`71 725. **Editor-only styles** (`styles/editor/*.css`, 5 files)73   - `_layout.css`: grid overrides (3-col symmetric, aligned with template), TOC drawer, responsive breakpoints (1100px collapse, 768px mobile)74   - `_chrome.css`: ProseMirror editing visuals (placeholder, selection, cursors, comment marks, math editing)75   - `_block-tools.css`: block handles (drag + add) and slash menu76   - `_panels.css`: image upload card, footnote tooltip, citation panel77   - `_hero-editable.css`: transparent click-to-edit inputs for FrontmatterHero78 796. **Publisher CSS** (`_publisher.css`)80   - Styles specific to the published static HTML page81   - Theme toggle animations, wide/fullWidth breakout, sidenote float, lightbox, PDF link82   - Only injected by the HTML renderer (backend), never loaded in the editor83 84### No CSS-in-JS85 86The project does not use MUI, Emotion, or any CSS-in-JS library. All styling is done via:87- CSS custom properties for theming88- Vanilla CSS files with BEM-like class naming89- `Floating UI` for tooltip positioning (lightweight, no CSS-in-JS)90 91### Color theming92 93- The article area uses `data-theme="light"` / `data-theme="dark"` with CSS variable overrides94- The editor chrome is always dark, using `--ed-*` tokens95- The primary accent color is controlled via `--primary-color` (synced via Yjs settings)96 97## HF Spaces constraints98 99### Iframe embedding100 101When deployed as a HF Space, the app runs inside an iframe. This affects:102 103- **Viewport width**: the iframe is ~968px wide, not the full browser width104- **No `target="_top"` navigation**: links open within the iframe unless using `target="_blank"`105- **OAuth flow**: the OAuth callback URL must match the Space URL106- **CSP restrictions**: sandboxed iframes may restrict certain APIs107 108### Two-page architecture109 110| URL | What it serves |111|-----|----------------|112| `/` | Published article (static HTML) or login prompt |113| `/editor` | The SPA editor (requires authentication) |114 115This means:116- The published article is a completely standalone HTML file117- It does NOT load React or any JS framework118- The editor is a separate React SPA at `/editor`119 120### CSS cascade121 122Because the published HTML is self-contained (all CSS inlined in `<style>`), there are no CSS conflicts with the HF Spaces iframe CSS. The editor uses Vite's CSS pipeline.123 124## Shared component registry125 126Component definitions (name, kind, fields, defaults) are defined once in `backend/src/shared/component-defs.ts`. This file is the single source of truth.127 128- **Backend** (`extensions.ts`): imports `SHARED_COMPONENT_DEFS` to generate TipTap server extensions for `generateHTML()`129- **Frontend** (`registry.ts`): imports `SHARED_COMPONENT_DEFS` via Vite alias `#shared` and decorates each entry with UI metadata (icon, label, description, placeholders)130 131Adding a new component:1321. Add the entry to `shared/component-defs.ts`1332. Add UI metadata to `frontend/src/editor/components/registry.ts` in `UI_META`1343. Add CSS for the published view to `frontend/src/styles/_publisher.css`135 136## Publisher pipeline137 138```139Y.Doc -> TiptapTransformer -> JSON -> generateHTML() -> postProcess() -> full HTML page140                                                                            |141                                                                            v142                                                                    PDF (Playwright)143                                                                            |144                                                                            v145                                                                    Upload to HF dataset146```147 148### Post-processing (linkedom)149 150The `postProcess()` function uses `linkedom` for DOM manipulation instead of regex:151- Accordion `<div>` -> `<details>/<summary>`152- Citation `<span>` -> `<a>` links with bibliography anchors153- Bibliography placeholder -> formatted HTML with entry IDs154- Mermaid `<div>` -> `<pre class="mermaid">`155- HtmlEmbed `<div>` -> `<iframe>`156- Footnotes -> superscript links + appended section157 158### Preview endpoint159 160`GET /api/preview/:docName` renders the HTML without saving or uploading. Useful for testing the publisher pipeline.161 162## Auth and security163 164- AI chat routes (`/api/chat`, `/api/embed-chat`) are auth-guarded when OAuth is enabled165- The `requireEditor` middleware checks cookie token and verifies write access166- Published articles are served without auth (public)167 168## Testing169 170Tests use Vitest + Supertest. Run with `npm test` from `backend/`.171 172| Test file | What it covers |173|-----------|----------------|174| `tests/publisher.test.ts` | Y.Doc extraction, HTML generation, post-processing, idempotency |175| `tests/html-renderer-snapshot.test.ts` | Snapshot tests for each postProcess transformation |176| `tests/security.test.ts` | XSS prevention in published HTML |177| `tests/css-resolution.test.ts` | @custom-media resolution |178| `tests/utils.test.ts` | Path sanitization utilities |179| `tests/auth.test.ts` | Token extraction, OAuth configuration |180| `tests/hf-storage.test.ts` | HF dataset storage configuration |181| `tests/persistence.test.ts` | Local file persistence, debounced save |182| `tests/api-routes.test.ts` | API route integration tests (publish, auth status) |183