Felipe97/llama-cpp-compiled
01.2k
1# llama-ui2 3A modern, feature-rich web interface for llama-server built with SvelteKit. This UI provides an intuitive chat interface with advanced file handling, conversation management, and comprehensive model interaction capabilities.4 5Llama UI supports two server operation modes:6 7- **MODEL mode** - Single model operation (standard llama-server)8- **ROUTER mode** - Multi-model operation with dynamic model loading/unloading9 10---11 12## Table of Contents13 14- [Features](#features)15- [Getting Started](#getting-started)16- [Tech Stack](#tech-stack)17- [Build Pipeline](#build-pipeline)18- [Architecture](#architecture)19- [Data Flows](#data-flows)20- [Architectural Patterns](#architectural-patterns)21- [Testing](#testing)22 23---24 25## Features26 27### Chat Interface28 29- **Streaming responses** with real-time updates30- **Reasoning content** - Support for models with thinking/reasoning blocks31- **Dark/light theme** with system preference detection32- **Responsive design** for desktop and mobile33 34### File Attachments35 36- **Images** - JPEG, PNG, GIF, WebP, SVG (with PNG conversion)37- **Documents** - PDF (text extraction or image conversion for vision models)38- **Audio** - MP3, WAV for audio-capable models39- **Text files** - Source code, markdown, and other text formats40- **Drag-and-drop** and paste support with rich previews41 42### Conversation Management43 44- **Branching** - Branch messages conversations at any point by editing messages or regenerating responses, navigate between branches45- **Regeneration** - Regenerate responses with optional model switching (ROUTER mode)46- **Import/Export** - JSON format for backup and sharing47- **Search** - Find conversations by title or content48 49### Advanced Rendering50 51- **Syntax highlighting** - Code blocks with language detection52- **Math formulas** - KaTeX rendering for LaTeX expressions53- **Markdown** - Full GFM support with tables, lists, and more54 55### Multi-Model Support (ROUTER mode)56 57- **Model selector** with Loaded/Available groups58- **Automatic loading** - Models load on selection59- **Modality validation** - Prevents sending images to non-vision models60- **LRU unloading** - Server auto-manages model cache61 62### Keyboard Shortcuts63 64| Shortcut | Action |65| ------------------ | -------------------- |66| `Shift+Ctrl/Cmd+O` | New chat |67| `Shift+Ctrl/Cmd+E` | Edit conversation |68| `Shift+Ctrl/Cmd+D` | Delete conversation |69| `Ctrl/Cmd+K` | Search conversations |70| `Ctrl/Cmd+B` | Toggle sidebar |71 72### Developer Experience73 74- **Request tracking** - Monitor token generation with `/slots` endpoint75- **Storybook** - Component library with visual testing76- **Hot reload** - Instant updates during development77 78---79 80## Getting Started81 82### Prerequisites83 84- **Node.js** 18+ (20+ recommended)85- **npm** 9+86- **llama-server** running locally (for API access)87 88### 1. Install Dependencies89 90```bash91cd tools/ui92npm ci93```94 95### 2. Start llama-server96 97In a separate terminal, start the backend server:98 99```bash100# Single model (MODEL mode)101./llama-server -m model.gguf102 103# Multi-model (ROUTER mode)104./llama-server --models-dir /path/to/models105```106 107### 3. Start Development Servers108 109```bash110npm run dev111```112 113This starts:114 115- **Vite dev server** at `http://localhost:5173` - The main UI frontend app116- **Storybook** at `http://localhost:6006` - Component documentation117 118The Vite dev server proxies API requests to `SERVER_ORIGIN` (with fallback to default llama-server `8080` port):119 120```typescript121// vite.config.ts proxy configuration122proxy: {123 '/v1': SERVER_ORIGIN,124 '/props': SERVER_ORIGIN,125 '/models': SERVER_ORIGIN,126 '/tools': SERVER_ORIGIN,127 '/slots': SERVER_ORIGIN,128 '/cors-proxy': SERVER_ORIGIN129},130```131 132### Development Workflow133 1341. Open `http://localhost:5173` in your browser1352. Make changes to `.svelte`, `.ts`, or `.css` files1363. Changes hot-reload instantly1374. Use Storybook at `http://localhost:6006` for isolated component development138 139---140 141## Tech Stack142 143| Layer | Technology | Purpose |144| ----------------- | ------------------------------- | -------------------------------------------------------- |145| **Framework** | SvelteKit + Svelte 5 | Reactive UI with runes (`$state`, `$derived`, `$effect`) |146| **UI Components** | shadcn-svelte + bits-ui | Accessible, customizable component library |147| **Styling** | TailwindCSS 4 | Utility-first CSS with design tokens |148| **Database** | IndexedDB (Dexie) | Client-side storage for conversations and messages |149| **Build** | Vite | Fast bundling with static adapter |150| **Testing** | Playwright + Vitest + Storybook | E2E, unit, and visual testing |151| **Markdown** | remark + rehype | Markdown processing with KaTeX and syntax highlighting |152 153### Key Dependencies154 155```json156{157 "svelte": "^5.0.0",158 "bits-ui": "^2.8.11",159 "dexie": "^4.0.11",160 "pdfjs-dist": "^5.4.54",161 "highlight.js": "^11.11.1",162 "rehype-katex": "^7.0.1"163}164```165 166---167 168## Build Pipeline169 170### Development Build171 172```bash173npm run dev174```175 176Runs Vite in development mode with:177 178- Hot Module Replacement (HMR)179- Source maps180- Proxy to llama-server181 182### Production Build183 184```bash185npm run build186```187 188The build process:189 1901. **Vite Build** - Bundles all TypeScript, Svelte, and CSS1912. **Static Adapter** - Outputs to `../../build/tools/ui/dist` (llama-server's static file directory)1923. **Post-Build Script** - Cleans up intermediate files1934. **Custom Plugin** - Creates `index.html` with:194 - Inlined favicon as base64195 - GZIP compression (level 9)196 - Deterministic output (zeroed timestamps)197 198```text199tools/ui/ → build → build/tools/ui/dist/200├── src/ ├── index.html (served by llama-server)201├── static/ └── (favicon inlined)202└── ...203```204 205### SvelteKit Configuration206 207```javascript208// svelte.config.js209adapter: adapter({210 pages: '../../build/tools/ui/dist', // Output directory211 assets: '../../build/tools/ui/dist', // Static assets212 fallback: 'index.html', // SPA fallback213 strict: true214}),215output: {216 bundleStrategy: 'inline' // Single-file bundle217}218```219 220### Integration with llama-server221 222llama-ui is embedded directly into the llama-server binary:223 2241. `npm run build` outputs `index.html` to `build/tools/ui/dist/`2252. llama-server compiles this into the binary at build time2263. When accessing `/`, llama-server serves the bundled HTML227 228This results in a **single portable binary** with the full Llama UI included.229 230---231 232## Architecture233 234Llama UI follows a layered architecture with unidirectional data flow:235 236```text237Routes → Components → Hooks → Stores → Services → Storage/API238```239 240### High-Level Architecture241 242```mermaid243flowchart TB244 subgraph Routes["📍 Routes"]245 R1["/ (Welcome)"]246 R2["/chat/[id]"]247 R3["/mcp-servers"]248 R4["/search"]249 R5["/settings"]250 RL["+layout.svelte"]251 end252 253 subgraph Components["🧩 Components"]254 C_Screen["ChatScreen"]255 C_Form["ChatForm"]256 C_Messages["ChatMessages"]257 C_Sidebar["ChatSidebar"]258 C_Models["ModelsSelector"]259 C_Settings["ChatSettings"]260 C_Mcp["McpServers"]261 end262 263 subgraph Hooks["🔌 Hooks"]264 H1["use-chat-screen-active-model"]265 H2["use-processing-state"]266 H3["use-context-gauge"]267 H4["use-models-selector"]268 H5["use-tools-panel"]269 end270 271 subgraph Stores["🗄️ Stores"]272 S1["chatStore"]273 S2["conversationsStore"]274 S3["modelsStore"]275 S4["mcpStore"]276 S5["agenticStore"]277 S6["serverStore"]278 S7["settingsStore"]279 S8["toolsStore"]280 end281 282 subgraph Services["⚙️ Services"]283 SV1["ChatService"]284 SV2["ModelsService"]285 SV3["PropsService"]286 SV4["DatabaseService"]287 SV5["MCPService"]288 SV6["ToolsService"]289 SV7["SandboxService"]290 end291 292 subgraph Storage["💾 Storage"]293 ST1["IndexedDB"]294 ST2["LocalStorage"]295 end296 297 subgraph APIs["🌐 llama-server"]298 API1["/v1/chat/completions"]299 API2["/props"]300 API3["/models/*"]301 API4["/tools"]302 end303 304 R1 & R2 --> C_Screen305 RL --> C_Sidebar306 C_Screen --> C_Form & C_Messages & C_Settings307 C_Screen --> H1 & H2 & H3308 C_Models --> H4309 C_Mcp --> S4310 C_Screen --> S1 & S2 & S3311 C_Models --> S3312 H1 --> S3313 S1 --> SV1 & SV4314 S2 --> SV4315 S3 --> SV2 & SV3316 S4 --> SV5317 S5 --> SV1 & SV5 & SV6 & SV7318 SV4 --> ST1319 SV1 --> API1320 SV2 --> API3321 SV3 --> API2322 SV6 --> API4323```324 325### Layer Breakdown326 327#### Routes (`src/routes/`)328 329- **`/`** - Welcome screen, creates new conversation330- **`/chat/[id]`** - Active chat interface331- **`/mcp-servers`** - MCP server management332- **`/search`** - Conversation search333- **`/settings`** - Settings (optional `[[section]]`)334- **`+layout.svelte`** - Sidebar, navigation, global initialization335 336#### Components (`src/lib/components/`)337 338Components are organized in `app/` (application-specific) and `ui/` (shadcn-svelte primitives).339 340**Chat Components** (`app/chat/`):341 342| Component | Responsibility |343| ------------------ | --------------------------------------------------------------------------- |344| `ChatScreen/` | Main chat container, coordinates message list, input form, and attachments |345| `ChatForm/` | Message input textarea with file upload, paste handling, keyboard shortcuts |346| `ChatMessages/` | Message list with branch navigation, regenerate/continue/edit actions |347| `ChatAttachments/` | File attachment previews, drag-and-drop, PDF/image/audio handling |348| `ChatSettings/` | Parameter sliders (temperature, top-p, etc.) with server default sync |349| `ChatSidebar/` | Conversation list, search, import/export, navigation |350 351**Dialog Components** (`app/dialogs/`):352 353| Component | Responsibility |354| ------------------------------- | -------------------------------------------------------- |355| `DialogChatSettings` | Full-screen settings configuration |356| `DialogModelInformation` | Model details (context size, modalities, parallel slots) |357| `DialogChatAttachmentPreview` | Full preview for images, PDFs (text or page view), code |358| `DialogConfirmation` | Generic confirmation for destructive actions |359| `DialogConversationTitleUpdate` | Edit conversation title |360 361**Server/Model Components** (`app/server/`, `app/models/`):362 363| Component | Responsibility |364| ------------------- | --------------------------------------------------------- |365| `ServerErrorSplash` | Error display when server is unreachable |366| `ModelsSelector` | Model dropdown with Loaded/Available groups (ROUTER mode) |367 368**Shared UI Components** (`app/misc/`):369 370| Component | Responsibility |371| -------------------------------- | ---------------------------------------------------------------- |372| `MarkdownContent` | Markdown rendering with KaTeX, syntax highlighting, copy buttons |373| `SyntaxHighlightedCode` | Code blocks with language detection and highlighting |374| `ActionButton`, `ActionDropdown` | Reusable action buttons and menus |375| `BadgeModality`, `BadgeInfo` | Status and capability badges |376 377#### Hooks (`src/lib/hooks/`)378 379Hooks are the thin view-layer between components and stores: they own UI concerns (scroll, drag-and-drop, keyboard shortcuts, pickers, selection) and translate store state into view state.380 381| Hook | Responsibility |382| ------------------------------- | -------------------------------------------------------------- |383| `use-chat-screen-active-model` | Active model resolution + modality capability detection |384| `use-processing-state` | View over `chatStore.processing` for streaming progress/tokens |385| `use-context-gauge` | View over `contextStatsStore` for the context usage gauge |386| `use-models-selector` | Model selector dropdown state (loaded/available groups) |387| `use-tools-panel` | Tools panel state |388| `use-reasoning-menu` | Reasoning-effort menu state |389| `use-attachment-menu` | Attachment menu + modality flags |390| `use-draft-messages` | Per-chat draft message/files persistence |391| `use-chat-form-pickers` | Chat form pickers (commands, mentions) |392| `use-debounced-search` | Shared debounced async search for pickers |393| `use-picker-navigation` | Picker keyboard navigation |394| `use-chat-message-edit-context` | Message edit context (content + extras) |395| `use-chat-screen-drag-and-drop` | Drag-and-drop state machine |396| `use-chat-screen-file-upload` | File upload queue + capability validation |397| `use-chat-screen-scroll` | Scroll container binding + navigation guard |398| `use-auto-scroll` | Auto-scroll controller for streaming |399| `use-marquee-selection` | Shift+click / marquee range selection |400| `use-keyboard-shortcuts` | Global keyboard shortcuts |401| `use-settings-navigation` | Settings section navigation |402| `use-pwa` | PWA install/update + version mismatch detection |403 404#### Stores (`src/lib/stores/`)405 406Stores own reactive application state as Svelte 5 runes. Larger stores are split into directories and compose focused sub-stores behind a narrow host interface (see Architectural Patterns).407 408| Store | Responsibility |409| -------------------- | --------------------------------------------------------------------------------------------------------------- |410| `chatStore` | Chat lifecycle, streaming, abort control, error handling; composes `processing`, `activity`, `streams`, `flows` |411| `conversationsStore` | Conversation CRUD, message branching, navigation, import/export; composes `preferences` |412| `modelsStore` | Model list, selection, loading/unloading (ROUTER); composes `props`, `status` |413| `mcpStore` | MCP host role: multi-server lifecycle, tool routing; composes `health`, `resources` |414| `agenticStore` | Multi-turn agentic loop orchestration, tool execution; composes `gates` |415| `serverStore` | Server connection state, `/props`, role detection, modalities |416| `settingsStore` | User preferences, theme, parameter sync with server defaults |417| `toolsStore` | Tool registry: server + MCP tools, enabled set for the LLM |418| `permissionsStore` | Persisted tool permission grants |419| `contextStatsStore` | Context window usage for the active conversation |420| `draftMessagesStore` | Per-chat draft message/files |421| `deviceStore` | Browser environment signals (mobile, OS, theme) |422| `versionStore` | Build version information |423 424#### Services (`src/lib/services/`)425 426Services are a stateless protocol layer: static methods, pure I/O, no reactive state. Stores consume them for all API and storage access.427 428| Service | Responsibility |429| ----------------------------- | ------------------------------------------------------------------------- |430| `ChatService` | `/v1/chat/completions` streaming + SSE parsing, message format conversion |431| `ModelsService` | `/models`, `/models/load`, `/models/unload` |432| `PropsService` | `/props`, `/props?model=` |433| `DatabaseService` | IndexedDB operations via Dexie |434| `MCPService` | MCP protocol: transports, connect, list/execute tools, prompts, resources |435| `ToolsService` | Server tool list/execute/stream (`/tools`) |436| `SandboxService` | Browser JS execution in a sandboxed worker |437| `ParameterSyncService` | Syncs settings with server defaults |438| `ConversationTransferService` | Conversation import/export JSONL + ZIP format |439| `MigrationService` | Non-destructive localStorage/IndexedDB migrations |440| `RouterService` | Dynamic route URL construction |441 442---443 444## Data Flows445 446### MODEL Mode (Single Model)447 448```mermaid449sequenceDiagram450 participant User451 participant UI452 participant Stores453 participant DB as IndexedDB454 participant API as llama-server455 456 Note over User,API: Initialization457 UI->>Stores: initStores() (awaited by route loads)458 Stores->>Stores: run migrations459 Stores->>DB: load conversations (background)460 Stores->>API: GET /props461 API-->>Stores: server config462 Stores->>API: GET /v1/models463 API-->>Stores: single model (auto-selected)464 465 Note over User,API: Chat Flow466 User->>UI: send message467 Stores->>DB: save user message468 Stores->>API: POST /v1/chat/completions (stream)469 loop streaming470 API-->>Stores: SSE chunks471 Stores-->>UI: reactive update472 end473 Stores->>DB: save assistant message474```475 476### ROUTER Mode (Multi-Model)477 478```mermaid479sequenceDiagram480 participant User481 participant UI482 participant Stores483 participant API as llama-server484 485 Note over User,API: Initialization486 Stores->>API: GET /props487 API-->>Stores: {role: "router"}488 Stores->>API: GET /models489 API-->>Stores: models[] with status490 491 Note over User,API: Model Selection492 User->>UI: select model493 alt model not loaded494 Stores->>API: POST /models/load495 loop poll status496 Stores->>API: GET /models497 end498 Stores->>API: GET /props?model=X499 end500 Stores->>Stores: validate modalities501 502 Note over User,API: Chat Flow503 Stores->>API: POST /v1/chat/completions {model: X}504 loop streaming505 API-->>Stores: SSE chunks + model info506 end507```508 509---510 511## Architectural Patterns512 513### 1. Reactive State with Svelte 5 Runes514 515All stores use Svelte 5's fine-grained reactivity:516 517```typescript518// Store with reactive state519class ChatStore {520 #isLoading = $state(false);521 #currentResponse = $state('');522 523 // Derived values auto-update524 get isStreaming() {525 return $derived(this.#isLoading && this.#currentResponse.length > 0);526 }527}528 529// Exported reactive accessors530export const isLoading = () => chatStore.isLoading;531export const currentResponse = () => chatStore.currentResponse;532```533 534### 2. Unidirectional Data Flow535 536Data flows in one direction, making state predictable:537 538```mermaid539flowchart LR540 subgraph UI["UI Layer"]541 A[User Action] --> B[Component]542 end543 544 subgraph State["State Layer"]545 B --> C[Store Method]546 C --> D[State Update]547 end548 549 subgraph IO["I/O Layer"]550 C --> E[Service]551 E --> F[API / IndexedDB]552 F -.->|Response| D553 end554 555 D -->|Reactive| B556```557 558Components dispatch actions to stores, stores coordinate with services for I/O, and state updates reactively propagate back to the UI.559 560### 3. Per-Conversation State561 562Enables concurrent streaming across multiple conversations. Loading is tracked563per conversation by the activity ledger (`chatStore.activity`), while streaming564state and abort controllers live in per-conversation maps:565 566```typescript567class ChatStore {568 chatStreamingStates = new SvelteMap<string, { response: string; messageId: string }>();569 abortControllers = new SvelteMap<string, AbortController>();570}571```572 573### 4. Message Branching with Tree Structure574 575Conversations are stored as a tree, not a linear list:576 577```typescript578interface DatabaseMessage {579 id: string;580 parent: string | null; // Points to parent message581 children: string[]; // List of child message IDs582 // ...583}584 585interface DatabaseConversation {586 currentNode: string; // Currently viewed branch tip587 // ...588}589```590 591Navigation between branches updates `currentNode` without losing history.592 593### 5. Layered Service Architecture594 595Stores handle state; services handle I/O:596 597```text598┌─────────────────┐599│ Stores │ Business logic, state management600├─────────────────┤601│ Services │ API calls, database operations602├─────────────────┤603│ Storage/API │ IndexedDB, LocalStorage, HTTP604└─────────────────┘605```606 607### 6. Server Role Abstraction608 609Single codebase handles both MODEL and ROUTER modes:610 611```typescript612// serverStore.ts613get isRouterMode() {614 return this.role === ServerRole.ROUTER;615}616 617// Components conditionally render based on mode618{#if isRouterMode()}619 <ModelsSelector />620{/if}621```622 623### 7. Modality Validation624 625Prevents sending attachments to incompatible models. The626`use-chat-screen-active-model` hook derives the active model's capabilities627from `modelsStore.props`:628 629```typescript630// use-chat-screen-active-model hook631const hasVisionModality = $derived.by(() => modelsStore.props.modelSupportsVision(activeModelId));632const hasAudioModality = $derived.by(() => modelsStore.props.modelSupportsAudio(activeModelId));633```634 635### 8. Persistent Storage Strategy636 637Data is persisted across sessions using two storage mechanisms:638 639```mermaid640flowchart TB641 subgraph Browser["Browser Storage"]642 subgraph IDB["IndexedDB (Dexie)"]643 C[Conversations]644 M[Messages]645 end646 subgraph LS["LocalStorage"]647 S[Settings Config]648 O[User Overrides]649 T[Theme Preference]650 end651 end652 653 subgraph Stores["Svelte Stores"]654 CS[conversationsStore] --> C655 CS --> M656 SS[settingsStore] --> S657 SS --> O658 SS --> T659 end660```661 662- **IndexedDB**: Conversations and messages (large, structured data)663- **LocalStorage**: Settings, user parameter overrides, theme (small key-value data)664- **Memory only**: Server props, model list (fetched fresh on each session)665 666---667 668## Testing669 670### Test Types671 672| Type | Tool | Location | Command |673| ------------- | ------------------ | ---------------- | ------------------- |674| **Unit** | Vitest | `tests/unit/` | `npm run test:unit` |675| **UI/Visual** | Storybook + Vitest | `tests/stories/` | `npm run test:ui` |676| **E2E** | Playwright | `tests/e2e/` | `npm run test:e2e` |677| **Client** | Vitest | `tests/client/`. | `npm run test:unit` |678 679### Running Tests680 681```bash682# All tests683npm run test684 685# Individual test suites686npm run test:e2e # End-to-end (requires llama-server)687npm run test:client # Client-side unit tests688npm run test:server # Server-side unit tests689npm run test:ui # Storybook visual tests690```691 692### Storybook Development693 694```bash695npm run storybook # Start Storybook dev server on :6006696npm run build-storybook # Build static Storybook697```698 699### Linting and Formatting700 701```bash702npm run lint # Check code style703npm run format # Auto-format with Prettier704npm run check # TypeScript type checking705```706 707---708 709## Project Structure710 711```text712tools/ui/713├── src/714│ ├── lib/715│ │ ├── components/ # UI components (app/, ui/)716│ │ ├── hooks/ # Svelte hooks717│ │ ├── stores/ # State management718│ │ ├── services/ # API and database services719│ │ ├── types/ # TypeScript interfaces720│ │ └── utils/ # Utility functions721│ ├── routes/ # SvelteKit routes722│ └── styles/ # Global styles723├── static/ # Static assets724├── tests/ # Test files725└── .storybook/ # Storybook configuration726```727 728---729 730## Related Documentation731 732- [llama.cpp Server README](../server/README.md) - Full server documentation733- [Multimodal Documentation](../../docs/multimodal.md) - Image and audio support734- [Function Calling](../../docs/function-calling.md) - Tool use capabilities735