Brunobkr/llama.cpp_AlgMor24_github

1# llama-ui2 3A modern, feature-rich web interface for llama-server built with SvelteKit. This UI provides an intuitive chat interface with advanced file handling, conversation management, and comprehensive model interaction capabilities.4 5Llama UI supports two server operation modes:6 7- **MODEL mode** - Single model operation (standard llama-server)8- **ROUTER mode** - Multi-model operation with dynamic model loading/unloading9 10---11 12## Table of Contents13 14- [Features](#features)15- [Getting Started](#getting-started)16- [Tech Stack](#tech-stack)17- [Build Pipeline](#build-pipeline)18- [Architecture](#architecture)19- [Data Flows](#data-flows)20- [Architectural Patterns](#architectural-patterns)21- [Testing](#testing)22 23---24 25## Features26 27### Chat Interface28 29- **Streaming responses** with real-time updates30- **Reasoning content** - Support for models with thinking/reasoning blocks31- **Dark/light theme** with system preference detection32- **Responsive design** for desktop and mobile33 34### File Attachments35 36- **Images** - JPEG, PNG, GIF, WebP, SVG (with PNG conversion)37- **Documents** - PDF (text extraction or image conversion for vision models)38- **Audio** - MP3, WAV for audio-capable models39- **Text files** - Source code, markdown, and other text formats40- **Drag-and-drop** and paste support with rich previews41 42### Conversation Management43 44- **Branching** - Branch messages conversations at any point by editing messages or regenerating responses, navigate between branches45- **Regeneration** - Regenerate responses with optional model switching (ROUTER mode)46- **Import/Export** - JSON format for backup and sharing47- **Search** - Find conversations by title or content48 49### Advanced Rendering50 51- **Syntax highlighting** - Code blocks with language detection52- **Math formulas** - KaTeX rendering for LaTeX expressions53- **Markdown** - Full GFM support with tables, lists, and more54 55### Multi-Model Support (ROUTER mode)56 57- **Model selector** with Loaded/Available groups58- **Automatic loading** - Models load on selection59- **Modality validation** - Prevents sending images to non-vision models60- **LRU unloading** - Server auto-manages model cache61 62### Keyboard Shortcuts63 64| Shortcut           | Action               |65| ------------------ | -------------------- |66| `Shift+Ctrl/Cmd+O` | New chat             |67| `Shift+Ctrl/Cmd+E` | Edit conversation    |68| `Shift+Ctrl/Cmd+D` | Delete conversation  |69| `Ctrl/Cmd+K`       | Search conversations |70| `Ctrl/Cmd+B`       | Toggle sidebar       |71 72### Developer Experience73 74- **Request tracking** - Monitor token generation with `/slots` endpoint75- **Storybook** - Component library with visual testing76- **Hot reload** - Instant updates during development77 78---79 80## Getting Started81 82### Prerequisites83 84- **Node.js** 18+ (20+ recommended)85- **npm** 9+86- **llama-server** running locally (for API access)87 88### 1. Install Dependencies89 90```bash91cd tools/ui92npm ci93```94 95### 2. Start llama-server96 97In a separate terminal, start the backend server:98 99```bash100# Single model (MODEL mode)101./llama-server -m model.gguf102 103# Multi-model (ROUTER mode)104./llama-server --models-dir /path/to/models105```106 107### 3. Start Development Servers108 109```bash110npm run dev111```112 113This starts:114 115- **Vite dev server** at `http://localhost:5173` - The main UI frontend app116- **Storybook** at `http://localhost:6006` - Component documentation117 118The Vite dev server proxies API requests to `SERVER_ORIGIN` (with fallback to default llama-server `8080` port):119 120```typescript121// vite.config.ts proxy configuration122proxy: {123	'/v1': SERVER_ORIGIN,124	'/props': SERVER_ORIGIN,125	'/models': SERVER_ORIGIN,126	'/tools': SERVER_ORIGIN,127	'/slots': SERVER_ORIGIN,128	'/cors-proxy': SERVER_ORIGIN129},130```131 132### Development Workflow133 1341. Open `http://localhost:5173` in your browser1352. Make changes to `.svelte`, `.ts`, or `.css` files1363. Changes hot-reload instantly1374. Use Storybook at `http://localhost:6006` for isolated component development138 139---140 141## Tech Stack142 143| Layer             | Technology                      | Purpose                                                  |144| ----------------- | ------------------------------- | -------------------------------------------------------- |145| **Framework**     | SvelteKit + Svelte 5            | Reactive UI with runes (`$state`, `$derived`, `$effect`) |146| **UI Components** | shadcn-svelte + bits-ui         | Accessible, customizable component library               |147| **Styling**       | TailwindCSS 4                   | Utility-first CSS with design tokens                     |148| **Database**      | IndexedDB (Dexie)               | Client-side storage for conversations and messages       |149| **Build**         | Vite                            | Fast bundling with static adapter                        |150| **Testing**       | Playwright + Vitest + Storybook | E2E, unit, and visual testing                            |151| **Markdown**      | remark + rehype                 | Markdown processing with KaTeX and syntax highlighting   |152 153### Key Dependencies154 155```json156{157	"svelte": "^5.0.0",158	"bits-ui": "^2.8.11",159	"dexie": "^4.0.11",160	"pdfjs-dist": "^5.4.54",161	"highlight.js": "^11.11.1",162	"rehype-katex": "^7.0.1"163}164```165 166---167 168## Build Pipeline169 170### Development Build171 172```bash173npm run dev174```175 176Runs Vite in development mode with:177 178- Hot Module Replacement (HMR)179- Source maps180- Proxy to llama-server181 182### Production Build183 184```bash185npm run build186```187 188The build process:189 1901. **Vite Build** - Bundles all TypeScript, Svelte, and CSS1912. **Static Adapter** - Outputs to `../../build/tools/ui/dist` (llama-server's static file directory)1923. **Post-Build Script** - Cleans up intermediate files1934. **Custom Plugin** - Creates `index.html` with:194   - Inlined favicon as base64195   - GZIP compression (level 9)196   - Deterministic output (zeroed timestamps)197 198```text199tools/ui/        →  build  →  build/tools/ui/dist/200├── src/                                 ├── index.html  (served by llama-server)201├── static/                              └── (favicon inlined)202└── ...203```204 205### SvelteKit Configuration206 207```javascript208// svelte.config.js209adapter: adapter({210  pages: '../../build/tools/ui/dist',      // Output directory211  assets: '../../build/tools/ui/dist',     // Static assets212  fallback: 'index.html',  // SPA fallback213  strict: true214}),215output: {216  bundleStrategy: 'inline' // Single-file bundle217}218```219 220### Integration with llama-server221 222llama-ui is embedded directly into the llama-server binary:223 2241. `npm run build` outputs `index.html` to `build/tools/ui/dist/`2252. llama-server compiles this into the binary at build time2263. When accessing `/`, llama-server serves the bundled HTML227 228This results in a **single portable binary** with the full Llama UI included.229 230---231 232## Architecture233 234Llama UI follows a layered architecture with unidirectional data flow:235 236```text237Routes → Components → Hooks → Stores → Services → Storage/API238```239 240### High-Level Architecture241 242See: [`docs/architecture/high-level-architecture-simplified.md`](docs/architecture/high-level-architecture-simplified.md)243 244```mermaid245flowchart TB246    subgraph Routes["📍 Routes"]247        R1["/ (Welcome)"]248        R2["/chat/[id]"]249        RL["+layout.svelte"]250    end251 252    subgraph Components["🧩 Components"]253        C_Sidebar["ChatSidebar"]254        C_Screen["ChatScreen"]255        C_Form["ChatForm"]256        C_Messages["ChatMessages"]257        C_ModelsSelector["ModelsSelector"]258        C_Settings["ChatSettings"]259    end260 261    subgraph Stores["🗄️ Stores"]262        S1["chatStore"]263        S2["conversationsStore"]264        S3["modelsStore"]265        S4["serverStore"]266        S5["settingsStore"]267    end268 269    subgraph Services["⚙️ Services"]270        SV1["ChatService"]271        SV2["ModelsService"]272        SV3["PropsService"]273        SV4["DatabaseService"]274    end275 276    subgraph Storage["💾 Storage"]277        ST1["IndexedDB"]278        ST2["LocalStorage"]279    end280 281    subgraph APIs["🌐 llama-server"]282        API1["/v1/chat/completions"]283        API2["/props"]284        API3["/models/*"]285    end286 287    R1 & R2 --> C_Screen288    RL --> C_Sidebar289    C_Screen --> C_Form & C_Messages & C_Settings290    C_Screen --> S1 & S2291    C_ModelsSelector --> S3 & S4292    S1 --> SV1 & SV4293    S3 --> SV2 & SV3294    SV4 --> ST1295    SV1 --> API1296    SV2 --> API3297    SV3 --> API2298```299 300### Layer Breakdown301 302#### Routes (`src/routes/`)303 304- **`/`** - Welcome screen, creates new conversation305- **`/chat/[id]`** - Active chat interface306- **`+layout.svelte`** - Sidebar, navigation, global initialization307 308#### Components (`src/lib/components/`)309 310Components are organized in `app/` (application-specific) and `ui/` (shadcn-svelte primitives).311 312**Chat Components** (`app/chat/`):313 314| Component          | Responsibility                                                              |315| ------------------ | --------------------------------------------------------------------------- |316| `ChatScreen/`      | Main chat container, coordinates message list, input form, and attachments  |317| `ChatForm/`        | Message input textarea with file upload, paste handling, keyboard shortcuts |318| `ChatMessages/`    | Message list with branch navigation, regenerate/continue/edit actions       |319| `ChatAttachments/` | File attachment previews, drag-and-drop, PDF/image/audio handling           |320| `ChatSettings/`    | Parameter sliders (temperature, top-p, etc.) with server default sync       |321| `ChatSidebar/`     | Conversation list, search, import/export, navigation                        |322 323**Dialog Components** (`app/dialogs/`):324 325| Component                       | Responsibility                                           |326| ------------------------------- | -------------------------------------------------------- |327| `DialogChatSettings`            | Full-screen settings configuration                       |328| `DialogModelInformation`        | Model details (context size, modalities, parallel slots) |329| `DialogChatAttachmentPreview`   | Full preview for images, PDFs (text or page view), code  |330| `DialogConfirmation`            | Generic confirmation for destructive actions             |331| `DialogConversationTitleUpdate` | Edit conversation title                                  |332 333**Server/Model Components** (`app/server/`, `app/models/`):334 335| Component           | Responsibility                                            |336| ------------------- | --------------------------------------------------------- |337| `ServerErrorSplash` | Error display when server is unreachable                  |338| `ModelsSelector`    | Model dropdown with Loaded/Available groups (ROUTER mode) |339 340**Shared UI Components** (`app/misc/`):341 342| Component                        | Responsibility                                                   |343| -------------------------------- | ---------------------------------------------------------------- |344| `MarkdownContent`                | Markdown rendering with KaTeX, syntax highlighting, copy buttons |345| `SyntaxHighlightedCode`          | Code blocks with language detection and highlighting             |346| `ActionButton`, `ActionDropdown` | Reusable action buttons and menus                                |347| `BadgeModality`, `BadgeInfo`     | Status and capability badges                                     |348 349#### Hooks (`src/lib/hooks/`)350 351- **`useModelChangeValidation`** - Validates model switch against conversation modalities352- **`useProcessingState`** - Tracks streaming progress and token generation353 354#### Stores (`src/lib/stores/`)355 356| Store                | Responsibility                                            |357| -------------------- | --------------------------------------------------------- |358| `chatStore`          | Message sending, streaming, abort control, error handling |359| `conversationsStore` | CRUD for conversations, message branching, navigation     |360| `modelsStore`        | Model list, selection, loading/unloading (ROUTER)         |361| `serverStore`        | Server properties, role detection, modalities             |362| `settingsStore`      | User preferences, parameter sync with server defaults     |363 364#### Services (`src/lib/services/`)365 366| Service                | Responsibility                                  |367| ---------------------- | ----------------------------------------------- |368| `ChatService`          | API calls to`/v1/chat/completions`, SSE parsing |369| `ModelsService`        | `/models`, `/models/load`, `/models/unload`     |370| `PropsService`         | `/props`, `/props?model=`                       |371| `DatabaseService`      | IndexedDB operations via Dexie                  |372| `ParameterSyncService` | Syncs settings with server defaults             |373 374---375 376## Data Flows377 378### MODEL Mode (Single Model)379 380See: [`docs/flows/data-flow-simplified-model-mode.md`](docs/flows/data-flow-simplified-model-mode.md)381 382```mermaid383sequenceDiagram384    participant User385    participant UI386    participant Stores387    participant DB as IndexedDB388    participant API as llama-server389 390    Note over User,API: Initialization391    UI->>Stores: initialize()392    Stores->>DB: load conversations393    Stores->>API: GET /props394    API-->>Stores: server config395    Stores->>API: GET /v1/models396    API-->>Stores: single model (auto-selected)397 398    Note over User,API: Chat Flow399    User->>UI: send message400    Stores->>DB: save user message401    Stores->>API: POST /v1/chat/completions (stream)402    loop streaming403        API-->>Stores: SSE chunks404        Stores-->>UI: reactive update405    end406    Stores->>DB: save assistant message407```408 409### ROUTER Mode (Multi-Model)410 411See: [`docs/flows/data-flow-simplified-router-mode.md`](docs/flows/data-flow-simplified-router-mode.md)412 413```mermaid414sequenceDiagram415    participant User416    participant UI417    participant Stores418    participant API as llama-server419 420    Note over User,API: Initialization421    Stores->>API: GET /props422    API-->>Stores: {role: "router"}423    Stores->>API: GET /models424    API-->>Stores: models[] with status425 426    Note over User,API: Model Selection427    User->>UI: select model428    alt model not loaded429        Stores->>API: POST /models/load430        loop poll status431            Stores->>API: GET /models432        end433        Stores->>API: GET /props?model=X434    end435    Stores->>Stores: validate modalities436 437    Note over User,API: Chat Flow438    Stores->>API: POST /v1/chat/completions {model: X}439    loop streaming440        API-->>Stores: SSE chunks + model info441    end442```443 444### Detailed Flow Diagrams445 446| Flow          | Description                                | File                                                        |447| ------------- | ------------------------------------------ | ----------------------------------------------------------- |448| Chat          | Message lifecycle, streaming, regeneration | [`chat-flow.md`](docs/flows/chat-flow.md)                   |449| Models        | Loading, unloading, modality caching       | [`models-flow.md`](docs/flows/models-flow.md)               |450| Server        | Props fetching, role detection             | [`server-flow.md`](docs/flows/server-flow.md)               |451| Conversations | CRUD, branching, import/export             | [`conversations-flow.md`](docs/flows/conversations-flow.md) |452| Database      | IndexedDB schema, operations               | [`database-flow.md`](docs/flows/database-flow.md)           |453| Settings      | Parameter sync, user overrides             | [`settings-flow.md`](docs/flows/settings-flow.md)           |454 455---456 457## Architectural Patterns458 459### 1. Reactive State with Svelte 5 Runes460 461All stores use Svelte 5's fine-grained reactivity:462 463```typescript464// Store with reactive state465class ChatStore {466	#isLoading = $state(false);467	#currentResponse = $state('');468 469	// Derived values auto-update470	get isStreaming() {471		return $derived(this.#isLoading && this.#currentResponse.length > 0);472	}473}474 475// Exported reactive accessors476export const isLoading = () => chatStore.isLoading;477export const currentResponse = () => chatStore.currentResponse;478```479 480### 2. Unidirectional Data Flow481 482Data flows in one direction, making state predictable:483 484```mermaid485flowchart LR486    subgraph UI["UI Layer"]487        A[User Action] --> B[Component]488    end489 490    subgraph State["State Layer"]491        B --> C[Store Method]492        C --> D[State Update]493    end494 495    subgraph IO["I/O Layer"]496        C --> E[Service]497        E --> F[API / IndexedDB]498        F -.->|Response| D499    end500 501    D -->|Reactive| B502```503 504Components dispatch actions to stores, stores coordinate with services for I/O, and state updates reactively propagate back to the UI.505 506### 3. Per-Conversation State507 508Enables concurrent streaming across multiple conversations:509 510```typescript511class ChatStore {512	chatLoadingStates = new Map<string, boolean>();513	chatStreamingStates = new Map<string, { response: string; messageId: string }>();514	abortControllers = new Map<string, AbortController>();515}516```517 518### 4. Message Branching with Tree Structure519 520Conversations are stored as a tree, not a linear list:521 522```typescript523interface DatabaseMessage {524	id: string;525	parent: string | null; // Points to parent message526	children: string[]; // List of child message IDs527	// ...528}529 530interface DatabaseConversation {531	currentNode: string; // Currently viewed branch tip532	// ...533}534```535 536Navigation between branches updates `currentNode` without losing history.537 538### 5. Layered Service Architecture539 540Stores handle state; services handle I/O:541 542```text543┌─────────────────┐544│     Stores      │  Business logic, state management545├─────────────────┤546│    Services     │  API calls, database operations547├─────────────────┤548│   Storage/API   │  IndexedDB, LocalStorage, HTTP549└─────────────────┘550```551 552### 6. Server Role Abstraction553 554Single codebase handles both MODEL and ROUTER modes:555 556```typescript557// serverStore.ts558get isRouterMode() {559  return this.role === ServerRole.ROUTER;560}561 562// Components conditionally render based on mode563{#if isRouterMode()}564  <ModelsSelector />565{/if}566```567 568### 7. Modality Validation569 570Prevents sending attachments to incompatible models:571 572```typescript573// useModelChangeValidation hook574const validate = (modelId: string) => {575	const modelModalities = modelsStore.getModelModalities(modelId);576	const conversationModalities = conversationsStore.usedModalities;577 578	// Check if model supports all used modalities579	if (conversationModalities.hasImages && !modelModalities.vision) {580		return { valid: false, reason: 'Model does not support images' };581	}582	// ...583};584```585 586### 8. Persistent Storage Strategy587 588Data is persisted across sessions using two storage mechanisms:589 590```mermaid591flowchart TB592    subgraph Browser["Browser Storage"]593        subgraph IDB["IndexedDB (Dexie)"]594            C[Conversations]595            M[Messages]596        end597        subgraph LS["LocalStorage"]598            S[Settings Config]599            O[User Overrides]600            T[Theme Preference]601        end602    end603 604    subgraph Stores["Svelte Stores"]605        CS[conversationsStore] --> C606        CS --> M607        SS[settingsStore] --> S608        SS --> O609        SS --> T610    end611```612 613- **IndexedDB**: Conversations and messages (large, structured data)614- **LocalStorage**: Settings, user parameter overrides, theme (small key-value data)615- **Memory only**: Server props, model list (fetched fresh on each session)616 617---618 619## Testing620 621### Test Types622 623| Type          | Tool               | Location         | Command             |624| ------------- | ------------------ | ---------------- | ------------------- |625| **Unit**      | Vitest             | `tests/unit/`    | `npm run test:unit` |626| **UI/Visual** | Storybook + Vitest | `tests/stories/` | `npm run test:ui`   |627| **E2E**       | Playwright         | `tests/e2e/`     | `npm run test:e2e`  |628| **Client**    | Vitest             | `tests/client/`. | `npm run test:unit` |629 630### Running Tests631 632```bash633# All tests634npm run test635 636# Individual test suites637npm run test:e2e      # End-to-end (requires llama-server)638npm run test:client   # Client-side unit tests639npm run test:server   # Server-side unit tests640npm run test:ui       # Storybook visual tests641```642 643### Storybook Development644 645```bash646npm run storybook     # Start Storybook dev server on :6006647npm run build-storybook  # Build static Storybook648```649 650### Linting and Formatting651 652```bash653npm run lint          # Check code style654npm run format        # Auto-format with Prettier655npm run check         # TypeScript type checking656```657 658---659 660## Project Structure661 662```text663tools/ui/664├── src/665│   ├── lib/666│   │   ├── components/   # UI components (app/, ui/)667│   │   ├── hooks/        # Svelte hooks668│   │   ├── stores/       # State management669│   │   ├── services/     # API and database services670│   │   ├── types/        # TypeScript interfaces671│   │   └── utils/        # Utility functions672│   ├── routes/           # SvelteKit routes673│   └── styles/           # Global styles674├── static/               # Static assets675├── tests/                # Test files676├── docs/                 # Architecture diagrams677│   ├── architecture/     # High-level architecture678│   └── flows/            # Feature-specific flows679└── .storybook/           # Storybook configuration680```681 682---683 684## Related Documentation685 686- [llama.cpp Server README](../server/README.md) - Full server documentation687- [Multimodal Documentation](../../docs/multimodal.md) - Image and audio support688- [Function Calling](../../docs/function-calling.md) - Tool use capabilities689