KBaba7/llama.cpp
0
1# GBNF Guide2 3GBNF (GGML BNF) is a format for defining [formal grammars](https://en.wikipedia.org/wiki/Formal_grammar) to constrain model outputs in `llama.cpp`. For example, you can use it to force the model to generate valid JSON, or speak only in emojis. GBNF grammars are supported in various ways in `examples/main` and `examples/server`.4 5## Background6 7[Backus-Naur Form (BNF)](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) is a notation for describing the syntax of formal languages like programming languages, file formats, and protocols. GBNF is an extension of BNF that primarily adds a few modern regex-like features.8 9## Basics10 11In GBNF, we define *production rules* that specify how a *non-terminal* (rule name) can be replaced with sequences of *terminals* (characters, specifically Unicode [code points](https://en.wikipedia.org/wiki/Code_point)) and other non-terminals. The basic format of a production rule is `nonterminal ::= sequence...`.12 13## Example14 15Before going deeper, let's look at some of the features demonstrated in `grammars/chess.gbnf`, a small chess notation grammar:16```17# `root` specifies the pattern for the overall output18root ::= (19 # it must start with the characters "1. " followed by a sequence20 # of characters that match the `move` rule, followed by a space, followed21 # by another move, and then a newline22 "1. " move " " move "\n"23 24 # it's followed by one or more subsequent moves, numbered with one or two digits25 ([1-9] [0-9]? ". " move " " move "\n")+26)27 28# `move` is an abstract representation, which can be a pawn, nonpawn, or castle.29# The `[+#]?` denotes the possibility of checking or mate signs after moves30move ::= (pawn | nonpawn | castle) [+#]?31 32pawn ::= ...33nonpawn ::= ...34castle ::= ...35```36 37## Non-Terminals and Terminals38 39Non-terminal symbols (rule names) stand for a pattern of terminals and other non-terminals. They are required to be a dashed lowercase word, like `move`, `castle`, or `check-mate`.40 41Terminals are actual characters ([code points](https://en.wikipedia.org/wiki/Code_point)). They can be specified as a sequence like `"1"` or `"O-O"` or as ranges like `[1-9]` or `[NBKQR]`.42 43## Characters and character ranges44 45Terminals support the full range of Unicode. Unicode characters can be specified directly in the grammar, for example `hiragana ::= [ใ-ใ]`, or with escapes: 8-bit (`\xXX`), 16-bit (`\uXXXX`) or 32-bit (`\UXXXXXXXX`).46 47Character ranges can be negated with `^`:48```49single-line ::= [^\n]+ "\n"50```51 52## Sequences and Alternatives53 54The order of symbols in a sequence matters. For example, in `"1. " move " " move "\n"`, the `"1. "` must come before the first `move`, etc.55 56Alternatives, denoted by `|`, give different sequences that are acceptable. For example, in `move ::= pawn | nonpawn | castle`, `move` can be a `pawn` move, a `nonpawn` move, or a `castle`.57 58Parentheses `()` can be used to group sequences, which allows for embedding alternatives in a larger rule or applying repetition and optional symbols (below) to a sequence.59 60## Repetition and Optional Symbols61 62- `*` after a symbol or sequence means that it can be repeated zero or more times (equivalent to `{0,}`).63- `+` denotes that the symbol or sequence should appear one or more times (equivalent to `{1,}`).64- `?` makes the preceding symbol or sequence optional (equivalent to `{0,1}`).65- `{m}` repeats the precedent symbol or sequence exactly `m` times66- `{m,}` repeats the precedent symbol or sequence at least `m` times67- `{m,n}` repeats the precedent symbol or sequence at between `m` and `n` times (included)68- `{0,n}` repeats the precedent symbol or sequence at most `n` times (included)69 70## Comments and newlines71 72Comments can be specified with `#`:73```74# defines optional whitespace75ws ::= [ \t\n]+76```77 78Newlines are allowed between rules and between symbols or sequences nested inside parentheses. Additionally, a newline after an alternate marker `|` will continue the current rule, even outside of parentheses.79 80## The root rule81 82In a full grammar, the `root` rule always defines the starting point of the grammar. In other words, it specifies what the entire output must match.83 84```85# a grammar for lists86root ::= ("- " item)+87item ::= [^\n]+ "\n"88```89 90## Next steps91 92This guide provides a brief overview. Check out the GBNF files in this directory (`grammars/`) for examples of full grammars. You can try them out with:93```94./llama-cli -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'95```96 97`llama.cpp` can also convert JSON schemas to grammars either ahead of time or at each request, see below.98 99## Troubleshooting100 101Grammars currently have performance gotchas (see https://github.com/ggerganov/llama.cpp/issues/4218).102 103### Efficient optional repetitions104 105A common pattern is to allow repetitions of a pattern `x` up to N times.106 107While semantically correct, the syntax `x? x? x?.... x?` (with N repetitions) may result in extremely slow sampling. Instead, you can write `x{0,N}` (or `(x (x (x ... (x)?...)?)?)?` w/ N-deep nesting in earlier llama.cpp versions).108 109## Using GBNF grammars110 111You can use GBNF grammars:112 113- In [llama-server](../examples/server)'s completion endpoints, passed as the `grammar` body field114- In [llama-cli](../examples/main), passed as the `--grammar` & `--grammar-file` flags115- With [llama-gbnf-validator](../examples/gbnf-validator) tool, to test them against strings.116 117## JSON Schemas โ GBNF118 119`llama.cpp` supports converting a subset of https://json-schema.org/ to GBNF grammars:120 121- In [llama-server](../examples/server):122 - For any completion endpoints, passed as the `json_schema` body field123 - For the `/chat/completions` endpoint, passed inside the `response_format` body field (e.g. `{"type", "json_object", "schema": {"items": {}}}` or `{ type: "json_schema", json_schema: {"schema": ...} }`)124- In [llama-cli](../examples/main), passed as the `--json` / `-j` flag125- To convert to a grammar ahead of time:126 - in CLI, with [examples/json_schema_to_grammar.py](../examples/json_schema_to_grammar.py)127 - in JavaScript with [json-schema-to-grammar.mjs](../examples/server/public_legacy/json-schema-to-grammar.mjs) (this is used by the [server](../examples/server)'s Web UI)128 129Take a look at [tests](../tests/test-json-schema-to-grammar.cpp) to see which features are likely supported (you'll also find usage examples in https://github.com/ggerganov/llama.cpp/pull/5978, https://github.com/ggerganov/llama.cpp/pull/6659 & https://github.com/ggerganov/llama.cpp/pull/6555).130 131```bash132llama-cli \133 -hfr bartowski/Phi-3-medium-128k-instruct-GGUF \134 -hff Phi-3-medium-128k-instruct-Q8_0.gguf \135 -j '{136 "type": "array",137 "items": {138 "type": "object",139 "properties": {140 "name": {141 "type": "string",142 "minLength": 1,143 "maxLength": 100144 },145 "age": {146 "type": "integer",147 "minimum": 0,148 "maximum": 150149 }150 },151 "required": ["name", "age"],152 "additionalProperties": false153 },154 "minItems": 10,155 "maxItems": 100156 }' \157 -p 'Generate a {name, age}[] JSON array with famous actors of all ages.'158```159 160<details>161 162<summary>Show grammar</summary>163 164You can convert any schema in command-line with:165 166```bash167examples/json_schema_to_grammar.py name-age-schema.json168```169 170```171char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})172item ::= "{" space item-name-kv "," space item-age-kv "}" space173item-age ::= ([0-9] | ([1-8] [0-9] | [9] [0-9]) | "1" ([0-4] [0-9] | [5] "0")) space174item-age-kv ::= "\"age\"" space ":" space item-age175item-name ::= "\"" char{1,100} "\"" space176item-name-kv ::= "\"name\"" space ":" space item-name177root ::= "[" space item ("," space item){9,99} "]" space178space ::= | " " | "\n" [ \t]{0,20}179```180 181</details>182 183Here is also a list of known limitations (contributions welcome):184 185- `additionalProperties` defaults to `false` (produces faster grammars + reduces hallucinations).186- `"additionalProperties": true` may produce keys that contain unescaped newlines.187- Unsupported features are skipped silently. It is currently advised to use the command-line Python converter (see above) to see any warnings, and to inspect the resulting grammar / test it w/ [llama-gbnf-validator](../examples/gbnf-validator/gbnf-validator.cpp).188- Can't mix `properties` w/ `anyOf` / `oneOf` in the same type (https://github.com/ggerganov/llama.cpp/issues/7703)189- [prefixItems](https://json-schema.org/draft/2020-12/json-schema-core#name-prefixitems) is broken (but [items](https://json-schema.org/draft/2020-12/json-schema-core#name-items) works)190- `minimum`, `exclusiveMinimum`, `maximum`, `exclusiveMaximum`: only supported for `"type": "integer"` for now, not `number`191- Nested `$ref`s are broken (https://github.com/ggerganov/llama.cpp/issues/8073)192- [pattern](https://json-schema.org/draft/2020-12/json-schema-validation#name-pattern)s must start with `^` and end with `$`193- Remote `$ref`s not supported in the C++ version (Python & JavaScript versions fetch https refs)194- `string` [formats](https://json-schema.org/draft/2020-12/json-schema-validation#name-defined-formats) lack `uri`, `email`195- No [`patternProperties`](https://json-schema.org/draft/2020-12/json-schema-core#name-patternproperties)196 197And a non-exhaustive list of other unsupported features that are unlikely to be implemented (hard and/or too slow to support w/ stateless grammars):198 199- [`uniqueItems`](https://json-schema.org/draft/2020-12/json-schema-validation#name-uniqueitems)200- [`contains`](https://json-schema.org/draft/2020-12/json-schema-core#name-contains) / `minContains`201- `$anchor` (cf. [dereferencing](https://json-schema.org/draft/2020-12/json-schema-core#name-dereferencing))202- [`not`](https://json-schema.org/draft/2020-12/json-schema-core#name-not)203- [Conditionals](https://json-schema.org/draft/2020-12/json-schema-core#name-keywords-for-applying-subsche) `if` / `then` / `else` / `dependentSchemas`204 205### A word about additionalProperties206 207> [!WARNING]208> The JSON schemas spec states `object`s accept [additional properties](https://json-schema.org/understanding-json-schema/reference/object#additionalproperties) by default.209> Since this is slow and seems prone to hallucinations, we default to no additional properties.210> You can set `"additionalProperties": true` in the the schema of any object to explicitly allow additional properties.211 212If you're using [Pydantic](https://pydantic.dev/) to generate schemas, you can enable additional properties with the `extra` config on each model class:213 214```python215# pip install pydantic216import json217from typing import Annotated, List218from pydantic import BaseModel, Extra, Field219class QAPair(BaseModel):220 class Config:221 extra = 'allow' # triggers additionalProperties: true in the JSON schema222 question: str223 concise_answer: str224 justification: str225 226class Summary(BaseModel):227 class Config:228 extra = 'allow'229 key_facts: List[Annotated[str, Field(pattern='- .{5,}')]]230 question_answers: List[Annotated[List[QAPair], Field(min_items=5)]]231 232print(json.dumps(Summary.model_json_schema(), indent=2))233```234 235<details>236<summary>Show JSON schema & grammar</summary>237 238```json239{240 "$defs": {241 "QAPair": {242 "additionalProperties": true,243 "properties": {244 "question": {245 "title": "Question",246 "type": "string"247 },248 "concise_answer": {249 "title": "Concise Answer",250 "type": "string"251 },252 "justification": {253 "title": "Justification",254 "type": "string"255 }256 },257 "required": [258 "question",259 "concise_answer",260 "justification"261 ],262 "title": "QAPair",263 "type": "object"264 }265 },266 "additionalProperties": true,267 "properties": {268 "key_facts": {269 "items": {270 "pattern": "^- .{5,}$",271 "type": "string"272 },273 "title": "Key Facts",274 "type": "array"275 },276 "question_answers": {277 "items": {278 "items": {279 "$ref": "#/$defs/QAPair"280 },281 "minItems": 5,282 "type": "array"283 },284 "title": "Question Answers",285 "type": "array"286 }287 },288 "required": [289 "key_facts",290 "question_answers"291 ],292 "title": "Summary",293 "type": "object"294}295```296 297```298QAPair ::= "{" space QAPair-question-kv "," space QAPair-concise-answer-kv "," space QAPair-justification-kv ( "," space ( QAPair-additional-kv ( "," space QAPair-additional-kv )* ) )? "}" space299QAPair-additional-k ::= ["] ( [c] ([o] ([n] ([c] ([i] ([s] ([e] ([_] ([a] ([n] ([s] ([w] ([e] ([r] char+ | [^"r] char*) | [^"e] char*) | [^"w] char*) | [^"s] char*) | [^"n] char*) | [^"a] char*) | [^"_] char*) | [^"e] char*) | [^"s] char*) | [^"i] char*) | [^"c] char*) | [^"n] char*) | [^"o] char*) | [j] ([u] ([s] ([t] ([i] ([f] ([i] ([c] ([a] ([t] ([i] ([o] ([n] char+ | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"a] char*) | [^"c] char*) | [^"i] char*) | [^"f] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"u] char*) | [q] ([u] ([e] ([s] ([t] ([i] ([o] ([n] char+ | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"e] char*) | [^"u] char*) | [^"cjq] char* )? ["] space300QAPair-additional-kv ::= QAPair-additional-k ":" space value301QAPair-concise-answer-kv ::= "\"concise_answer\"" space ":" space string302QAPair-justification-kv ::= "\"justification\"" space ":" space string303QAPair-question-kv ::= "\"question\"" space ":" space string304additional-k ::= ["] ( [k] ([e] ([y] ([_] ([f] ([a] ([c] ([t] ([s] char+ | [^"s] char*) | [^"t] char*) | [^"c] char*) | [^"a] char*) | [^"f] char*) | [^"_] char*) | [^"y] char*) | [^"e] char*) | [q] ([u] ([e] ([s] ([t] ([i] ([o] ([n] ([_] ([a] ([n] ([s] ([w] ([e] ([r] ([s] char+ | [^"s] char*) | [^"r] char*) | [^"e] char*) | [^"w] char*) | [^"s] char*) | [^"n] char*) | [^"a] char*) | [^"_] char*) | [^"n] char*) | [^"o] char*) | [^"i] char*) | [^"t] char*) | [^"s] char*) | [^"e] char*) | [^"u] char*) | [^"kq] char* )? ["] space305additional-kv ::= additional-k ":" space value306array ::= "[" space ( value ("," space value)* )? "]" space307boolean ::= ("true" | "false") space308char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})309decimal-part ::= [0-9]{1,16}310dot ::= [^\x0A\x0D]311integral-part ::= [0] | [1-9] [0-9]{0,15}312key-facts ::= "[" space (key-facts-item ("," space key-facts-item)*)? "]" space313key-facts-item ::= "\"" "- " key-facts-item-1{5,} "\"" space314key-facts-item-1 ::= dot315key-facts-kv ::= "\"key_facts\"" space ":" space key-facts316null ::= "null" space317number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space318object ::= "{" space ( string ":" space value ("," space string ":" space value)* )? "}" space319question-answers ::= "[" space (question-answers-item ("," space question-answers-item)*)? "]" space320question-answers-item ::= "[" space question-answers-item-item ("," space question-answers-item-item){4,} "]" space321question-answers-item-item ::= QAPair322question-answers-kv ::= "\"question_answers\"" space ":" space question-answers323root ::= "{" space key-facts-kv "," space question-answers-kv ( "," space ( additional-kv ( "," space additional-kv )* ) )? "}" space324space ::= | " " | "\n" [ \t]{0,20}325string ::= "\"" char* "\"" space326value ::= object | array | string | number | boolean | null327```328 329</details>330 331If you're using [Zod](https://zod.dev/), you can make your objects to explicitly allow extra properties w/ `nonstrict()` / `passthrough()` (or explicitly no extra props w/ `z.object(...).strict()` or `z.strictObject(...)`) but note that [zod-to-json-schema](https://github.com/StefanTerdell/zod-to-json-schema) currently always sets `"additionalProperties": false` anyway.332 333```js334import { z } from 'zod';335import { zodToJsonSchema } from 'zod-to-json-schema';336 337const Foo = z.object({338 age: z.number().positive(),339 email: z.string().email(),340}).strict();341 342console.log(zodToJsonSchema(Foo));343```344 345<details>346<summary>Show JSON schema & grammar</summary>347 348```json349{350 "type": "object",351 "properties": {352 "age": {353 "type": "number",354 "exclusiveMinimum": 0355 },356 "email": {357 "type": "string",358 "format": "email"359 }360 },361 "required": [362 "age",363 "email"364 ],365 "additionalProperties": false,366 "$schema": "http://json-schema.org/draft-07/schema#"367}368```369 370```371age-kv ::= "\"age\"" space ":" space number372char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})373decimal-part ::= [0-9]{1,16}374email-kv ::= "\"email\"" space ":" space string375integral-part ::= [0] | [1-9] [0-9]{0,15}376number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space377root ::= "{" space age-kv "," space email-kv "}" space378space ::= | " " | "\n" [ \t]{0,20}379string ::= "\"" char* "\"" space380```381 382</details>383