Modelpublic

Compactbot/ldt-10m

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes1.7kdownloads

tokenizer_config.json9 linesDownload Raw Back to root

1{2  "backend": "tokenizers",3  "bos_token": "[[ctrl:endoftext]]",4  "clean_up_tokenization_spaces": false,5  "eos_token": "[[ctrl:endoftext]]",6  "model_max_length": 512,7  "pad_token": "[[ctrl:endoftext]]",8  "tokenizer_class": "TokenizersBackend"9}