Modelpublic

facebook-llama/custom_code

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes22downloads

tokenizer_config.json19 linesDownload Raw Back to root

1{2  "name_or_path": "THUDM/glm-large-chinese",3  "eos_token": "<|endoftext|>",4  "pad_token": "<|endoftext|>",5  "cls_token": "[CLS]",6  "mask_token": "[MASK]",7  "unk_token": "[UNK]",8  "additional_special_tokens": ["<|startofpiece|>", "<|endofpiece|>", "[gMASK]", "[sMASK]"],9  "add_prefix_space": false,10  "tokenizer_class": "GLMChineseTokenizer",11  "use_fast": false,12  "auto_map": {13    "AutoTokenizer": [14      "tokenization_glm.GLMChineseTokenizer",15      null16      ]17  }18}19