Modelpublic

lmeninato/mt5-small-codesearchnet-python3

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

0likes12downloads

tokenizer_config.json12 linesDownload Raw Back to root

1{2  "additional_special_tokens": null,3  "clean_up_tokenization_spaces": true,4  "eos_token": "</s>",5  "extra_ids": 0,6  "model_max_length": 1000000000000000019884624838656,7  "pad_token": "<pad>",8  "sp_model_kwargs": {},9  "tokenizer_class": "T5Tokenizer",10  "unk_token": "<unk>"11}12