Modelpublic

bluspark/custom-code-python-tokenizer

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes

special_tokens_map.json6 linesDownload Raw Back to root

1{2  "bos_token": "<|endoftext|>",3  "eos_token": "<|endoftext|>",4  "unk_token": "<|endoftext|>"5}6