toksuite/quick-tokenizer-accuracy

Update supertoken.py

Merge branch 'main' of https://huggingface.co/spaces/toksuite/quick-tokenizer-accuracy

Merged readme

Updated hf paths for toksuite models

Update README.md

- Added summary that could be directly copy pasted into sheets

- Save with prefix

Fixed issue with tokenmonster and tekken tokenizers

UI updates

Now accepts multiline!

Fixed Farsi copy-paste error, Coding [WIP]

- Removed header

Updated app for summary markdown tables

Fixed hf issue, fixed OOM

Fixed issue with job_id

Improved serving script

Killarney script

Fixed supertoken tokenizer loading

Updated script a bit

Added app

initial commit