Modelpublic

flexitok/parity-aware-bpe-devset

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes

Model Card

Parity-Aware BPE Tokenizer (dev-set balanced)

Languages

Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev) per language:

Language	Training source	FLORES-200 code
English	`fw_edu`	`eng_Latn`
German	`deu_Latn`	`deu_Latn`
French	`fra_Latn`	`fra_Latn`
Arabic	`arb_Arab`	`arb_Arab`
Farsi	`fas_Arab`	`fas_Arab`
Czech	`ces_Latn`	`ces_Latn`
Chinese	`cmn_Hani`	`cmn_Hani`
Danish	`dan_Latn`	`dan_Latn`
Dutch	`nld_Latn`	`nld_Latn`
Greek	`ell_Grek`	`ell_Grek`
Hungarian	`hun_Latn`	`hun_Latn`
Indonesian	`ind_Latn`	`ind_Latn`
Italian	`ita_Latn`	`ita_Latn`
Japanese	`jpn_Jpan`	`jpn_Jpan`
Polish	`pol_Latn`	`pol_Latn`
Portuguese	`por_Latn`	`por_Latn`
Russian	`rus_Cyrl`	`rus_Cyrl`
Spanish	`spa_Latn`	`spa_Latn`
Swedish	`swe_Latn`	`swe_Latn`
Turkish	`tur_Latn`	`tur_Latn`
Vietnamese	`vie_Latn`	`vie_Latn`