Modelpublic

flexitok/parity-aware-bpe-ratio

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes

Model Card

Parity-aware-bpe trained with the following ratios:

Language	FLORES-200	Ratio
Arabic	arb_Arab	1.6
Czech	ces_Latn	1.09
Chinese	cmn_Hani	0.94
Danish	dan_Latn	1.05
German	deu_Latn	1.19
Greek	ell_Grek	2.18
English	eng_Latn	1
Farsi	fas_Arab	1.09
French	fra_Latn	1.23
Hungarian	hun_Latn	1.16
Indonesian	ind_Latn	1.09
Italian	ita_Latn	1.18
Japanese	jpn_Jpan	1.28
Dutch	nld_Latn	1.11
Polish	pol_Latn	1.13
Portuguese	por_Latn	1.12
Russian	rus_Cyrl	2.01
Spanish	spa_Latn	1.21
Swedish	swe_Latn	1.05
Turkish	tur_Latn	1.12
Vietnamese	vie_Latn	1.4