Team Ai
Datasetpublic

procesaur/sr-tokenizer-test

Sr Tokenizer test This dataset provides a large Serbian text corpus designed for training and evaluating of tokenizers for Serbian language models. It combines multiple sources of Serbian text in both Cyrillic and Latin scripts, unified into a consistent JSONL format with id and text fields. Dataset Structure Metadata has been stripped; Each record is a JSON object with: id: unique identifier text: raw Serbian text Source coprora Znanje(sr)… See the full description on the dataset page: https://huggingface.co/datasets/procesaur/sr-tokenizer-test.

sourceHugging Facecc-by-nc-sa-4.0updated 5mo agoView on Hugging Face
0likes76downloads
Dataset Card

<!--img src="cover.png" class="cover"-->

Sr Tokenizer test

This dataset provides a large Serbian text corpus designed for training and evaluating of tokenizers for Serbian language models. It combines multiple sources of Serbian text in both Cyrillic and Latin scripts, unified into a consistent JSONL format with id and text fields.

Dataset Structure

Metadata has been stripped; Each record is a JSON object with:

  • —id: unique identifier
  • —text: raw Serbian text

Source coprora

total 11.3 GB with 1.3 billion words

Splits:
  • —train.jsonl (75%)
  • —test.jsonl (25%)
python
from datasets import load_dataset
dataset = load_dataset("procesaur/sr-tokenizer-test", split="train")

<div class="inline-flex flex-col" style="line-height: 1.5;padding-right:50px"> <div style="text-align: center; margin-top: 3px; font-size: 16px; font-weight: 800">Editor</div> <a href="https://huggingface.co/procesaur"> <div class="flex"> <div style="display:DISPLAY_1; margin-left: auto; margin-right: auto; width: 92px; height:92px; border-radius: 50%; background-size: cover; background-image: url(&#39;https://cdn-uploads.huggingface.co/production/uploads/1673534533167-63bc254fb8c61b8aa496a39b.jpeg?w=200&h=200&f=face&#39;)"> </div> </div> </a> <div style="text-align: center; font-size: 16px; font-weight: 800">Mihailo Škorić</div> <div> <a href="https://huggingface.co/procesaur"> <div style="text-align: center; font-size: 14px;">@procesaur</div> </a> </div> </div> </div>

Citation:

bibtex
soon

<!--table style="width:100%;height:100%"> ISLRN: 476-040-938-571-9 <tr style="width:100%;height:100%"> <td width=50%> <p>Istraživanje je sprovedeno uz podršku Fonda za nauku Republike Srbije, #7276, Text Embeddings – Serbian Language Applications – TESLA.</p> </td> <td> <p>This research was supported by the Science Fund of the Republic of Serbia, #7276, Text Embeddings - Serbian Language Applications - TESLA.</p> </td> </tr> </table-->

<div id="zastava"> <div class="grb"> <img src="https://www.ai.gov.rs/img/logo_60x120-2.png" style="position:relative; left:30px; z-index:10; height:85px"> </div> <table width=100% style="border:0px"> <tr style="background-color:#C6363C;width:100%;border:0px;height:30px"><td style="width:100vw"></td></tr> <tr style="background-color:#0C4076;width:100%;border:0px;height:30px"><td></td></tr> <tr style="background-color:#ffffff;width:100%;border:0px;height:30px"><td></td></tr> </table> </div>

<style> .ffeat: { color:red }

.cover { width: 100%; margin-bottom: 5pt }

.highlight-container, .highlight { position: relative; text-decoration:none }

.highlight-container { display: inline-block;

}

.highlight{ color:white; text-transform:uppercase; font-size: 16pt; }

.highlight-container{ padding:5px 10px }

.highlight-container:before { content: " "; display: block; height: 100%; width: 100%; margin-left: 0px; margin-right: 0px; position: absolute; background: #e80909; transform: rotate(2deg); top: -1px; left: -1px; border-radius: 20% 25% 20% 24%; padding: 10px 18px 18px 10px; }

div.grb, #zastava>table { position:absolute; top:0px; left: 0px; margin:0px }

div.grb>img, #zastava>table{ margin:0px }

#zastava { position: relative; margin-bottom:120px }

p { font-size:14pt } </style>