Datasetpublic

german-tokenizer-benchmark/ud-hdt

UD German-HDT: Tokenized Sentences This datasets hosts a sentence-tokenized version of the Universal Dependencies German-HDT dataset. Creation The following script can be used to reproduce the creation of the dataset: import json from flair.datasets import UD_GERMAN_HDT corpus = UD_GERMAN_HDT() with open("./train.jsonl", "wt") as f_out: for sentence in corpus.train: current_example = { "text": sentence.to_tokenized_string() }… See the full description on the dataset page: https://huggingface.co/datasets/german-tokenizer-benchmark/ud-hdt.

sourceHugging Facecc-by-4.0updated 11mo agoView on Hugging Face

0likes9downloads

Dataset Card

UD German-HDT: Tokenized Sentences

This datasets hosts a sentence-tokenized version of the Universal Dependencies German-HDT dataset.

Creation

The following script can be used to reproduce the creation of the dataset:

python

import json

from flair.datasets import UD_GERMAN_HDT

corpus = UD_GERMAN_HDT()

with open("./train.jsonl", "wt") as f_out:
    for sentence in corpus.train:
        current_example = {
            "text": sentence.to_tokenized_string()
        }
        f_out.write(json.dumps(current_example) + "\n")

The extracted dataset has 153,035 sentences.