Datasetpublic

turkish-nlp-suite/Treebank-Benchmarking

Turkish Treebank Benchmarking This is the repo for Turkish treebank benchmarking, namely evaluating Tranformer models on POS-Dep-Morph task. For the data, we used two treebank, IMST and BOUN. We converted conllu format to json lines for being compatible to HF dataset formats. Here are treebank sizes at a glance: Dataset train lines dev lines test lines BOUN 7803 979 979 IMST 3435 1100 1100 A typical instance from the dataset looks like: { "id": "ins_1267"… See the full description on the dataset page: https://huggingface.co/datasets/turkish-nlp-suite/Treebank-Benchmarking.

sourceHugging Facecc-by-sa-4.0updated 8mo agoView on Hugging Face

0likes34downloads

Dataset Card

Turkish Treebank Benchmarking

This is the repo for Turkish treebank benchmarking, namely evaluating Tranformer models on POS-Dep-Morph task. For the data, we used two treebank, IMST and BOUN. We converted conllu format to json lines for being compatible to HF dataset formats.

Here are treebank sizes at a glance:

Dataset	train lines	dev lines	test lines
BOUN	7803	979	979
IMST	3435	1100	1100

A typical instance from the dataset looks like:

{
  "id": "ins_1267",
  "tokens": [
    "Rüzgâr",
    "yine",
    "güçlü",
    "esiyor",
    "du",
    "."
  ],
  "upos": [
    "NOUN",
    "ADV",
    "ADV",
    "VERB",
    "AUX",
    "PUNCT"
  ],
  "heads": [
    4,
    4,
    4,
    0,
    4,
    4
  ],
  "rels": [
    "nsubj",
    "advmod",
    "advmod",
    "root",
    "cop",
    "punct"
  ],
  "feats": [
    "Case=Nom|Number=Sing|Person=3",
    "_",
    "_",
    "Aspect=Imp|Polarity=Pos|VerbForm=Part",
    "Aspect=Perf|Evident=Fh|Number=Sing|Person=3|Tense=Past",
    "_"
  ],
  "text": "Rüzgâr yine güçlü esiyor du .",
  "feats_dict_json": [
    "{\"Case\":\"Nom\",\"Number\":\"Sing\",\"Person\":\"3\"}",
    "{}",
    "{}",
    "{\"Aspect\":\"Imp\",\"Polarity\":\"Pos\",\"VerbForm\":\"Part\"}",
    "{\"Aspect\":\"Perf\",\"Evident\":\"Fh\",\"Number\":\"Sing\",\"Person\":\"3\",\"Tense\":\"Past\"}",
    "{}"
  ]
}

Benchmarking

Benchmarking is done by scripts on accompanying Github repo. Please proceed to this repo for running the experiments. Here are the benchmarking results for BERTurk with our scripts:

Metric	BOUN	IMST
pos_acc	0.9263	0.9377
uas	0.8151	0.7680
las	0.7459	0.6960
morphAbbracc	0.4657	0.6705
morphAspectacc	0.1141	0.1152
morphCaseacc	0.1196	0.0586
morphEchoacc	0.4261	0.4875
morphEvidentacc	0.3072	0.3953
morphMoodacc	0.0654	0.0651
morphNumTypeacc	0.2694	0.2991
morphNumberacc	0.3986	0.4782
morphNumber[psor]acc	0.4348	0.2333
morphPersonacc	0.4021	0.4726
morphPerson[psor]acc	0.2490	0.0671
morphPolarityacc	0.3350	0.1674
morphPronTypeacc	0.1535	0.2680
morphReflexacc	0.5620	0.7051
morphTenseacc	0.2149	0.1241
morphTypoacc	0.5081	—
morphVerbFormacc	0.4912	0.2364
morphVoiceacc	0.0201	0.2602
morphPoliteacc	—	0.1436
morphmicroacc	0.3076	0.2915

Notes:

—— means that metric wasn’t present in that dataset’s reported results (e.g., morph_Typo_acc only in BOUN; morph_Polite_acc only in IMST).

Acknowledgments

This research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC), like most of our projects. Many thanks to TRC team once again.