Thalesian/cuneiformBase-400m

cuneiformBase-400m

Introducing cuneiformBase-400m, a multilingual model capable of handling translation, transliteration, and script conversion tasks across multiple ancient languages: Akkadian, Sumerian, Hittite, Linear B, and Elamite.

1. Model Description

This is an instruct model based on Google's umt5-base (768 hidden dimensions, 12 encoder layers, 12 decoder layers). Unlike the original UMT5 architecture which uses untied input/output embeddings, this model uses tied embeddings (~396M parameters). It supports translation to and from English (and German for Hittite), transliteration between cuneiform signs and Latin characters, and script conversion across five ancient writing systems.

Three styles of transliteration are supported where applicable:

• —Plain transliteration -- standard scholarly transliteration following CDLI notation style
• —Complex transliteration -- includes special symbols, subscript numbers, and determinatives
• —Simple transliteration -- stripped of all special symbols and diacritics, syllables merged to form words

Akkadian Instructions

Translation:

Transliteration:

Script Conversion:

Sumerian Instructions

Translation:

Transliteration:

Script Conversion:

Hittite Instructions

Translation:

Linear B Instructions

Translation:

Transliteration:

Script Conversion:

Elamite

Elamite was included in training on a limited corpus. Due to insufficient validation data, no evaluation metrics are reported for Elamite at this time. Use with caution and expect lower accuracy than the other supported languages.

Base Model

This is a finetuned version of Google's umt5-base, but with tied embeddings.

2. Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_path = "Thalesian/cuneiformBase-400m"
tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_path)

# Example: Translate Akkadian cuneiform to English
prompt = "Translate Akkadian cuneiform to English: "
input_text = "𒅆 𒁹 𒀭 𒉺 𒉽 𒀀 𒁹 𒄿 𒌋 𒐊 𒀴 𒃻 𒀀 𒌋 𒌋 𒀀 𒌋 𒌋"

inputs = tokenizer(prompt + input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=64)

prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Prediction:", prediction)
> "witness  Nabu-naṣir  son  Na  di-Issar  servant  of  son  king"

3. Training and Evaluation Data

Data was used from the Akkademia project, previously published in PNAS Nexus. Additional data for pre-training and training came from CDLI data for Akkadian and Sumerian, the OARE dataset for Akkadian, Hittite data from the HPM corpus, Linear B data from published syllabary resources, and a limited Elamite corpus. More information on the training data, as well as the test and validation splits, can be found on both the GitHub and published methodology.

Training Procedure

The model was trained in multiple stages with different datasets and collators across all supported languages.

Framework Versions

• —Transformers 5.0.0.dev0
• —PyTorch 2.6.0+cu126
• —Tokenizers 0.21.1

4. Evaluation Metrics

4.1 Akkadian

4.1.1 Akkadian Metrics by Line

4.1.2 Akkadian Metrics by Document

4.1.3 Akkadian Metrics by Line (CDLI Test Set)

4.1.4 Akkadian Metrics by Document (CDLI Test Set)

4.1.5 Akkadian Metrics by Document (OARE Test Set)

4.2 Sumerian

4.2.1 Sumerian Metrics by Line

4.2.2 Sumerian Metrics by Document

4.3 Hittite

4.3.1 Hittite Metrics by Line

4.3.2 Hittite Metrics by Document

The Hittite validation scripts were based on CTH numbers - however the English bleu score for lines (95.62) is implausibly high - we believe there was data leakage for a manually generated training set. This may impact the German as well, but German scores are consistent with past models deployed before the additional English set.

4.4 Linear B

Note: Line-level and document-level metrics are identical for Linear B, as the validation set consists of single-line documents.

4.4.1 Linear B Metrics

4.5 Elamite

Elamite was included during training on a limited corpus. Due to insufficient validation data, no evaluation metrics are available. Results should be treated as experimental.

5. Intended Uses

• —Translation of short cuneiform lines across Akkadian, Sumerian, Hittite, and Linear B
• —Transliteration pipelines converting between cuneiform signs and Latin-script representations
• —Reverse translation from English/German back to ancient language transliterations or cuneiform
• —Comparative studies across multiple ancient writing systems
• —Educational and research applications in digital Assyriology, Sumerology, Hittitology, and Aegean scripts

6. Limitations

• —Context window is limited to 512 tokens; longer texts should be split into individual lines.
• —Sumerian translation quality is notably lower than other languages due to the complexity and limited parallel data for Sumerian.
• —Elamite support is experimental with minimal training data.
• —OARE out-of-domain Akkadian data shows significantly degraded performance, indicating domain sensitivity.
• —The model was trained on scholarly transliterations and may not generalize well to non-standard input formats.
• —Linear B prompts use the term "cuneiform" for the syllabary script for consistency with the prompt format; Linear B is a syllabic script, not cuneiform.

7. How to Cite

@misc{drake2025cuneiformBase400m,
  title        = {{cuneiformBase-400m}: A Multilingual T5 Model for Ancient Script Translation and Transliteration},
  author       = {Drake, B. Lee},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/Thalesian/cuneiformBase-400m}}
}