agentlans/multilingual-sentences

Multilingual Sentences

Dataset contains sentences from 50 languages, grouped by their two-letter ISO 639-1 codes. The "all" configuration includes sentences from all languages.

Dataset Overview

Multilingual Sentence Dataset is a comprehensive collection of high-quality, linguistically diverse sentences. Dataset is designed to support a wide range of natural language processing tasks, including but not limited to language modeling, machine translation, and cross-lingual studies.

Methods

Rigorous methodology consisted of three main stages: text preprocessing, language detection, and dataset processing.

Text Preprocessing

Sophisticated text cleaning pipeline using the textacy library, which included:

• —Removal of HTML tags, email addresses, URLs, and emojis
• —Unicode and whitespace normalization
• —Standardization of punctuation and word formats

Language Detection

Google CLD3 library utilized for accurate language identification:

• —Implemented NNetLanguageIdentifier
• —Configured for processing texts between 0-1000 bytes
• —Included reliability assessment for each language detection

Dataset Processing

Workflow for dataset creation involved the following steps:

1. 1.Streamed loading of the LinguaNova multilingual dataset
2. 2.Application of the text preprocessing pipeline
3. 3.Sentence segmentation using PyICU for accurate boundary detection
4. 4.Quality filtering:
5. 5.Length constraint (maximum 2048 characters per sentence)
6. 6.High-reliability language verification
7. 7.Extraction of unique sentences
8. 8.Random shuffling for unbiased sampling
9. 9.Generation of language-specific files

Technical Details

Libraries and Tools

• —textacy: Advanced text preprocessing
• —Google CLD3: State-of-the-art language detection
• —Hugging Face datasets: Efficient data handling and processing
• —SentenceBreaker: Accurate sentence segmentation

Implementation Notes

• —Process was executed consistently across all 50 languages to ensure uniformity and high quality in the multilingual dataset preparation.
• —Special attention was given to maintaining the integrity of each language's unique characteristics throughout the processing pipeline.

Data Splits

Dataset is organized into the following splits:

• —Individual language files: Contains sentences for each of the 50 languages
• —"all" configuration: Aggregates sentences from all languages into a single dataset

Limitations and Biases

While extensive efforts were made to ensure dataset quality, users should be aware of potential limitations:

• —Language detection accuracy may vary for very short texts or closely related languages
• —Dataset may not fully represent all dialects or regional variations within each language
• —Potential biases in the original LinguaNova dataset could be carried over

Ethical Considerations

Users of this dataset should be mindful of:

• —Potential biases in language representation
• —Need for responsible use in AI applications, especially in multilingual contexts
• —Privacy considerations, although personal identifiable information should have been removed