Process-Venue/Language_Identification_v1

Dataset Card for Language Identification Dataset

Dataset Description

• —Repository: processvenue/language_identification
• —Total Samples: 9627
• —Number of Languages: 10
• —Splits:
• —Train: 6741 samples (70%)
• —Validation: 1441 samples (15%)
• —Test: 1445 samples (15%)

Dataset Summary

A comprehensive dataset for Indian language identification and text classification. The dataset contains text samples across 10 major Indian languages, making it suitable for developing language identification systems and multilingual NLP applications.

Languages and Distribution

Language Distribution:
Urdu         1000
Hindi        1000
Odia         1000
Tamil        1000
Kannada      1000
Bengali      1000
Gujarati     1000
Malayalam    1000
Marathi      1000
Punjabi       627

Language Details

1. 1.Hindi (hi): Major language of India, written in Devanagari script
2. 2.Urdu (ur): Written in Perso-Arabic script
3. 3.Bengali (bn): Official language of Bangladesh and several Indian states
4. 4.Gujarati (gu): Official language of Gujarat
5. 5.Kannada (kn): Official language of Karnataka
6. 6.Malayalam (ml): Official language of Kerala
7. 7.Marathi (mr): Official language of Maharashtra
8. 8.Odia (or): Official language of Odisha
9. 9.Punjabi (pa): Official language of Punjab
10. 10.Tamil (ta): Official language of Tamil Nadu and Singapore

Data Fields

• —text: The input text sample
• —language: The language label (one of the 10 languages listed above)

Usage Example

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("processvenue/language_identification")

# Access splits
train_data = dataset['train']
validation_data = dataset['validation']
test_data = dataset['test']

# Example usage
print(f"Sample text: {train_data[0]['text']}")
print(f"Language: {train_data[0]['language']}")

Applications

1. 1.Language Identification Systems
2. 2.Automatic language detection
3. 3.Text routing in multilingual systems
4. 4.Content filtering by language

1. 1.Machine Translation
2. 2.Language-pair identification
3. 3.Translation system selection

1. 1.Content Analysis
2. 2.Multilingual content categorization
3. 3.Language-specific content analysis

Citation

If you use this dataset in your research, please cite:

@dataset{language_identification_2024,
  author = {ProcessVenue Team},
  title = {Multilingual Headlines Language Identification Dataset},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/processvenue/language-identification}
}

###reference

  @misc{disisbig_news_datasets,
  author = {Gaurav},
  title = {Indian Language News Datasets},
  year = {2019},
  publisher = {Kaggle},
  url = {https://www.kaggle.com/datasets/disisbig/}
  }