language-classification
sunflower_language_classification_v2Indian-sign-language-classificationsunflower_language_classificationsign-language-classificationspecific_bengali_language_token_classification_modelsalt_language_Classificationspecific_arabic_language_token_classification_modelspecific_indonesian_language_token_classification_model
code-natural-language-classification-datasetSampling from codeparrot/github-code under more permissive license ['mit', 'apache-2.0', 'bsd-3-clause', 'bsd-2-clause', 'cc0-1.0'] + sampling from minipile.
It is intended to be used for training code natural language classifier.
Code-Language-Classification
Programming Language Classification Dataset
A large-scale, balanced dataset for programming language identification from source code snippets.
Overview
This dataset contains 1.664 million cleaned and labeled source code samples across 16 programming languages, specifically designed for programming language classification and identification tasks.
Unlike many code datasets that are primarily built for code generation or retrieval, this dataset was curated… See the full description on the dataset page: https://huggingface.co/datasets/kaushik-harsh-99/Code-Language-Classification.task1577_amazon_reviews_multi_japanese_language_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1577_amazon_reviews_multi_japanese_language_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1577_amazon_reviews_multi_japanese_language_classification.LanguageClassification
LanguageClassification
An MTEB dataset
Massive Text Embedding Benchmark
A language identification dataset for 20 languages.
Task category
t2c
Domains
Reviews, Web, Non-fiction, Fiction, Government, Written
Referencehttps://huggingface.co/datasets/papluca/language-identification
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["LanguageClassification"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/LanguageClassification.task1576_amazon_reviews_multi_english_language_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1576_amazon_reviews_multi_english_language_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1576_amazon_reviews_multi_english_language_classification.cyrillic-language-classification
Cyrillic Language Classification Corpus
This corpus contains texts in 26 languages using the Cyrillic alphabet, designed for the task of automatic language identification.
Summary
Languages: 26 individual Cyrillic languages, plus 17 mixed language combinations created through augmentation
Source: texts collected from Wikipedia and augmented using various methods
Size: 20,232 examples in total, including 18,334 original articles collected from Wikipedia and 1,898 texts… See the full description on the dataset page: https://huggingface.co/datasets/AlmaznayaGroza/cyrillic-language-classification.