datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
code-natural-language-classification-datasetSampling from codeparrot/github-code under more permissive license ['mit', 'apache-2.0', 'bsd-3-clause', 'bsd-2-clause', 'cc0-1.0'] + sampling from minipile.
It is intended to be used for training code natural language classifier.
Code-Language-Classification
Programming Language Classification Dataset
A large-scale, balanced dataset for programming language identification from source code snippets.
Overview
This dataset contains 1.664 million cleaned and labeled source code samples across 16 programming languages, specifically designed for programming language classification and identification tasks.
Unlike many code datasets that are primarily built for code generation or retrieval, this dataset was curated… See the full description on the dataset page: https://huggingface.co/datasets/kaushik-harsh-99/Code-Language-Classification.task1577_amazon_reviews_multi_japanese_language_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1577_amazon_reviews_multi_japanese_language_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1577_amazon_reviews_multi_japanese_language_classification.LanguageClassification
LanguageClassification
An MTEB dataset
Massive Text Embedding Benchmark
A language identification dataset for 20 languages.
Task category
t2c
Domains
Reviews, Web, Non-fiction, Fiction, Government, Written
Referencehttps://huggingface.co/datasets/papluca/language-identification
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["LanguageClassification"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/LanguageClassification.task1576_amazon_reviews_multi_english_language_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1576_amazon_reviews_multi_english_language_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1576_amazon_reviews_multi_english_language_classification.cyrillic-language-classification
Cyrillic Language Classification Corpus
This corpus contains texts in 26 languages using the Cyrillic alphabet, designed for the task of automatic language identification.
Summary
Languages: 26 individual Cyrillic languages, plus 17 mixed language combinations created through augmentation
Source: texts collected from Wikipedia and augmented using various methods
Size: 20,232 examples in total, including 18,334 original articles collected from Wikipedia and 1,898 texts… See the full description on the dataset page: https://huggingface.co/datasets/AlmaznayaGroza/cyrillic-language-classification.Language-Agency-Classification
Language Agency Classifier Dataset
The Language Agency Classifier Dataset was created by (Wan et al., 2023) and aims to provide data for training models to
classify sentences based on the level of agency expressed in each sentence. This repository is organized by Miri Liu (github: emmatliu).
Classifying sentence agency can help expose latent gender bias, where women may
be described with more communal (community-oriented) words and men may be described with more agentic… See the full description on the dataset page: https://huggingface.co/datasets/elaine1wan/Language-Agency-Classification.language_classification_dataset0_Kenyan_Sign_Language_Classificationtask896_miam_language_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task896_miam_language_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks}… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task896_miam_language_classification.flan_combined_task896_miam_language_classificationflan_combined_task1576_amazon_reviews_multi_english_language_classificationjigsaw-reddit-base-model-language-head-binary-classificationLanguage-Classification-Datasetindic-voices-indian-language-classification
Indic Language Speech Classification Dataset
A curated subset of the AI4Bharat IndicVoices-R dataset created for
Indian-language speech classification experiments on consumer hardware.
The original IndicVoices-R corpus is extremely large (~750 GB). This
dataset intentionally samples a much smaller subset so that researchers
and developers can download, preprocess, and experiment with the data
locally on machines with limited storage and memory.
Languages
The… See the full description on the dataset page: https://huggingface.co/datasets/iqbaltaha3/indic-voices-indian-language-classification.Lots-of-LoRAs_task1576_amazon_reviews_multi_english_language_classificationCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données Lots-of-LoRAs/task1576_amazon_reviews_multi_english_language_classification.
Lots-of-LoRAs_task1577_amazon_reviews_multi_japanese_language_classificationCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données Lots-of-LoRAs/task1577_amazon_reviews_multi_japanese_language_classification.
supergoose_flan_combined_task1576_amazon_reviews_multi_english_language_classificationCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données supergoose/flan_combined_task1576_amazon_reviews_multi_english_language_classification.
Lots-of-LoRAs_task896_miam_language_classificationCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données Lots-of-LoRAs/task896_miam_language_classification.
supergoose_flan_combined_task896_miam_language_classificationCe répertoire est vide, il a été créé pour améliorer le référencement du jeu de données supergoose/flan_combined_task896_miam_language_classification.