CCB/cis5300-language-models

CIS 5300 Language Models Dataset

Dataset for Homework 3 of CIS 5300 (Natural Language Processing) at Penn.

Cities config

Country-of-origin classification over short city-name strings, drawn from nine countries (Afghanistan, China, Germany, Finland, France, India, Iran, Pakistan, South Africa).

from datasets import load_dataset
cities = load_dataset("CCB/cis5300-language-models", "cities")

Each row has city, country (ISO code), and country_name. For the test split, country and country_name are empty strings -- ground truth is held back so students must submit predictions for grading.

Raw corpus files

For character-level language modeling and cross-domain perplexity work:

Fetch with huggingface_hub.hf_hub_download(..., repo_type="dataset").

Licensing and attribution

• —shakespeare_input.txt and shakespeare_sonnets.txt are in the public domain.
• —modern_english_sample.txt is an excerpt from the English Wikipedia article "Natural language processing" (https://en.wikipedia.org/wiki/Naturallanguageprocessing), used under the Creative Commons Attribution-ShareAlike 4.0 license (https://creativecommons.org/licenses/by-sa/4.0/). The excerpt covers the article's introduction and history sections, with Wikipedia section headers removed so the text reads as continuous prose. See ATTRIBUTIONS.md for details.
• —The cities config is released under CC BY 4.0.