kasuboski/gleam-code-corpus

Gleam Code Corpus

A structured, attributed corpus of 22,581 Gleam source files collected from 1,528 GitHub repositories for continued pre-training and code generation research.

Overview

This dataset contains .gleam source files from the Gleam programming language ecosystem. Every file includes full attribution — repo owner, name, URL, license, star count, and Hex.pm package metadata where available.

What this is for:

• —Continued pre-training (CPT) of code-focused language models
• —Fine-tuning models for Gleam code generation
• —Research on small programming language ecosystems
• —Reference corpus for Gleam idioms and patterns

What this is NOT:

• —Instruction/response pairs (see the companion gleam-sft dataset, coming later)
• —Documentation (language tour, hexdocs — separate dataset, coming later)
• —Raw/unfiltered data (generated files, unlicensed code, and duplicates have been removed)

Dataset Configs

All configs contain only permissively licensed, deduplicated code.

Schema

Each record is a single .gleam file with full attribution:

Collection Methodology

1. 1.Discovery: GitHub search API with language:gleam, using star-range + date-range chunking to bypass the 1,000-result-per-query limit. Discovered 4,660 unique repositories across 17 query chunks.

1. 1.Enrichment: Cross-referenced with Hex.pm search=gleam (1,035 packages) to add package names and download counts.

1. 1.Collection: Shallow-cloned each repo (git clone --depth 1), extracted all .gleam files. 4,489 repos processed, 0 clone failures.

1. 1.Filtering: Applied a multi-stage quality pipeline:

69,413 raw files → 22,581 clean files (67% reduction)

Statistics

Corpus

• —22,581 files from 1,528 repos
• —4.3M lines, 121M characters
• —Median file: 2,692 chars; Mean: 5,352 chars; Max: 49,945 chars

License Distribution

• —MIT: 57% (13,068 files)
• —Apache-2.0: 40% (8,723 files)
• —Other permissive: 3% (MPL-2.0, BSD, ISC, Unlicense, 0BSD, CC0-1.0)

Source Distribution

• —src/ files: 48%
• —test/ files: 24%
• —examples/ files: 6%
• —Other: 22%

Star Distribution

Known Limitations

• —`fork` field is unreliable: GitHub search API doesn't consistently mark forks. All records show fork=false. Some near-duplicate fork content may be present (content dedup catches exact copies only).
• —Near-duplicate forks: Diverged forks (e.g., Ivan-Vakhula07/lustre vs lustre-labs/lustre) survived dedup because their files differ slightly. These are valid Gleam code and not harmful for training.
• —Semi-generated content: dusty-phillips/monks_of_style contributes 496 files (2.2%) of auto-generated CSS property wrappers from MDN documentation. Each file is valid Gleam but follows an identical template.
• —Hex.pm enrichment incomplete: Only 44% of files (10,015) have Hex package metadata. The rest are from repos not published on Hex.
• —Collection snapshot: Data was collected on 2026-05-28. Repositories may have changed since.

Licensing

This dataset aggregates code from many repositories, each with their own license. The license field in each record contains the normalized SPDX identifier. All included code is under permissive licenses:

• —MIT
• —Apache-2.0
• —MPL-2.0
• —BSD-3-Clause
• —BSD-2-Clause
• —ISC
• —Unlicense
• —0BSD
• —CC0-1.0

The dataset card and curation scripts are released under the same license terms as the underlying code. See individual records for specific license attribution.

How to Use

from datasets import load_dataset

# Load full corpus
ds = load_dataset("kasuboski/gleam-code-corpus", config="all")

# Load only official Gleam repos
ds = load_dataset("kasuboski/gleam-code-corpus", config="official")

# Load community top (stars >= 10 or hex_downloads >= 1000)
ds = load_dataset("kasuboski/gleam-code-corpus", config="community_top")

# Access a record
record = ds["train"][0]
print(record["code"])           # the Gleam source code
print(record["repo_url"])       # GitHub URL for attribution
print(record["license"])        # SPDX license identifier

Acknowledgments

• —The Gleam language team and community
• —Hex.pm for package metadata
• —All the repository authors whose code is included in this corpus