ibragim-bad/github-repos-metadata-40M

📊 Metadata for 40 million GitHub repositories

 ![Dataset]() ![HF Hub]()

A cleaned, analysis-ready dataset with per-repository statistics aggregated from GH Archive events: stars, forks, pull requests, open issues, visibility, language signals, and more. Column names mirror the GH Archive / GitHub API semantics where possible. GitHub repo: https://github.com/ibragim-bad/github-repos-metadata-40M

Source: GH Archive (public GitHub event stream).

✅ Projects with complementary ideas

GitHub Repo Embeddings dataset: Puzer/github-repo-embeddings and a cool demo built on top of this embeddings dataset

🚀 Quickstart

from datasets import load_dataset

ds = load_dataset("ibragim-bad/github-repos-metadata-40M", split="full")  # or 'sample' for random sample of 1 million repositories.

📈 EDA Notebook

• —data_visualisation.ipynb: Simple exploratory data analysis of the dataset (column overview, basic distributions, missing-value checks, sample visualizations).

💡 Use cases

• —Trend analysis of stars/forks/PRs
• —Language mix & repo growth signals
• —Ranking by activity / stability
• —Input features for ML models (quality or popularity prediction)

📦 What’s inside

Each row aggregates repository-level stats derived from GH Archive events and GitHub metadata snapshots.

Schema (columns & types)

Values may be null when unavailable from the source.

🏗️ How it’s built (ETL)

• —Data source: GH Archive public event stream; window through 2025-07-23 (UTC).
• —Event filter:
• —CreateEvent (ref_type=repository) to capture initial repo creation and timestamps.
• —PullRequestEvent to obtain rich repo snapshots embedded in PR payloads.

• —Extraction:
• —Parse event.repo and payload.*.repo snapshots for: reponame (owner/name), description, language, licensekey, size, visibility, createdat, openissuescount, forkscount, watchers_count.
• —Track latest observed PR identifier as lastprid.
• —Aggregation:
• —Group by repo_name; keep the most recent snapshot within the window.

Notes

• —Coverage primarily comes from PullRequestEvent snapshots; repositories without any PRs are generally missing.
• —Stars/watchers and forks are taken from event-time snapshots and may be slightly stale versus live GitHub.
• —Values can be null if not present in the underlying event payloads.

⚠️ Limitations

• —GH Archive reflects public events only; private repos are out of scope.
• —Some GitHub fields (e.g., watchers vs stargazers) have historical quirks.
• —Missing values where source is unavailable.

📚 Citation

If you use this dataset, please cite:

@misc{github_repos_activity_stats_2025,
  title        = {GitHub Repos Activity Stats (from GH Archive)},
  author       = {Ibragim Badertdinov},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/ibraigm-bad/github-repos-metadata-40M}}
}

📜 License

Distributed under the MIT License. See LICENSE for more information.