ci-benchmark-user/ci-repair-bench

CI-REPAIR-BENCH

Overview

CI-REPAIR-BENCH is a benchmark dataset for research on Continuous Integration (CI) failures and automated repair in Python repositories.

The dataset contains 567 CI failure instances collected from 105 real-world GitHub repositories, all written in Python. Each instance captures a CI workflow failure, its logs, the corresponding code diff, and repository-level metadata.

Dataset Statistics

• —Programming language: Python
• —Number of repositories: 105
• —Number of CI instances: 567
• —Domain: Continuous Integration
• —Format: Parquet
• —Size category: 10M < n < 100M
• —Metadata language: English

Data Description

Each row in the dataset represents a CI failure scenario extracted from a GitHub repository.

Main fields

• —language: Programming language of the repository
• —id: Unique identifier of the CI instance
• —repo_owner: GitHub repository owner
• —repo_name: GitHub repository name
• —head_branch: Branch associated with the CI run
• —workflow_name: Name of the CI workflow
• —workflow_filename: CI workflow file name
• —workflow_path: Path to the workflow file
• —sha_fail: Commit SHA where the CI failed
• —sha_success: Commit SHA where the CI succeeded
• —workflow: Full CI workflow configuration
• —logs: CI execution logs capturing failure details
• —diff: Code diff between failing and fixed versions
• —changed_files: List of files modified in the fix
• —commit_link: Link to the failing commit
• —error_type: Categorized CI error type

Example error types

• —Code Linting
• —Code Formatting
• —Dependency Issues
• —Package Installation Error
• —Test Failure
• —Runtime Error
• —Syntax Error
• —Configuration Error
• —Environment Error
• —Assertion Error
• —Documentation / Docstring Error
• —Type Checking Error

Intended Use

This dataset can be used for:

• —CI Log Analysis
• —Automated Program Repair
• —Learning from CI logs and diffs
• —Benchmarking CI repair tools
• —Empirical software engineering research

How to Use the Dataset

Download from Hugging Face Hub

from huggingface_hub import hf_hub_download

dataset_path = hf_hub_download(
    repo_id="ci-benchmark-user/ci-repair-bench",
    filename="ci_repair_dataset.parquet",
    repo_type="dataset",
    token=config.get("HUGGINGFACE_TOKEN")
)