qiubinjun/CWE-Bench-Java

1---2configs:3  - config_name: advisory4    data_files: "data/advisory/*.arrow"5  - config_name: project_info6    data_files: "data/project_info/*.arrow"7  - config_name: build_info8    data_files: "data/build_info/*.arrow"9  - config_name: fix_info10    data_files: "data/fix_info/*.arrow"11license: mit12language:13- en14size_categories:15- n<1K16tags:17- code18- cybersecurity19- vulnerability20---21 22# CWE-Bench-Java23 24<!-- Provide a quick summary of the dataset. -->25 26This repository contains the dataset CWE-Bench-Java presented in the paper [LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://arxiv.org/abs/2405.17238). At a high level, this dataset contains 120 CVEs spanning 4 CWEs, namely path-traversal, OS-command injection, cross-site scripting, and code-injection. Each CVE includes the buggy and fixed source code of the project, along with the information of the fixed files and functions. We provide the seed information for each CVE in this repository, as well as advisories.27 28**CWE-Bench-Java Github Repository** (https://github.com/iris-sast/cwe-bench-java) - The CWE-Bench-Java repository contains more details about the benchmark, and how to reproduce the benchmark for the paper. For any feedback, please open an issue.29 30**IRIS Paper Github Repository** (https://github.com/iris-sast/iris) - The IRIS repository contains instructions on reproducing the paper. 31 32## Dataset Details33 34The `raw_data` directory contains the data files from the Github repository.35 36The `data` directory contains the contents of `raw_data`, processed for usage with the Hugging Face datasets library. Includes Arrow-formatted files. 37 38### Project Identifier39 40In this dataset, each project is uniquely identified with a **Project Slug**, encompassing its repository name, CVE ID, and a tag corresponding to the buggy version of the project.41We show one example below:42 43```44DSpace__DSpace_CVE-2016-10726_4.445^^^^^^  ^^^^^^ ^^^^^^^^^^^^^^ ^^^46|       |      |              |--> Version Tag47|       |      |--> CVE ID48|       |--> Repository name49|--> Github Username50```51 52All the patches, advisory information, build information, and fix information are associated with project slugs.53Since there are 120 projects in the CWE-Bench-Java dataset, we have 120 unique project slugs.54Note that a single repository may be found to have different CVEs in different versions.55 56### Packaged Data57 58```59- data/60  - project_info.csv61  - build_info.csv62  - fix_info.csv63- advisory/<project_slug>.json64```65 66The core set of information in this dataset lies in two files, `data/project_info.csv` and `data/fix_info.csv`.67We also provide other essential information such as CVE advisory, and build information for the projects.68We now go into the project information and fix information CSVs.69 70### Advisory 71 72vuln_id | schema_version | published_date | modified_date | aliases | summary | details | cvss_version | cvss_vector | cvss_score | severity_rating | cwe_ids | ecosystem | package_name | introduced_version | fixed_version | references | github_reviewed | github_reviewed_at | nvd_published_at73--------|----------------|----------------|---------------|---------|---------|---------|---------------|--------------|-------------|------------------|---------|-----------|----------------|--------------------|---------------|------------|------------------|---------------------|------------------74CVE-2022-12345 | 1.4.0 | 2022-01-01 | 2022-01-10 | GHSA-xxxx-yyyy-zzzz | Example XSS vuln | Reflected XSS in parameter `q` | CVSS:3.1 | AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N | 6.1 | MODERATE | CWE-79 | Maven | com.example:library | 1.0.0 | 1.0.1 | https://example.com/advisory | true | 2022-01-05T12:00:00Z | 2022-01-01T00:00:00Z75 76This data is extracted from the CWE-Mitre net database and converted to JSON format.77We now get into each field and explain what they are.78- `vuln_id`: a string like `CVE-2021-44667` or `GHSA-xxxx-yyyy-zzzz` representing the unique ID of the vulnerability.79- `schema_version`: a string indicating the schema used to encode this data (e.g., `"1.4.0"`).80- `published_date`: a date (in ISO 8601 format, e.g. `2022-03-12`) representing when the vulnerability was first disclosed.81- `modified_date`: a date representing when the record was last updated.82- `aliases`: a list of strings (e.g., `[ "CVE-2021-44667" ]`) capturing alternate identifiers.83- `summary`: a short string summarizing the vulnerability in one sentence.84- `details`: a longer string giving a full description of how the vulnerability occurs and its impact.85- `cvss_version`: a string like `CVSS:3.1` indicating the version of the CVSS specification used.86- `cvss_vector`: a CVSS vector string (e.g., `AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N`) describing the severity dimensions.87- `cvss_score`: a float between 0.0 and 10.0 quantifying the vulnerability severity.88- `severity_rating`: a string with one of `LOW`, `MODERATE`, `HIGH`, or `CRITICAL` as a qualitative severity label.89- `cwe_ids`: a list of strings like `["CWE-79"]` referring to Common Weakness Enumeration identifiers.90- `ecosystem`: a string such as `"Maven"` or `"npm"` indicating the software package manager ecosystem affected.91- `package_name`: a string like `com.alibaba.nacos:nacos-common` identifying the specific package.92- `introduced_version`: a string denoting the first version where the vulnerability was introduced (e.g., `"0"`).93- `fixed_version`: a string indicating the version where the issue was patched (e.g., `"2.0.4"`).94- `references`: a list of URLs pointing to advisory pages, commits, issue trackers, etc.95- `github_reviewed`: a boolean (`true` or `false`) showing whether GitHub has reviewed this vulnerability.96- `github_reviewed_at`: a timestamp (e.g., `2022-03-14T23:25:35Z`) of when GitHub reviewed the advisory.97- `nvd_published_at`: a timestamp of when the NVD officially published the vulnerability.98 99### Project Info100 101| id | project_slug | cve_id | cwe_id | cwe_name | github_username | github_repository_name | github_tag | github_url | advisory_id | buggy_commit_id | fix_commit_ids |102| -- | ------------ | ------ | -------|----------|-----------------|------------------------|------------|------------|-------------|-----------------|----------------|103| 1 | DSpace__DSpace_CVE-2016-10726_4.4 | CVE-2016-10726 | CWE-022 | Path Traversal | DSpace | DSpace | 4.4 | https://github.com/DSpace/DSpace | GHSA-4m9r-5gqp-7j82 | ca4c86b1baa4e0b07975b1da86a34a6e7170b3b7 | 4239abd2dd2ae0dedd7edc95a5c9f264fdcf639d |104 105Each row in `data/project_info.csv` looks like the example above.106We now get into each field and explain what they are.107 108- `id`: an integer from 1 to 120109- `project_slug`: (explained in the previous section)110- `cve_id`: a common vulnerability identifier `CVE-XXXX-XXXXX`111- `cwe_id`: a common weakness enumeration (CWE) identifier. In our dataset, there is only `CWE-022`, `CWE-078`, `CWE-079`, `CWE-094`112- `cwe_name`: the name of the CWE113- `github_username`: the user/organization that owns the repository on Github114- `github_repository_name`: the repository name on Github115- `github_tag`: the tag associated with the version where the vulnerability is found; usually a version tag116- `github_url`: the URL to the github repository117- `advisory_id`: the advisory ID in Github Security Advisory database118- `buggy_commit_id`: the commit hash (like `ca4c86b1baa4e0b07975b1da86a34a6e7170b3b7`) where the vulnerability can be reproduced119- `fix_commit_ids`: the set of commit hashes (sequentially ordered and separated with semicolon `;`) corresponding to the fix of the vulnerability120 121### Fix Info122 123The `data/fix_info.csv` file contains the fixed Java methods and classes to each CVE.124In general, the fix could span over multiple commits, and a change could be made to arbitrary files in the repository, including resources (like `.txt`, `.html`) and Java source files (including core source code and test cases).125In this table, we only include the methods and classes that are considered core.126Many of the rows in this table is manually vetted and labeled.127Note that there may be fixes on class variables, in which case there will not be method information associated with the fix.128A single function may be "fixed" by multiple commits.129 130Each row in `data/fix_info.csv` looks like the following.131 132| project_slug | cve | github_username | github_repository_name | commit | file | class | class_start | class_end | method | method_start | method_end | signature |133|--------------|-----|-----------------|------------------------|--------|------|-------|-------------|-----------|--------|--------------|------------|-----------|134| apache__activemq_CVE-2014-3576_5.10.2 | CVE-2014-3576 | apache | activemq | `00921f22ff9a8792d7663ef8fadd4823402a6324` | `activemq-broker/src/main/java/org/apache/activemq/broker/TransportConnection.java` | `TransportConnection` | 104 | 1655 | `processControlCommand` | 1536 | 1541 | `Response processControlCommand(ControlCommand)` |135 136- `project_slug`: the unique identifier of each project137- `cve_id`: the CVE id138- `github_username`: the user/organization that owns the repository on Github139- `github_repository_name`: the repository name on Github140- `commit`: the commit hash containing this fix141- `file`: the `.java` file that is fixed142- `class`: the name of the class that is fixed143- `class_start`, `class_end`: the start and end line number of the class144- `method`: the name of the method that is fixed145- `method_start`, `method_end`: the start and end line number of the method146- `signature`: the signature of the method. Note that we might have multiple overloaded methods with the same name but with different signatures147 148 149### Dataset Sources150 151An extension of this dataset can be found on the Github repository, which provides utilities to fetch and build the relevant projects, and a simple website visualizer.152 153- **Curated by:** Ziyang Li, Saikat Dutta 154- **License:** MIT 155 156<!-- Provide the basic links for the dataset. -->157 158- **Repository:** [CWE-Bench-Java](https://github.com/iris-sast/cwe-bench-java/tree/master)159- **Paper [optional]:** [LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://arxiv.org/abs/2405.17238)160 161## Uses162 163<!-- Address questions around how the dataset is intended to be used. -->164 165- Study patterns from previous vulnerability fixes to address current vulnerabilities effectively.166- Evaluate and compare performance of static analysis tools (e.g., CodeQL, Semgrep) on real-world vulnerabilities.167- Assess and measure accuracy, recall, precision, and false-positive rates of various security tools across different vulnerability types.168- Use detailed fix information to improve automatic patch generation and vulnerability remediation systems.169- Provide examples of code vulnerabilities and their respective fixes to educate developers, cybersecurity professionals, and students.170 171### Direct Use Examples172 173<!-- This section describes suitable use cases for the dataset. -->174 175- Analyzing past Java-based CVE fixes on CWE-022 classifications to develop guidelines for addressing file-access vulnerabilities176- Conducting controlled experiments to systematically quantify false positives and true positive detections of security tools for injection177- Developing interactive security training modules that showcase vulnerabilities alongside detailed explanations of the actual patches178 179### Out-of-Scope Use180 181- Because the dataset covers only specific vulnerability types and limited CVEs, it should not be treated as a complete security benchmark for evaluating the entire security posture of software projects.182- Sole reliance on the provided CVE data without additional context or tooling may lead to misinterpretations on what the vulnerability actually is.183- Tools unrelated to static analysis or vulnerability patching (e.g., antivirus software) would likely see limited benefit from this dataset.184 185### Misuse and Malicious Use186 187- Attackers could analyze vulnerable code examples to understand how to exploit similar software weaknesses188- Detailed information about vulnerabilities might aid in crafting targeted attacks against unpatched software versions.189 190### Curation Rationale191 192The dataset was created to provide a high-quality benchmark for evaluating the ability of IRIS to detect and fix real-world vulnerabilities in Java code. 193Existing benchmarks often lack direct links to real CVEs and actionable fixes. 194This dataset bridges that gap with reproducible, well-labeled examples tied to CVEs and CWEs.195 196### Source Data197 198<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->199- GitHub Security Advisories (https://github.com/advisories): Used to extract structured CVE metadata, severity ratings, affected packages, ecosystem information.200- [Github commits](https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/about-commits): Commit logs and diffs were used to identify the buggy and fixed versions of code and determine class/method-level changes needed to fix the vulnerability.201- MITRE CWE Database (https://cwe.mitre.org): Provided the classification, naming, description, and available links related to each vulnerability type.202 203#### Data Collection and Processing204 205<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->206 207- Projects were selected based on the availability of Java source code and documentation on how to fix the error 208- Buggy and fixed code versions using git diff and commit history209- Associated advisory information from GitHub Security Advisories and NVD210- Class and method boundaries using AST analysis 211- Patch validation was performed via manual review and automated testing where possible.212 213## Bias, Risks, and Limitations214 215<!-- This section is meant to convey both technical and sociotechnical limitations. -->216 217- Biased toward Java: Only Java-based CVEs are included; generalization to other languages (e.g., C/C++, Python) should be done with caution.218- Limited in CWE scope: Covers only 4 CWEs — CWE-022, CWE-078, CWE-079, CWE-094.219- Biased toward open-source: Enterprise and closed-source vulnerabilities are excluded, limiting the scope of evaluation.220- Manually vetted, which introduces potential human error or subjective judgment in what counts as the “core fix.”221 222## Citation223 224<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->225Consider citing our paper:226 227```228@article{li2024iris,229      title={LLM-Assisted Static Analysis for Detecting Security Vulnerabilities},230      author={Ziyang Li and Saikat Dutta and Mayur Naik},231      year={2024},232      eprint={2405.17238},233      archivePrefix={arXiv},234      primaryClass={cs.CR},235      url={https://arxiv.org/abs/2405.17238},236}237```238 239## Glossary240 241<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->242 243- CVE (Common Vulnerabilities and Exposures): A unique identifier for a known security vulnerability.244- CWE (Common Weakness Enumeration): A formalized taxonomy of software vulnerability types.245- CVSS (Common Vulnerability Scoring System): A standard for assessing the severity of security vulnerabilities.246- Static Analysis: Code analysis technique that examines source code without executing it.247- Patch: A set of changes to a program designed to fix a known issue or vulnerability248 249## Dataset Card Authors 250 251- Ziyang Li (University of Pennsylvania)252- Saikat Dutta (Cornell University)253- Mayur Naik (University of Pennsylvania)254- Claire Wang (University of Pennsylvania)255- Kevin Xue (University of Pennsylvania)256- Amartya Das257 258## Dataset Card Contact259 260For any feedback, questions, concerns - please [open an issue](https://github.com/iris-sast/cwe-bench-java/issues) on the CWE-Bench-Java Github repository.