hreyulog/arkts-code-docstring
Recommended Version: ArkTS-CodeSearch-Pro We strongly recommend using ArkTS-CodeSearch-Pro for all new research and development.The Pro version is the actively maintained release and provides improved data quality through stricter open-source license filtering, repository quality control (stars > 0), comment normalization, removal of license/copyright headers and code-only comments, minimum docstring length filtering, and global code/query deduplication. The original… See the full description on the dataset page: https://huggingface.co/datasets/hreyulog/arkts-code-docstring.
792
1---2dataset_info:3 features:4 - name: nwo5 dtype: string6 - name: sha7 dtype: string8 - name: path9 dtype: string10 - name: language11 dtype: string12 - name: identifier13 dtype: string14 - name: docstring15 dtype: string16 - name: function17 dtype: string18 - name: ast_function19 dtype: string20 - name: obf_function21 dtype: string22 - name: url23 dtype: string24 - name: function_sha25 dtype: string26 - name: source27 dtype: string28 splits:29 - name: train30 num_bytes: 16212350431 num_examples: 1956132 - name: validation33 num_bytes: 1884385834 num_examples: 244535 - name: test36 num_bytes: 1986779737 num_examples: 244638 download_size: 4231308839 dataset_size: 20083515940configs:41- config_name: default42 data_files:43 - split: train44 path: data/train-*45 - split: validation46 path: data/validation-*47 - split: test48 path: data/test-*49task_categories:50- text-retrieval51language:52- en53- zh54---55 56# Recommended Version: ArkTS-CodeSearch-Pro57 58**We strongly recommend using [ArkTS-CodeSearch-Pro](https://huggingface.co/datasets/hreyulog/arkts-code-docstring-pro) for all new research and development.** 59The Pro version is the actively maintained release and provides improved data quality through **stricter open-source license filtering**, **repository quality control (`stars > 0`)**, comment normalization, removal of license/copyright headers and code-only comments, minimum docstring length filtering, and global code/query deduplication.60 61The original [ArkTS-CodeSearch](https://huggingface.co/datasets/hreyulog/arkts-code-docstring) dataset is preserved only for backward compatibility and reproducing results from previous versions. **New experiments should use ArkTS-CodeSearch-Pro instead.**62 63[Paper](https://huggingface.co/papers/2602.05550) | [Evaluation Code](https://github.com/hreyulog/retrieval_eval) |[Dataset processing Code](https://github.com/hreyulog/arkts-codesearch) | [Fine-Tuned Model](https://huggingface.co/hreyulog/embedinggemma_arkts) | [Repository-Level Split](https://huggingface.co/datasets/hreyulog/arkts-code-docstring-resplit-by-repo) |64 65 66 67 68 69 70 71 72This dataset collects **function-level information from ArkTS (HarmonyOS Ark TypeScript) projects**, including original functions, docstrings, abstract syntax tree (AST) representations, obfuscated versions, and source code metadata. It is suitable for tasks such as code analysis, code understanding, AST research, and code search.73 74ArkTS is a core programming language in the OpenHarmony ecosystem. This dataset was constructed from open-source repositories crawled from GitHub and Gitee, targeting code retrieval and code evaluation tasks.75 76Note: ⚠️ Not Enterprise-Ready: This dataset is designed for research only. Some of the collected code may come from repositories without an explicit open-source license. Users should be aware that usage of such code may be subject to copyright restrictions.7778## Dataset Structure79 80The dataset contains three splits:81 82- `train`: Training set83- `validation`: Validation set84- `test`: Test set85 86Each split is a **JSON Lines (.jsonl) file**, where each line is a JSON object representing a single function.87 88## Features / Columns89 90| Field | Type | Description |91|-------|------|------------|92| `nwo` | string | Repository name |93| `sha` | string | Commit SHA |94| `path` | string | File path |95| `language` | string | Programming language |96| `identifier` | string | Function identifier / name |97| `docstring` | string | Function docstring |98| `function` | string | Original function source code |99| `ast_function` | string | AST representation of the function |100| `obf_function` | string | Obfuscated function source code |101| `url` | string | URL to the code in the repository |102| `function_sha` | string | Function-level SHA |103| `source` | string | Code source (GitHub / Gitee) |104 105## Usage106 107```python108from datasets import load_dataset109 110# Load the dataset111dataset = load_dataset("hreyulog/arkts-code-docstring")112 113# Inspect the first training example114print(dataset["train"][0])115 116# Check dataset features117print(dataset["train"].features)118```119 120### Repo License Configuration121 122Contains repository license information from two sources:123 124- `github`: repos_license_github.jsonl125- `gitee`: repos_license_gitee.jsonl126 127Each line is a JSON object with repository metadata including source platform, repository name, license type, and URL.128 129## Citation130 131If you use this dataset in your research, please cite the following paper:132 133```bibtex134@misc{he2026arktscodesearchopensourcearktsdataset,135 title={ArkTS-CodeSearch: A Open-Source ArkTS Dataset for Code Retrieval}, 136 author={Yulong He and Artem Ermakov and Sergey Kovalchuk and Artem Aliev and Dmitry Shalymov},137 year={2026},138 eprint={2602.05550},139 archivePrefix={arXiv},140 primaryClass={cs.SE},141 url={https://arxiv.org/abs/2602.05550}, 142}143```144