opencsg/CIMD

CIMD

<div align="center"> <a href="#中文说明" target="self">[[中文]]</a> | <a href="#english" target="self">[[English]]</a> </div>

<div align="center">

CSGHub Dataset Page | Hugging Face | OpenCSG Community

</div>

<div align="center"> <img src="assets/cimd-overview.png" alt="CIMD overview infographic" width="900"> </div>

中文说明

数据集概述

CIMD 是一个面向文档智能任务的跨来源、多语言 JSONL 语料库。当前公开快照包含 111,308 条解析记录，覆盖制度参考、学术与长文档资料、机构分析、企业运营、公共讨论和市场相关材料等来源家族。每条记录都把正文与来源类型、语言、时间、关键词、授权标签和来源字段放在同一个结构里，用户拿到数据后可以直接做检索、抽样、审计和数据治理。

公开数据已转换为统一字段，并按来源家族拆分为可单独加载的子集；它不是原始文件夹的简单打包。用户可以只读取制度参考、学术长文档或公共讨论记录，也可以合并多个子集构建检索库、抽取训练候选样本、构造评测样本池，并按来源、语言和时间字段继续筛选。

CIMD 和通用网页语料的差别在于记录级元数据。它不只提供可索引文本，还提供 file_id、data_id、source_type、时间字段、语言标签、关键词和授权标记。和单一文献集合相比，它同时保留多类文档来源；和普通文件导出相比，它有稳定的 JSONL schema、子集配置和 manifest。研发团队可以用它做向量化、检索样本构建和模型数据筛选，数据团队可以用它做样本核验、版本对比和资产登记。

数据特点

来源家族清晰。 CIMD 把制度参考、学术出版物、长文档资料、机构分析、企业资料、公开观点和市场相关材料放进同一套记录体系。不同来源保留各自的 source_type，也能通过统一字段一起检索、筛选和统计。需要多类证据共同支持的任务，可以在同一份语料内完成跨来源召回。

长文档更容易加载。 公开数据主要来自 PDF、DOCX、JSONL 等载体，发布时统一整理为按行读取的 JSONL。raw_chunk 保存解析后的文本块，单个源文件可以对应多条记录。当前块长不是固定 token 长度，公开快照的 raw_chunk 中位数字符数约为 3,951，P95 约为 4,091；进入向量库或长上下文模型前，用户可按窗口长度重新切分。

元数据跟随正文。 file_id、data_id、file_name、source_type、original_time、content_time、language、keywords、license_type 和 source_details 等字段都在记录体内。使用者可以按来源、时间、语言和主题过滤，也可以把检索命中、模型回答或抽检样本回溯到具体记录。当前公开快照的 source_details 仅在有可发布来源说明时填充，记录级追溯主要依赖 file_id、data_id、file_name 和 source_type。

覆盖中英文和其他语言标签。 当前快照包含 en、zh 和 other 三类语言标签，可用于构建跨语种检索样本、多语言文档分类语料和双语知识库。需要精确到具体小语种的任务，应先检查或重新标注 other 类记录。

子集可以单独使用。 仓库按 reference_governance、scholarly_literature、enterprise_operations、public_discourse 等来源家族组织。用户可以只加载一个子集做专项任务，也可以把多个子集合并成完整语料。这个结构适合做分组实验、增量验证和权限分层；如果要发布标准 benchmark，需要另行构造查询、标注和评价集。

提供固定加载配置。 当前版本声明了完整的 dataset configs，每个子集都有固定配置名和路径。用户可以通过 Git LFS 获取完整文件，也可以用 Hugging Face datasets 的配置名流式加载；ModelScope 用户也能按 subset_name 读取。下载、抽样和预处理可以复用同一套路径约定。

支持检索结果回溯。 RAG 系统除了正文，还需要标题、来源、时间和语言等字段。CIMD 将这些字段保留在记录中，用户可以在检索时做来源过滤和时间窗口筛选，并在生成回答后回到记录层面核验证据。

公开版本经过发布前筛选。 当前版本仅保留本次可公开发布的记录。筛选范围包括元数据完整性、来源可追溯性、授权标记和解析质量；用户在训练、分发或商用前仍需结合具体来源核验授权范围。

方便做数据资产盘点。 dataset_manifest.json 保留了公开快照的总体规模、子集规模、语言分布、格式分布和来源类型分布。使用者可以把它作为数据清单，也可以用来做后续版本对比、质量抽检和数据目录登记。

数据组成

当前公开快照包含 111,308 条 JSONL 记录，覆盖 9,655 个去重 file_id，保留 35 类 source_type。

快照统计

语言分布：

格式分布：

目录结构

data/
  corpus/
    reference_governance/train.jsonl
    scholarly_literature/train.jsonl
    public_discourse/train.jsonl
    enterprise_operations/train.jsonl
    institutional_analysis/train.jsonl
    market_observations/train.jsonl
    miscellaneous_records/train.jsonl
dataset_manifest.json
OpenCSG数据集许可协议.md

数据结构

当前仓库以 JSONL 为主，每行对应一条解析记录。单个源文件可以对应多条记录，因此文件数与记录数不是同一统计口径。

示例：

{
  "format": "pdf",
  "file_id": "7d49b52f6e2f48f7a56f169b0f8d9214",
  "file_name": "policy_reference.pdf",
  "data_id": "7d49b52f6e2f48f7a56f169b0f8d9214-0001",
  "title": "Administrative reference note",
  "source_type": "国家法律法规",
  "author": "Public agency",
  "original_time": "2024-01-04 00:00:00",
  "content_time": "2024-01-03 00:00:00",
  "data_version": "1.0.0",
  "is_generated": "0",
  "country": "中国",
  "language": "zh",
  "keywords": [
    "政策法规"
  ],
  "license_type": "商业授权",
  "raw_chunk": "...",
  "source_details": "https://example.com/reference/2024-01-03"
}

获取与加载

通过 Git 获取：

git lfs install
git clone https://opencsg.com/datasets/OpenCSG/CIMD.git
cd CIMD
git lfs pull

使用 Hugging Face datasets：

from datasets import load_dataset

dataset = load_dataset(
    "opencsg/CIMD",
    "reference_governance",
    split="train",
    streaming=True,
)

使用 ModelScope：

from modelscope.msdatasets import MsDataset

dataset = MsDataset.load(
    dataset_name="CIMD",
    namespace="OpenCSG",
    subset_name="reference_governance",
    split="train",
)

适用场景

• —多来源文档检索与 RAG
• —长文档问答与证据归因
• —文档分类、来源识别和主题标注
• —数据目录、质量抽检和授权审计
• —继续训练语料筛选与评测集构建
• —企业知识库和内部文档智能应用

使用注意

• —当前计数单位为解析记录，不等同于去重后的原始文档数。
• —当前公开子集通过 Git LFS 管理。
• —不同来源之间可能存在重复、近重复或解析噪声。
• —时间字段可能表示发布时间、内容时间或抽取时间。
• —用于训练、分发或商用前，需要结合来源信息核验实际授权范围。

许可说明

使用本数据集需要遵循 OpenCSG 数据集许可协议。仓库 metadata 中的 license: other 表示本数据集采用平台预设列表之外的许可协议，实际许可条款以该协议为准。

本数据集可按 OpenCSG 数据集许可协议申请商业用途。若计划将本数据集，或基于本数据集训练、增强的模型、系统、Agent、API 服务和商业产品用于商业场景，请发送邮件至 lorraineg@opencsg.com 获取许可。

当前公开快照中的 license_type: 商业授权 是记录级授权来源标记，不替代仓库级许可协议。

引用

@dataset{opencsg_cimd_2026,
  title        = {CIMD: A Cross-Source Multilingual Document Corpus},
  author       = {OpenCSG},
  year         = {2026},
  url          = {https://opencsg.com/datasets/OpenCSG/CIMD},
  note         = {OpenCSG dataset repository}
}

<div align="center"> <a href="#中文说明" target="self">[[中文]]</a> | <a href="#english" target="self">[[English]]</a> </div>

English

Overview

CIMD is a 111,308-record JSONL corpus for document-intelligence work across publicly available and authorized document sources. The snapshot covers governance references, scholarly and long-form documents, institutional analysis, enterprise operations, public discourse, market-related records, and miscellaneous records. Each text chunk uses a consistent record schema that keeps source, language, time, license, keyword, and provenance fields alongside the text.

The public release puts heterogeneous materials into one schema and splits them into source-family subsets that can be loaded independently. Users can load only governance references or long-form scholarly records, combine several subsets for retrieval, or use the full snapshot for corpus filtering and model-data preparation. The layout is meant to work in ordinary data pipelines rather than only as a static archive.

Unlike general web corpora, CIMD exposes record-level provenance, source-family splits, license labels, language tags, timestamps, and stable record IDs in every sample. Unlike a single-source publication dump, it includes several document families under one schema. That makes it easier to build RAG systems with citations, evaluate retrieval behavior, inspect training samples, and maintain a data catalog without rebuilding the metadata layer from scratch.

Key Properties

Cross-source structure. CIMD combines policy and reference material, academic publications, long-form documents, institutional reports, enterprise material, public commentary, and market-related records. Source families remain visible through source_type and subset names, while the fields stay consistent enough for unified loading and retrieval. Applications that need evidence from more than one kind of document can work within one corpus.

Prepared for long-document workflows. Most records originate from document formats such as PDF and DOCX, then ship as line-oriented JSONL. A source file can produce multiple records, which keeps the file-level relationship while giving retrieval and review systems a manageable unit of text. This works well for chunk indexing, vector stores, long-document QA, and batch quality checks.

Record-level metadata. Fields such as file_id, data_id, source_type, original_time, content_time, language, keywords, license_type, and source_details are stored inside each record. Users can filter by source, language, time range, topic, or license label, and they can trace a retrieved passage or sampled training record back to its record identifier.

English, Chinese, and records labeled `other`. The snapshot includes en, zh, and other language labels. It can support single-language experiments, bilingual retrieval, and mixed-language document classification. Users who need a specific language inside other should inspect or reclassify those records before relying on language-specific results.

Organized by source family. The repository is organized by source family: reference_governance, scholarly_literature, enterprise_operations, public_discourse, and related subsets. This makes it possible to load only reference material, isolate public commentary, or compare retrieval behavior across different document families without writing custom split logic first.

Ready for common data tooling. The current release declares subset config names and paths for each subset. Users can pull the files with Git LFS, stream a subset through Hugging Face datasets, or load the same subset name through ModelScope. Offline processing jobs, sample selection, index builds, and serving jobs can use one path convention.

Useful for explainable retrieval. RAG systems need text passages together with titles, source labels, dates, languages, and license fields. CIMD keeps those fields in the record, so downstream systems can return attributed answers, apply source filters, audit retrieved evidence, and reproduce sampling decisions.

Public-release filtered snapshot. The released version keeps records that meet the rules for this public snapshot. The filter covers metadata and body text, with checks for release eligibility, license and provenance traceability, field completeness, content suitability, and parsing quality. This release filter is a publication step, not a substitute for the user's own review of provenance, authorization, privacy boundaries, and downstream-use rights.

Manifest for data operations. dataset_manifest.json records snapshot size, subset counts, language distribution, format distribution, and retained source-type distribution. It can be used as a quick inventory for data catalogs, release checks, quality review, and future version comparisons.

Data Composition

The current public snapshot contains 111,308 JSONL records, 9,655 unique file_id values, and 35 retained source_type values.

Snapshot Statistics

Language distribution:

Format distribution:

Repository Layout

data/
  corpus/
    reference_governance/train.jsonl
    scholarly_literature/train.jsonl
    public_discourse/train.jsonl
    enterprise_operations/train.jsonl
    institutional_analysis/train.jsonl
    market_observations/train.jsonl
    miscellaneous_records/train.jsonl
dataset_manifest.json
OpenCSG数据集许可协议.md

Schema

The repository primarily stores JSONL records. Each line corresponds to one parsed record. A source file can produce more than one record.

Access and Loading

Access via Git:

git lfs install
git clone https://opencsg.com/datasets/OpenCSG/CIMD.git
cd CIMD
git lfs pull

Load with Hugging Face datasets:

from datasets import load_dataset

dataset = load_dataset(
    "opencsg/CIMD",
    "reference_governance",
    split="train",
    streaming=True,
)

Load with ModelScope:

from modelscope.msdatasets import MsDataset

dataset = MsDataset.load(
    dataset_name="CIMD",
    namespace="OpenCSG",
    subset_name="reference_governance",
    split="train",
)

Intended Uses

• —Multi-source document retrieval and RAG
• —Long-document QA and evidence attribution
• —Document classification, source identification, and topic tagging
• —Data cataloging, quality review, and authorization checks
• —Training-corpus filtering and benchmark construction
• —Enterprise knowledge-base and document-intelligence applications

Limitations

• —Counts use parsed records rather than deduplicated source documents.
• —Public subset files are stored with Git LFS.
• —Records from different sources may contain duplicates, near-duplicates, or parsing noise.
• —Time fields may represent publication time, content time, or extraction time depending on the source.
• —Users should verify provenance and the effective authorization scope before redistribution or downstream commercial use.

Licensing

Use of this dataset is governed by the OpenCSG Dataset License Agreement. The license: other metadata value indicates that the license is outside the platform's preset license list. The license file in this repository is authoritative. This dataset supports commercial use. For commercial use of the dataset, models, systems, agents, APIs, or products trained or enhanced with this dataset, follow the license agreement and contact lorraineg@opencsg.com for authorization.

Citation

@dataset{opencsg_cimd_2026,
  title        = {CIMD: A Cross-Source Multilingual Document Corpus},
  author       = {OpenCSG},
  year         = {2026},
  url          = {https://opencsg.com/datasets/OpenCSG/CIMD},
  note         = {OpenCSG dataset repository}
}