SIRIS-Lab/scilake-fulltext-corpus

SciLake Fulltext Corpus

The SciLake Fulltext Corpus is a collection of scientific papers parsed and segmented by section, primarily designed for research in the development and evaluation of NLP models. This dataset contains 1,000 full-text papers from various scientific domains, including Neuroscience, Cancer, Transport, and Energy, along with an additional 5,000 random papers from general scientific domains. All papers have been curated with licenses that allow for legal usage, specifically CC-BY and Public Domain.

The dataset provides detailed metadata and full-text sections, offering a robust resource for domain-specific and general scientific research, dataset annotation, model training, and evaluation.

Corpus Overview

• —1,000 Full-Text Papers Segmented by Section:
• —Domain-specific sections: Neuroscience 🧠, Cancer 🦀, Transport 🛻, Energy 🪫.
• —Each paper is segmented into sections such as Introduction, Methods, Results, etc.
• —5,000 Random Papers from General Scientific Domains:
• —Mix of stratified sampled by MAG level 0 to ensure diversity across multiple domains and disciplines, and random sample.

Example of Dataset Structure:

{
'doi': DOI,
'title': TITLE,
'description': ABSTRACT,
'fulltext_sections': [
        {
            'section_name': SECTION_NAME_1,
            'section_num': SECTION_NUM_1,
            'section_content': SECTION_CONTENT_1,
        },
        ...
],
'fulltext_additional': [
        {
            'section_name': SECTION_NAME_1,
            'section_num': SECTION_NUM_1,
            'section_content': SECTION_CONTENT_1,
        },
        ...
]

How to use

from datasets import load_dataset

dataset_ds = load_dataset("SIRIS-Lab/scilake-fulltext-corpus")

Licensing Information

The SciLake Fulltext Corpus is released under the following licenses:

CC-BY (Creative Commons Attribution), licenses have been obtained from the publisher’s landing page, PDFs, metadata in OpenAire, and Unpaywall, filtering fro those with license CC-BY or Public Domain.

Dataset Acquisition

The papers included in this dataset were sourced through the OpenAIRE index, with random selection to ensure diverse content. The license information was verified by cross-referencing the publisher’s landing pages, metadata from OpenAire, and the Unpaywall database. Papers were retained if they had a CC-BY license or were in the Public Domain.

Funding

This work was partially funded by a projects under EU’s HORIZON Research and Innovation Programme:

• —SciLake (grant agreement No 101058573).

Contact

For more information, or if you have questions, please contact us at sirislab[at]sirisacademic.com.