Team Ai
Datasetpublic

Alaamer/medium-articles-posts-with-content

Medium Articles Dataset Generator This project combines multiple datasets from Kaggle and Hugging Face to create a comprehensive collection of Medium articles. The combined dataset is available on Hugging Face Hub. Dataset Description This dataset is a unique compilation that not only combines multiple sources but also ensures data quality through normalization and deduplication. A key feature is that all entries in the text column are unique - there are no… See the full description on the dataset page: https://huggingface.co/datasets/Alaamer/medium-articles-posts-with-content.

sourceHugging Facemitupdated 2y agoView on Hugging Face
3likes306downloads
Dataset Card

Medium Articles Dataset Generator

This project combines multiple datasets from Kaggle and Hugging Face to create a comprehensive collection of Medium articles. The combined dataset is available on Hugging Face Hub.

Dataset Description

This dataset is a unique compilation that not only combines multiple sources but also ensures data quality through normalization and deduplication. A key feature is that all entries in the text column are unique - there are no duplicate articles in the final dataset.

Data Sources:

Kaggle Sources:
  • —aiswaryaramachandran/medium-articles-with-content
  • —hsankesara/medium-articles
  • —meruvulikith/1300-towards-datascience-medium-articles-dataset
Hugging Face Sources:
  • —fabiochiu/medium-articles
  • —Falah/mediumarticlesposts

Features

  • —Combines multiple data sources into a single, unified dataset
  • —Ensures uniqueness: Each article appears only once in the dataset
  • —Quality control:
  • —Removes duplicate entries based on article text
  • —Handles missing values
  • —Normalizes data format
  • —Saves the final dataset in efficient Parquet format
  • —Publishes the dataset to Hugging Face Hub

Requirements

bash
pip install datasets
pip install kagglehub huggingface_hub tqdm

Usage

  1. 1.Set up your Hugging Face authentication token
  2. 2.Run the script:
bash
python combined_medium_ds_generator.py

Data Processing Steps

  1. 1.Downloads datasets from Kaggle and Hugging Face
  2. 2.Normalizes each dataset by:
  3. 3.Removing null values
  4. 4.Eliminating duplicates
  5. 5.Standardizing column names
  6. 6.Combines all datasets into a single DataFrame
  7. 7.Saves the result as a Parquet file
  8. 8.Uploads the final dataset to Hugging Face Hub

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author

Acknowledgments

Special thanks to the original dataset creators:

  • —aiswaryaramachandran
  • —hsankesara
  • —meruvulikith
  • —fabiochiu
  • —Falah