cvnberk/amazon-products

Dataset Creation and Processing Overview

This dataset underwent a comprehensive process of loading, cleaning, processing, and preparing, incorporating a range of data manipulation and NLP techniques to optimize its utility for machine learning models, particularly in natural language processing.

Data Loading and Initial Cleaning

• —Source: Loaded from the Hugging Face dataset repository bprateek/amazon_product_description.
• —Conversion to Pandas DataFrame: For ease of data manipulation.
• —Null Value Removal: Rows with null values in the 'About Product' column were discarded.

Data Cleaning and NLP Processing

• —Sentence Extraction: 'About Product' descriptions were split into sentences, identifying common phrases.
• —Emoji and Special Character Removal: A regex function removed these elements from the product descriptions.
• —Common Phrase Elimination: A function was used to strip common phrases from each product description.
• —Improving Writing Standards: Adjusted capitalization, punctuation, and replaced '&' with 'and' for better readability and formalization.

Sentence Similarity Analysis

• —Model Application: The pre-trained Sentence Transformer model 'all-MiniLM-L6-v2' was used.
• —Sentence Comparison: Identified the most similar sentence to each product name within the cleaned product descriptions.

Dataset Refinement

• —Column Selection: Retained relevant columns for final dataset.
• —Image URL Processing: Split multiple image URLs into individual URLs, removing specific unwanted URLs.

Image Validation

• —Image URL Validation: Implemented a function to verify the validity of each image URL.
• —Filtering Valid Images: Retained only rows with valid image URLs.

Dataset Splitting for Machine Learning

• —Creation of Train, Test, and Eval Sets: Used scikit-learn's train_test_split for dataset division.

For further details or to contribute to enhancing the dataset card, please refer to the Hugging Face Dataset Card Contribution Guide.