MongoDB/devcenter-articles

Overview

This dataset consists of ~600 articles from the MongoDB Developer Center.

Dataset Structure

The dataset consists of the following fields:

• —sourceName: The source of the article. This value is devcenter for the entire dataset.
• —url: Link to the article
• —action: Action taken on the article. This value is created for the entire dataset.
• —body: Content of the article in Markdown format
• —format: Format of the content. This value is md for all articles.
• —metadata: Metadata such as tags, content type etc. associated with the articles
• —title: Title of the article
• —updated: The last updated date of the article

Usage

This dataset can be useful for prototyping RAG applications. This is a real sample of data we have used to build the MongoDB Documentation Chatbot.

Ingest Data

To experiment with this dataset using MongoDB Atlas, first create a MongoDB Atlas account.

You can then use the following script to load this dataset into your MongoDB Atlas cluster:

import os
from pymongo import MongoClient
import datasets
from datasets import load_dataset
from bson import json_util


uri = os.environ.get('MONGODB_ATLAS_URI')
client = MongoClient(uri)
db_name = 'your_database_name'  # Change this to your actual database name
collection_name = 'devcenter_articles'

collection = client[db_name][collection_name]

dataset = load_dataset("MongoDB/devcenter-articles")

insert_data = []

for item in dataset['train']:
    doc = json_util.loads(json_util.dumps(item))
    insert_data.append(doc)

    if len(insert_data) == 1000:
        collection.insert_many(insert_data)
        print("1000 records ingested")
        insert_data = []

if len(insert_data) > 0:
    collection.insert_many(insert_data)
    insert_data = []

print("Data ingested successfully!")