Chemin-AI/STEM-en-ms

A Bilingual Dataset for Evaluating Reasoning Skills in STEM Subjects

This dataset provides a comprehensive evaluation set for tasks assessing reasoning skills in Science, Technology, Engineering, and Mathematics (STEM) subjects. It features questions in both English and Malay, catering to a diverse audience.

Key Features

• —Bilingual: Questions are available in English and Malay, promoting accessibility for multilingual learners.
• —Visually Rich: Questions are accompanied by figures to enhance understanding and support visual and contextual reasoning.
• —Focus on Reasoning: The dataset emphasizes questions requiring logical reasoning and problem-solving skills, as opposed to simple recall of knowledge.
• —Real-World Context: Questions are derived from real-world scenarios, such as past SPM (Sijil Pelajaran Malaysia) examinations, making them relatable to students.

Dataset Structure

The dataset is comprised of two configurations: data_en (English) and data_ms (Malay). Both configurations share the same features and structure.

Data Fields

• —FileName: Unique identifier for the source file (alphanumeric).
• —IBSN: International Standard Book Number of the source book (if available).
• —Subject: Academic subject (e.g., Physics, Mathematics).
• —Topic: Specific topic of the question within the subject (may be missing).
• —Questions: Main body of the question or problem statement.
• —Figures: List of associated image files related to the question (empty if no figures are present).
• —Label: Original caption or description of each image in the imgs list.
• —Options: Possible answer choices for the question, with keys (e.g., "A", "B", "C", "D") and corresponding text.
• —Answers: Correct answer to the question, represented by the key of the correct option (e.g., "C").

Data Instance Example

{
    "FileName": "FC064244",
    "ISBN": "9786294703681",
    "Subject": "Physics",
    "Topic": "Measurement",
    "Questions": "State the physical quantity that can be measured using the measuring device shown in Diagram 1.",
    "Figures": [
        {
            "label": "Diagram 1",
            "path": "FC064244_C1_Q12_ImageFile_0.png"
        }
    ],
    "Options": {
        "A": "Weight",
        "B": "Mass",
        "C": "Amount of substance",
        "D": "Volume"
    },
    "Answers": "B"
}

Data Split

The dataset is split between Physics and Mathematics subjects, with some questions lacking topic categorization.

Known Limitations

• —Subject Coverage: The current version focuses on Physics and Mathematics. Future releases will include more STEM subjects.
• —Answer Accuracy: Answers are extracted from various sources and may contain inaccuracies.

Source

The dataset is derived from a combination of resources, including:

• —SPM past-year exams
• —SPM mock exams
• —Educational exercise books

Data Acquisition Method

• —Optical Character Recognition (OCR) for text extraction
• —Manual quality control (QC) to ensure data accuracy

Versioning and Maintenance

• —Current Version: 1.0.0
• —Release Date: December 27, 2024
• —Contact: We welcome any feedback or corrections to improve the dataset quality.

License

This dataset is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

Getting Started

You can access the dataset on Hugging Face using the following commands:

# For English data
pip install datasets

from datasets import load_dataset

dataset = load_dataset("Supa-AI/STEM-en-ms", name="data_en")

# For Malay data
dataset = load_dataset("Supa-AI/STEM-en-ms", name="data_ms")

Bilingual STEM Dataset LLM Leaderboard

This document summarizes the evaluation results for various language models based on 5-shot and First Token Accuracy. The evaluation was conducted across four configurations:

Notes on eval.py

eval.py is a template for evaluating large language models (LLMs), update the script to integrate your API calls or local model logic.

• —The "First Token Accuracy" metric highlights initial token prediction accuracy.
• —The evaluation results are based on the specific dataset and methodology employed.
• —Further analysis might be needed to determine the models' suitability for specific tasks.

Attribution for Evaluation Code

The eval.py script is based on work from the MMLU-Pro repository:

• —Repository: TIGER-AI-Lab/MMLU-Pro
• —License: Apache License 2.0 (included in the NOTICE file)

Contributors

• —**Gele**
• —**Ken Boon**
• —**Wei Wen**