THULab/eea_pm25_forecasting

EEA PM2.5 Air Quality Dataset - Europe (TsFile)

Apache TsFile version of `cosuleabianca/eea-pm25-forecasting`.

Overview

Hourly air-quality and meteorology measurements from European Environment Agency (EEA) monitoring sites prepared for PM2.5 forecasting research: 1,945,152 hourly rows (one per country + site + UTC hour) with PM2.5, NO2 and PM10 concentrations, ERA5-style meteorology (temperature, humidity, dew point, precipitation, pressure, wind components) and engineered calendar, lag (1-168 h) and rolling (3-24 h) features for the three pollutants. The repo's station reference table site_metadata.csv (EEA sampling-point metadata) and config.yaml have no time axis and are mirrored unmodified into this repository.

• —Rows: 1,945,152 hourly observations; ~hundreds of EEA sites across countries (sample: 2018-01-08 start).
• —Column names containing ./- (e.g. PM2.5, PM2.5_lag_1h, StationArea_rural-nearcity) are stored with _ instead (PM25, PM25lag1h, StationArearuralnearcity).

Schema (TsFile structure)

• —Time (INT64, milliseconds) — dt_utc (explicit UTC offsets) as epoch.
• —Country, SiteNumber (TAG, STRING) — one device per monitoring site.
• —dt_local, Start (FIELD, STRING) — source wall-clock labels kept as-is.
• —All other columns FIELD: PM25 / NO2 / PM10 (+ their lag/rolling features), meteorology columns, calendar/cyclical features (hour, dayofweek, ... season, hoursin ...), and the station-type/area boolean flags.

Usage

Install the Apache TsFile Python SDK (pip install tsfile) and read a converted file:

from pathlib import Path
from tsfile import TsFileReader

path = Path("eea_pm25_forecasting_series_1.tsfile")
with TsFileReader(str(path)) as reader:
    schemas = reader.get_all_table_schemas()
    print("tables:", list(schemas))
    table_name = next(iter(schemas))
    table = schemas[table_name]
    columns = [column.get_column_name() for column in table.get_columns()]
    print("columns:", columns)
    field_names = [
        column.get_column_name()
        for column in table.get_columns()
        if column.get_column_name() not in {"Time", "time"}
    ]
    if field_names:
        with reader.query_table(table_name, field_names[:3], batch_size=1024) as result:
            batch = result.read_arrow_batch()
            if batch is not None:
                print(batch.to_pandas().head())

Source & license

• —Original dataset: <https://huggingface.co/datasets/cosuleabianca/eea-pm25-forecasting>
• —License: CC-BY-4.0