THULab/deep-space-probes

Deep Space Probes -- Merged Hourly Data (TsFile)

This dataset is a lossless conversion to the [Apache TsFile](https://tsfile.apache.org/) format of the HuggingFace dataset `juliensimon/deep-space-probes`. The original observations come from the NASA Space Physics Data Facility (SPDF).

Original dataset

• —Source dataset: juliensimon/deep-space-probes
• —Author: Julien Simon
• —Data origin: NASA SPDF (https://spdf.gsfc.nasa.gov/)
• —License: CC-BY-4.0
• —Content: Merged hourly measurements of magnetic field, solar-wind plasma, and energetic-particle fluxes from humanity's four most distant spacecraft (Voyager 1/2, Pioneer 10/11), spanning 1972-01-01 .. 2025-12-31 (UTC).

Scale

• —1,183,368 hourly records, 49 columns
• —4 spacecraft (each stored as an independent device / TAG):
• —Voyager 1: 403,224 rows
• —Voyager 2: 385,704 rows
• —Pioneer 10: 210,384 rows
• —Pioneer 11: 184,056 rows

TsFile storage mapping (table model)

Conversion notes

• —No columns were dropped: all 47 original measurement columns are preserved as DOUBLE.
• —Nulls are kept as-is (not filled, not removed). The spacecraft carry different instrument suites, so some columns are entirely null for a given spacecraft (e.g. CRS channels exist only on the Voyagers, CRT channels only on the Pioneers). This is a property of the data itself; when stored per-device, a column that is all-null for a device is simply not written for that device — this is not an active column drop.
• —Time precision is ms (the source datetime is on exact hour boundaries, so milliseconds are lossless).
• —Within each spacecraft, rows are sorted ascending by Time; (spacecraft, datetime) is already unique and free of duplicates in the source.

Data integrity / read-back notes

The converted files were verified timestamp-by-timestamp and value-by-value against the source and match exactly (max absolute difference on matched values = 0.0, null vs. 0.0 distinguished correctly, per-spacecraft row counts identical to the source parquet, 1,183,368 in total).

⚠️ Read-back tip: this dataset is highly sparse (a given timestamp usually has only a few populated columns, and some columns have long leading runs of nulls). When using the TsFile Python SDK query_table, querying on a single sparse column alone may return fewer rows than expected (a known query-layer behaviour; the file contents themselves are complete). Include one almost-always-populated column (e.g. heliocentric_distance_au) in the query to reliably obtain the full set of device rows.

Layout

The TsFiles live under data/. The tool emits the data as multiple .tsfile shards (one new shard per 1,000,000 rows), which together form the complete dataset:

data/
├── deep_space_probes_1.tsfile
└── deep_space_probes_2.tsfile

Usage

from tsfile import TsFileReader

reader = TsFileReader("data/deep_space_probes_1.tsfile")
schemas = reader.get_all_table_schemas()
tname = next(iter(schemas))

# Include a dense column to avoid row truncation on sparse-column queries
cols = ["spacecraft", "heliocentric_distance_au", "b_magnitude_nt", "flow_speed_kms"]
with reader.query_table(tname, cols, batch_size=65536) as rs:
    while (batch := rs.read_arrow_batch()) is not None:
        df = batch.to_pandas()
        # ... process ...
reader.close()

Citation

@dataset{deep_space_probes,
  title     = {Deep Space Probes -- Merged Hourly Data},
  author    = {juliensimon},
  year      = {2026},
  url        = {https://huggingface.co/datasets/juliensimon/deep-space-probes},
  publisher = {Hugging Face}
}

Data origin: NASA Space Physics Data Facility (SPDF, https://spdf.gsfc.nasa.gov/). Original dataset licensed under CC-BY-4.0.