RGES-PIT/MachineLearning

1---2license: cc0-1.03configs:4  - config_name: observations5    data_files: '*_obs.parquet'6  - config_name: metadata7    data_files: '*_meta.parquet'8  - config_name: epochs9    data_files: '*_epoch.parquet'10---11 12# Machine Learning Tier13 14This dataset is a collection of synthetic microlensing light curves from the Nancy Grace Roman Space Telescope Galactic Bulge Time Domain Survey. It is intended for the training and benchmarking of machine learning models for microlensing event classification, parameter estimation, and anomaly detection.15 16The raw distribution of event properties is not representative of what Roman will see, but should span a statistically larger set of events. More representative samples can be drawn using the `final_weight` column, which is roughly proportional to the rate at which events occur relative to others in their class.17 18The machine learning tier data is provided as a set of normalized tables in Parquet format: **RMDC26_ML_Data_obs.parquet**, **RMDC26_ML_Data_meta.parquet**, and **RMDC26_ML_Data_epoch.parquet**.19 20## Entity-Relationship Diagram21 22```23┌───────────────────┐        ┌───────────────────┐        ┌───────────────────┐24│     metadata      │        │   observations    │        │      epochs       │25├───────────────────┤        ├───────────────────┤        ├───────────────────┤26│ event_id (PK)     │◄───────├ event_id (FK)     │    ┌──►│ epoch_id (PK)     │27│ name              │        │ epoch_id (FK) ────┼────┘   │ bjd               │28│ sim_label         │        │ filt              │        │ obs_x             │29│ tE_helio          │        │ flux_uJy          │        │ obs_y             │30│ u0lens1           │        │ flux_err_uJy      │        │ obs_z             │31│ …(physical params)│        │ true_flux_uJy     │        └───────────────────┘32└───────────────────┘        │ saturation_flag   │33                             └───────────────────┘34```35 36**Relationships:**37- **observations → metadata**: many-to-one on `event_id` (each event has one metadata record)38- **observations → epochs**: many-to-one on `epoch_id` (each epoch has many observations)39- To reconstruct a full lightcurve: join all three tables on `event_id` and `epoch_id`40 41## Tables42 43### 1. Observations Table (`_obs.parquet`)44Contains the primary time-series data for all events.45 46| Column Name | Data Type | Description |47| :--- | :--- | :--- |48| `event_id` | int64 | Unique event identifier. |49| `epoch_id` | int64 | Identifier for the specific observation epoch. |50| `filt` | string | Filter used for the observation. Values: `F087`, `F146`, `F213`. |51| `flux_uJy` | float64 | Measured flux in microjanskys (μJy). |52| `flux_err_uJy` | float64 | Uncertainty in the measured flux in μJy. |53| `true_flux_uJy` | float64 | True flux in microjanskys (μJy). |54| `saturation_flag` | float64 | Saturation flag: 0 = unsaturated, 1 = saturated. |55 56### 2. Metadata Table (`_meta.parquet`)57Contains the simulated physical parameters and lensing variables for each event.58 59| Column Name | Data Type | Description |60| :--- | :--- | :--- |61| `name` | string | Unique event identifier (e.g., `RMDC26_XXXXXX`). |62| `event_id` | int64 | Integer ID matching the observations table. |63| `sim_label` | string | Simulation category (e.g., `RMDC26_1S1L_ML`). |64| `SubRun` | float64 | Sub-run identifier. |65| `Field` | float64 | Field identifier. |66| `EventID` | float64 | Event ID. |67| `final_weight` | float64 | Population weight for the event. |68| `ObsGroup_0_chi2` | float64 | For binary events, the $\Delta\chi^2$ between the best-fitting single-lens (PSPL) model and the true simulated model ($\chi^2_{\text{PSPL\_fit}} - \chi^2_{\text{true}}$). For single-source single-lens events, this instead stores the $\Delta\chi^2$ of the true simulated model vs a flat baseline fit ($\chi^2_{\text{flat}} - \chi^2_{\text{true}}$). |69| `is_out_of_season` | boolean | Flag indicating whether the event peak ($t_0$) falls in a seasonal gap (when Roman cannot observe the bulge). |70| `t0lens1` | float64 | Time of closest approach for the primary lens. |71| `tE_ref` | float64 | Einstein timescale (observer reference frame). |72| `tE_helio` | float64 | Einstein timescale (barycentric frame). |73| `u0lens1` | float64 | Impact parameter. |74| `rho` | float64 | Ratio of source radius to the Einstein ring radius. |75| `piE` | float64 | Microlens parallax magnitude. |76| `piEE` | float64 | Parallax-East term. In ecliptic coordinates. |77| `piEN` | float64 | Parallax-North term. In ecliptic coordinates. |78| `piEll` | float64 | Parallax-component parallel to solar acceleration. |79| `piErp` | float64 | Parallax-component perpendicular to solar acceleration. |80| `rE` | float64 | Einstein radius of the primary lens mass (AU). |81| `thetaE` | float64 | Einstein angular radius of the primary lens mass (mas). |82| `ra_deg` | float64 | Event right ascension in degrees. |83| `dec_deg` | float64 | Event declination in degrees. |84| `galactic_l` | float64 | Galactic longitude (degrees). |85| `galactic_b` | float64 | Galactic latitude (degrees). |86| `fs_F146` | float64 | Source flux fraction in F146. |87| `fs_F087` | float64 | Source flux fraction in F087. |88| `fs_F213` | float64 | Source flux fraction in F213. |89| `fs2_F146` | float64 | Second source flux fraction in F146. |90| `fs2_F087` | float64 | Second source flux fraction in F087. |91| `fs2_F213` | float64 | Second source flux fraction in F213. |92| `Lens_F146` | float64 | Lens magnitude in F146. |93| `Lens_F087` | float64 | Lens magnitude in F087. |94| `Lens_F213` | float64 | Lens magnitude in F213. |95| `Source_F146` | float64 | Source magnitude in F146. |96| `Source_F087` | float64 | Source magnitude in F087. |97| `Source_F213` | float64 | Source magnitude in F213. |98| `Source2_F146` | float64 | Second source magnitude in F146. |99| `Source2_F087` | float64 | Second source magnitude in F087. |100| `Source2_F213` | float64 | Second source magnitude in F213. |101| `Source_Is_Binary` | float64 | Flag indicating if the source is binary. |102| `Source_q` | float64 | Mass ratio of the binary source. |103| `Source_combined_logP` | float64 | log period/days of binary source. |104| `Source_total_mass` | float64 | Total mass of binary source (MSun). |105| `Source2_rho` | float64 | Ratio of second source radius to the Einstein ring radius. |106| `Source2_s` | float64 | Projected separation of second source. |107| `Source2_alpha` | float64 | Angle of second source. |108| `Source2_inc` | float64 | Inclination of second source. |109| `Source2_phase` | float64 | Phase of second source. |110| `Planet_mass` | float64 | Mass of the planet (if present). |111| `Planet_q` | float64 | Planet-to-star mass ratio. |112| `Planet_s` | float64 | Projected separation of the planet. |113| `alpha` | float64 | Trajectory angle (degrees)  |114| `Planet_period` | float64 | Orbital period of the planet (years). |115| `Planet_semimajoraxis` | float64 | Semimajor axis of the planet (AU). |116| `Planet_orbphase` | float64 | Orbital phase of the planet (degrees). |117| `Planet_inclination` | float64 | Inclination of the planet (degrees). |118| `Lens_Mass` | float64 | Mass of the lens (MSun). |119| `Lens_Dist` | float64 | Distance to the lens (kpc). |120| `Lens_Teff` | float64 | Effective temperature of the lens (K). |121| `Lens_logg` | float64 | Surface gravity of the lens (cgs). |122| `Lens_Fe_H` | float64 | Metallicity of the lens. |123| `Source_Mass` | float64 | Mass of the source (MSun). |124| `Source_Dist` | float64 | Distance to the source (kpc). |125| `Source_Teff` | float64 | Effective temperature of the source (K). |126| `Source_logg` | float64 | Surface gravity of the source (cgs). |127| `Source_Fe_H` | float64 | Metallicity of the source. |128| `Source2_Mass` | float64 | Mass of the second source (MSun). |129| `Source2_Dist` | float64 | Distance to the second source (kpc). |130| `Source2_Teff` | float64 | Effective temperature of the second source (K). |131| `Source2_logg` | float64 | Surface gravity of the second source (cgs). |132| `Source2_Fe_H` | float64 | Metallicity of the second source. |133| `extinction_F146` | float64 | Extinction in F146. |134| `extinction_F087` | float64 | Extinction in F087. |135| `extinction_F213` | float64 | Extinction in F213. |136 137### 3. Epochs Table (`_epoch.parquet`)138Maps each unique observation time (BJD) and corresponding telescope coordinates (X, Y, Z relative to the Solar System Barycenter) to a unique epoch ID.139 140| Column Name | Data Type | Description |141| :--- | :--- | :--- |142| `epoch_id` | int64 | Unique identifier for the epoch. |143| `bjd` | float64 | Barycentric Julian Date. |144| `obs_x` | float64 | Observatory X-coordinate in AU. |145| `obs_y` | float64 | Observatory Y-coordinate in AU. |146| `obs_z` | float64 | Observatory Z-coordinate in AU. |147 148## Simulation Details & Disclaimer149 150**Simulation Engine:** All events have been generated using the **gulls** simulator. More information is available at [gulls-microlensing.github.io](https://gulls-microlensing.github.io/).151 152**Stellar Populations:** The input stellar population catalogs used for the source and lens stars were generated using **synthpop** ([github.com/synthpop-galaxy/synthpop](https://github.com/synthpop-galaxy/synthpop)).153 154**Event Complexity:**155- There are three simulations included in this dataset:156    - `RMDC26_1S1L_ML`: Single lens and single source events.157    - `RMDC26_1S2L_ML`: One star - one planet lens and single source events.158    - `RMDC26_2S2L_ML`: One star - one planet lens and potentially binary source events without source orbital motion. 159- Note that not all lenses and sources are necessarily involved in the microlensing event. `ObsGroup_0_chi2` > 160 for events from the binary lens simulations indicates that an anomaly may be present in the light curve. The anomalies may be planetary perturbations and caustic crossings, finite source effects, or parallax effects.160 161**Representativeness & Weighting:**162- This data set only includes potentially detectable microlensing events where the chi-squared difference between the best fit flat model and the true model is greater than 60, i.e. $\chi^2_{\text{flat}} - \chi^2_{\text{true}} > 60$.163- The full set of events is **not directly representative** of the raw astrophysical population due to detection significance cuts and GULLS coordinate transformations (e.g. in `croin` caustic-relative coordinates, the host star $t_0$ and $u_0$ are often transformed far outside of normal ranges or observing seasons, even though the planetary anomaly itself is actively observed in-season).164- To make the dataset more representative of the true underlying Galactic population, you can apply the population weights stored in the **`final_weight`** column directly during training and analysis, or use them to draw a resampled, less biased dataset.165- The **`is_out_of_season`** flag indicates whether the closest approach time of the primary lens (`t0lens1`) falls in a seasonal gap when the Galactic Bulge is unobservable. While the lightcurves of these events still show signature features of the microlensing event, their peak stellar magnification is not covered by observations. As a result, these events might be difficult for machine learning models to classify correctly, and you may want to filter them out depending on your specific modeling goals.166 167 168**Dataset Structure:**169- The dataset is split into **observations**, **metadata**, and **epochs**.170- To reconstruct a full lightcurve for an event, join the `obs` table with the `meta` table on `event_id` and the `epoch` table on `epoch_id`.171 172**Disclaimer:** This dataset is intended for testing and development purposes and should be used with the understanding of the biases inherent in the simulation sampling.173 174## Sample Light Curves175 176Here are normalized light curves for 9 sample events from the dataset (3 from each simulation type: 1S1L, 1S2L, 2S2L), colored by filter band. Each filter's flux is divided by its median value (per event) so all bands appear on the same relative scale — the dashed gray line marks the baseline (ratio = 1). The time axis is zoomed to $t_0 \pm 5 \cdot t_E$.177 178![Sample Light Curves](lightcurves_sample.png)179 180## Code Examples181 182The dataset is hosted on Hugging Face as `RGES-PIT/MachineLearning` with three configs: `observations`, `metadata`, and `epochs`. Below are examples for loading and working with the data.183 184### Prerequisites185 186To load the metadata and epochs tables directly from Hugging Face, install the `datasets` and `pandas` libraries:187 188```bash189pip install datasets pandas pyarrow duckdb huggingface_hub190```191 192Set your Hugging Face access token:193 194```bash195export HF_TOKEN="hf_your_token_here"196```197 198### 1. Load the metadata and epochs tables199 200```python201from datasets import load_dataset202import os203 204token = os.environ.get("HF_TOKEN")205assert token, "Set HF_TOKEN environment variable first"206 207dataset_name = "RGES-PIT/MachineLearning"208 209# Load metadata and epochs tables (which fit easily in RAM)210meta_df = load_dataset(dataset_name, "metadata", split="train", token=token).to_pandas()211epochs_df = load_dataset(dataset_name, "epochs", split="train", token=token).to_pandas()212 213print(f"Metadata     : {meta_df.shape[0]:,} rows")214print(f"Epochs       : {epochs_df.shape[0]:,} rows")215```216 217**Expected output:**218```text219Observations : 18,441,855,367 rows220Metadata     : 372,655 rows221Epochs       : 98,974 rows222```223 224### 2. Reconstruct a single event's lightcurve225 226Because the observations table is extremely large (~160 GB in total), loading it in full is not recommended for local machines. Instead, we use DuckDB's remote `hf://` protocol to fetch observations for a single event without downloading the entire table:227 228```python229import duckdb230from huggingface_hub import HfFileSystem231import os232 233token = os.environ.get("HF_TOKEN")234con = duckdb.connect()235con.register_filesystem(HfFileSystem(token=token))236 237obs_p = "hf://datasets/RGES-PIT/MachineLearning/RMDC26_ML_Data_obs.parquet"238meta_p = "hf://datasets/RGES-PIT/MachineLearning/RMDC26_ML_Data_meta.parquet"239epoch_p = "hf://datasets/RGES-PIT/MachineLearning/RMDC26_ML_Data_epoch.parquet"240 241event_id = 3000242 243# Reconstruct a single event's lightcurve244event_obs = con.execute(f"""245    SELECT 246        o.event_id,247        m.name,248        e.bjd,249        o.filt,250        o.flux_uJy,251        m.tE_helio,252        m.sim_label253    FROM '{obs_p}' o254    JOIN '{epoch_p}' e ON o.epoch_id = e.epoch_id255    JOIN '{meta_p}' m ON o.event_id = m.event_id256    WHERE o.event_id = {event_id}257    ORDER BY e.bjd258""").df()259 260print(f"Event {event_id}: {event_obs['name'].iloc[0]} ({event_obs['sim_label'].iloc[0]})")261print(f"  {len(event_obs):,} observations")262print(f"  Einstein timescale tE_helio = {abs(event_obs['tE_helio'].iloc[0]):.2f} days")263```264 265**Expected output:**266```text267Event 3000: RMDC26_003000 (RMDC26_1S1L_ML)268  49,488 observations269  Einstein timescale tE_helio = 3.55 days270```271 272### 3. Filter events by metadata constraints273 274You can perform filtering on the metadata table first, and then fetch observations for matched events. To run this efficiently over the network without full-table scans, query observations only for the target event IDs:275 276```python277from datasets import load_dataset278import os279import duckdb280from huggingface_hub import HfFileSystem281 282token = os.environ.get("HF_TOKEN")283meta_df = load_dataset("RGES-PIT/MachineLearning", "metadata", split="train", token=token).to_pandas()284 285# Select low-significance single-lens events (chi2 <= 160)286good_events = meta_df[287    (meta_df["sim_label"] == "RMDC26_1S1L_ML") &288    (meta_df["ObsGroup_0_chi2"] <= 160)289]290 291print(f"{len(good_events)} events matched filters")292 293# Query observations count for matched events via DuckDB filter pushdown294con = duckdb.connect()295con.register_filesystem(HfFileSystem(token=token))296obs_p = "hf://datasets/RGES-PIT/MachineLearning/RMDC26_ML_Data_obs.parquet"297 298event_ids_str = ", ".join(map(str, good_events["event_id"].tolist()))299good_obs_count = con.execute(f"SELECT COUNT(*) FROM '{obs_p}' WHERE event_id IN ({event_ids_str})").fetchone()[0]300print(f"{good_obs_count:,} corresponding observations")301```302 303**Expected output:**304```text30559222 events matched filters3062,930,778,336 corresponding observations307```308 309### 4. Sum final_weight per simulation310 311To verify the population representation or check the total astrophysical weights of each simulation, you can sum the `final_weight` column grouped by simulation label:312 313```python314# Sum weights and count events per simulation315weights_summary = meta_df.groupby("sim_label").agg(316    total_weight=("final_weight", "sum"),317    event_count=("final_weight", "count")318).reset_index()319 320print(weights_summary.to_string(index=False))321```322 323**Expected output:**324```text325     sim_label  total_weight  event_count326RMDC26_1S1L_ML  60411.874322       299466327RMDC26_1S2L_ML   1588.290136        31633328RMDC26_2S2L_ML   1940.221485        41556329```330 331 332 333