PerturbReason/PerturbReason_dataset_code

1# Qualitative Analysis Toolkit2 3Current pipeline assumptions:4- Three models: `Qwen Base`, `SFT (Ours)`, `GRPO (Ours)`5- Two contexts: `noisy` and `hidden`6- Every analysis keeps `noisy` and `hidden` separate7- Presentation renames `other_test` to `transfer_task`8- Group-level performance uses balanced accuracy, and edge-F1 plots/tables use strict edge F19- All inputs are resolved from the local repository root10 11---12 13## Quick Start14 15From the repository root:16 17```bash18# Run the full pipeline19bash qualitative/run_all_0330.sh20 21# Or run individual phases22bash qualitative/run_all_0330.sh phase123bash qualitative/run_all_0330.sh phase224bash qualitative/run_all_0330.sh phase325bash qualitative/run_all_0330.sh phase426bash qualitative/run_all_0330.sh phase527```28Outputs are written to `qualitative_output_local/` by default.29Override with:30 31```bash32QUAL_OUT_DIR=/custom/output/path bash qualitative/run_all_0330.sh33```34 35---36 37## Input Layout38 39All paths below are relative to the repository root.40 41### Evaluation Results42 43| Model | Noisy eval dir | Hidden eval dir |44|------|------|------|45| Qwen Base | `eval_result_for_qualitative/Qwen_base_noisy/` | `eval_result_for_qualitative/Qwen_base_hidden/` |46| SFT (Ours) | `eval_result_for_qualitative/SFT_noisy/` | `eval_result_for_qualitative/SFT_hidden/` |47| GRPO (Ours) | `eval_result_for_qualitative/RL_GRPO_noisy/` | `eval_result_for_qualitative/RL_GRPO_hidden/` |48 49Each eval directory is expected to contain exactly one file matching each pattern:50- `samples_*.csv`51- `summary_*.json`52- `file_summary_*.csv`53 54### Prediction Results55 56| Model | Noisy predictions | Hidden predictions |57|------|------|------|58| Qwen Base | `prediction_result_for_qualitative/Qwen base/noisy_context_output_0426/` | `prediction_result_for_qualitative/Qwen base/hidden_context_output_0426/` |59| SFT (Ours) | `prediction_result_for_qualitative/SFT/noisy/` | `prediction_result_for_qualitative/SFT/hidden/` |60| GRPO (Ours) | `prediction_result_for_qualitative/GRPO/noisy_context/` | `prediction_result_for_qualitative/GRPO/hidden_context/` |61 62### Shared Inputs63 64| Input | Path |65|------|------|66| GT noisy | `prediction_result_for_qualitative/gt/noisy_context/` |67| GT hidden | `prediction_result_for_qualitative/gt/hidden_context/` |68| OmniPath KG | `eval_v3/omnipath_gene_graph_allinteraction_1229.pkl` |69 70---71 72## Output Structure73 74```text75qualitative_output_local/76|77|-- qwen_base/78|   |-- error_taxonomy/79|   |-- generalization/80|   |-- chem_vs_genetic/81|   |-- pathway_complexity/82|   |-- parallel_pathways/83|   |   |-- noisy/84|   |   `-- hidden/85|   |-- hallucination/86|   |   |-- noisy/87|   |   `-- hidden/88|   |-- chain_depth/89|   `-- case_studies/90|91|-- sft/92|-- grpo/93|-- cross_model/94`-- cross_model_enhanced/95```96 97---98 99## Phase Overview100 101### Phase 1: Broad Failure Taxonomy102 103Runs independently for `Qwen Base`, `SFT`, and `GRPO`.104 105Outputs per model:106- `error_taxonomy/`107- `generalization/`108- `chem_vs_genetic/`109 110Key reporting conventions:111- `generalization/` uses `balanced_accuracy` and `edge_f1_strict_mean`112- Transfer results are shown as `transfer_task`113- The `chemical2genetic` transfer task is excluded, so each condition now contributes 10 file-summary rows instead of 11114 115Primary scripts:116- `error_taxonomy.py`117- `generalization_heatmap.py`118- `chem_vs_genetic.py`119 120### Phase 2: Pathway-Level Analysis121 122Runs independently for each model.123 124Context handling:125- `pathway_complexity/` still compares `noisy` and `hidden` together126- `chain_depth/` still compares `noisy` and `hidden` together127- `parallel_pathways/` now writes separate outputs to `noisy/` and `hidden/`128- `hallucination/` now writes separate outputs to `noisy/` and `hidden/`129 130Primary scripts:131- `pathway_complexity.py`132- `parallel_pathways.py`133- `hallucination_patterns.py`134- `chain_depth.py`135 136### Phase 3: Case Studies137 138Runs independently for each model.139 140Outputs per model:141- `case_studies/noisy_path/`142- `case_studies/correct_exemplar/`143- `case_studies/noisy_vs_hidden/`144- `case_studies/case_study_summary.md`145 146Notes:147- Sample matching now uses stable string sample IDs instead of row indices148- Prediction files without explicit `id` fields are matched via prompt metadata149- Single-case markdown now records condition, presented split, raw split, sample ID, test file, cell type, perturbation, and effect gene150- Noisy-vs-hidden pair markdown now records the same case metadata for both contexts151 152Primary script:153- `case_study_select.py`154 155### Phase 4: Standard Cross-Model Comparison156 157Compares all three models for each condition.158 159Outputs:160- `balanced_accuracy_comparison_noisy.md`161- `balanced_accuracy_comparison_hidden.md`162- `error_comparison_noisy.png`163- `error_comparison_hidden.png`164- `error_by_split_noisy.png`165- `error_by_split_hidden.png`166- `error_shift_noisy.md`167- `error_shift_hidden.md`168- `metric_comparison_noisy.png`169- `metric_comparison_hidden.png`170- `samecase_noisy.md`171- `samecase_hidden.md`172- `verdict_comparison_noisy.png`173- `verdict_comparison_hidden.png`174 175Primary script:176- `cross_model_comparison.py`177 178Notes:179- The comparison tables now report balanced accuracy rather than raw accuracy180- Metric plots use strict edge F1 and a corrected path-connected rate181 182### Phase 5: Enhanced Cross-Model Comparison183 184The enhanced phase is now ordered-model aware and supports `Qwen Base -> SFT -> GRPO` directly.185 186Outputs:187- `condition_robustness.md`188- `pairwise_metric_deltas.md`189- `per_class_recall.md`190- `chain_depth_summary.md`191- `mcnemar_noisy.md`192- `mcnemar_hidden.md`193- `win_loss_tie_noisy.md`194- `win_loss_tie_hidden.md`195- `label_distribution_noisy.png`196- `label_distribution_hidden.png`197- `win_loss_tie_noisy.png`198- `win_loss_tie_hidden.png`199- `cross_model_chain_depth_noisy.png`200- `cross_model_chain_depth_hidden.png`201- `split_delta_balanced_accuracy_noisy.png`202- `split_delta_balanced_accuracy_hidden.png`203- `split_delta_edge_f1_strict_mean_noisy.png`204- `split_delta_edge_f1_strict_mean_hidden.png`205 206Primary script:207- `cross_model_enhanced.py`208 209---210 211## Script Reference212 213| Script | Role |214|------|------|215| `utils.py` | Shared loading, parsing, plotting, stable sample-key helpers |216| `error_taxonomy.py` | Error-label and rescue-verdict analysis |217| `generalization_heatmap.py` | Split x pert_type heatmaps |218| `chem_vs_genetic.py` | Chemical vs genetic comparison |219| `pathway_complexity.py` | Prompt complexity vs performance |220| `parallel_pathways.py` | Multi-path and sign-conflict analysis |221| `hallucination_patterns.py` | KG-based hallucination analysis |222| `chain_depth.py` | GT vs model chain-length analysis |223| `case_study_select.py` | Representative case selection and pathway rendering |224| `cross_model_comparison.py` | Standard three-model comparison |225| `cross_model_enhanced.py` | Enhanced ordered-model comparison |226| `run_all_0330.sh` | Local pipeline entrypoint |227 228---229 230## Practical Notes231 232- `run_all_0330.sh` auto-detects the repository root from its own location.233- `~/anancond3/bin/python` is checked first for compatibility with earlier instructions, but this workspace currently resolves successfully with `~/anaconda3/bin/python`.234- The pipeline now resolves eval artifacts dynamically from `samples_*.csv`, `summary_*.json`, and `file_summary_*.csv` patterns instead of hard-coding timestamps.235- Sample and file-summary loaders exclude the `chemical2genetic` transfer file globally, so counts across phases reflect that filtered evaluation set.236- `parallel_pathways/hidden/` may legitimately show zero prompt paths, because the hidden-context prompts do not provide retrieved knowledge.237- Hallucination analysis still uses a gene-only OmniPath KG, so drug-target relations can be flagged as out-of-KG even when biologically reasonable.238- If you replace any eval or prediction snapshot, keep the directory names stable or update `run_all_0330.sh` accordingly.239