onconpc/onconpc-visualization
2
1U
2 2f�u �
@ s6 d dl Z d dlmZmZmZmZmZ d dlZd dlZd dl Z d dl3m Z d dl m
Z
mZ d dlmZ d dlZd dlZd dlZd dl mZ d dl mZ d dlmZ d dlZ d dl m Z d dl4m Z e j j ej!ee" ej!d�d d5�Z#e j$j%ej!ee" ej!d�d d �Z&e j j ej!ej'd
�dd�Z(e j$j%ej!ej'd
�dd�Z)ee" d�dd�Z*e"ee"ee" f e"d�dd�Z+dFej!ej!e,e,e"e"e"e"e-e.d �6d d �Z/dGej!ej0ee" ee" e-ee"ee"ef f d �d!d"�Z1dHee" ej!ej2ej!ee" e"e"e.eee"ef d#� d$d%�Z3dIej!ej!ej!ej!e"e"e"eej!ej!f d)�d*d+�Z4dJej!e"e"e"e"e"ee" ej!d-�d.d/�Z5dKej!e"ej!d1�d2d3�Z6dLej!ej!ej!ej!eee" e"e"eej!ej!f d6�d7d8�Z7dMej!ej!ej!ej!ee" eee" e"eej!ej!ej!f d:�d;d<�Z8ej!ee" ej!d=�d>d?�Z9ej!ee" ej!d@�dAdB�Z:ee" ee" dC�dDdE�Z;dS )N� N)�List�Mapping�Optional�Any�Tuple��Patch)�7conversion�default_converter)� pandas2ri)�localconverter)� adjust_text)� xgb_model�features_test_df� cancer_types�returnc s` | � |j�}|jdd�}dd� t||�D �}tj||j� d�}||d< � fdd�|D �|d< |S ) � Returns cancer type predictions for test set using XGBoost model.8 9 Args:10 xgb_model: XGBoost model.11 features_test_df: Test set features.12 cancer_types: List of cancer types.13 Returns:14 pd.DataFrame containing cancer type predictions and prediction probabilities.15 � ��axisc S s g | ]\}}|| �qS � r ��.0Z pred_dist�max_idxr r �:/Users/jenniferzhou/onconpc-visualization/./codes/utils.py�16<listcomp>' s z1get_xgboost_cancer_type_preds.<locals>.<listcomp>��index�columns�
max_posteriorc s g | ] }� | �qS r r �r r �r r r r - s � cancer_type)Z
predict_proba�values�argmax�zip�pd� DataFramer )r r r �ckp_test_pred_probs�ckp_test_preds�max_posteriors�ckp_test_preds_dfr r! r � get_xgboost_cancer_type_preds s ��r, c s� t �|j�}| j|dd�}t�|�}||jddd� }|jdd�}dd� t||�D �}t j17||j � d�}||d < � fd18d�|D �|d < |S ) r T)Z
output_marginr )r Zkeepdimsr c S s g | ]\}}|| �qS r r r r r r r B s z8get_xgboost_latest_cancer_type_preds.<locals>.<listcomp>r r c s g | ] }� | �qS r r r r! r r r H s r" ) �xgbZDMatrixr# Zpredict�npZexp�sumr$ r% r& r' r )r r r Zdtestr( r) r* r+ r r! r �$get_xgboost_latest_cancer_type_preds0 s 19 ��r0 )�model�datar c s@ | � � }|�� dd� � d� fdd� }||_t�|�}|�|�S )��Returns SHAP values for predictions based on data.20 21 Args:22 model: XGBoost model.23 data: Data to obtain SHAP values for.24 Returns:25 Numpy array containing SHAP values.26 � Nc s � S �Nr )�self�Zmodel_bytearrayr r � in_bytearrayX s z(obtain_shap_values.<locals>.in_bytearray)N)Z get_boosterZsave_raw�shap�
TreeExplainer� shap_values)r1 r2 Z myboosterr8 �shap_exr r7 r �obtain_shap_valuesK s 27r= c C s t �| �}|�|�S )r3 )r9 r: r; )r1 r2 r< r r r �&obtain_shap_values_with_latest_xgboost^ s 28r> )� fature_namesc C s� t �t�}i }| D ]z}d|kr6|d �|� d||< q|dkrV|d �|� d||< qd|krv|d �|� d||< q|d �|� d||< q||fS )z�Partitions feature names into groups.29 30 Args:31 feature_names: List of feature names.32 Returns:33 Dictionary mapping feature groups to feature names and vice versa.34 �SBS� signature)�Age�Sex�clinical�CNA�cna�mutation)� collections� defaultdict�list�append)r? � feature_group_to_features_dict� feature_to_feature_group_dict�featr r r � partition_feature_names_by_groupl s 3536373839rO )� feature_name�feature_group_dictr c C s8 ddddd�}|� � D ] \}}| |kr|| S qdS )a' 40 Determines the color for a given feature based on its group.41 42 Args:43 feature_name: Name of the feature.44 feature_group_dict: Dictionary mapping feature groups to feature names.45 46 Returns:47 A string representing the color associated with the feature's group.48 �#984ea3�#f781bf�#4daf4a�#377eb8)rG rF rA rD Zblack)�items)rP rQ Z
color_mapping�group�featuresr r r � get_color� s49 rY � ./others_prediction_explanation�$./data/combined_cohort_age_stats.pkl�50 F)51�shap_pred_sample_df�feature_sample_dfrL rM � sample_info�filename�filepath�cohort_age_stats_path�top_feature_num� save_plotc52 sv t j�ddd�� t �� \}53} | jd �d� | jd �d� t� �� g �} t| �fdd�d d54�}
t|
d |� �fd d�dd55�}t �56|�j | |j | � fd
d�|D �d��}| j |d |d |d d� | �
d� | �|� | �g � t|d��}t�|�}W 5 Q R X | �� d d d }tt|d |d ��D ]�\}\}}|| dk�rVt|�� n|d�}|d k�rz|d k�rvd nd }|d k�r�t||d! |d" �}|d#d � d$k�r�t|�� }| j|||� d%|� �dd&d'd(� �q2| j||d)dd&d'd(� d*d� td+d,d-d.gd/d0d1d2g�D �}| j|d3d4� t �� | �r^t �|� d5|� d6�� t �|� d5|� d7�� t �� |� d5|� d6�S )8a( 57 Dynamic version of the function for individual prediction interpretation for a given tumor sample.58 59 Args:60 shap_pred_sample_df: DataFrame containing SHAP values for a given tumor sample.61 feature_sample_df: DataFrame containing feature values for a given tumor sample.62 feature_group_to_features_dict: Dictionary mapping feature groups to feature names.63 sample_info: Sample information to be displayed.64 filename: Filename to save the figure.65 top_feature_num: Number of top features to display.66 � ZArial)z font.sizez font.family�rightF�topc s t � j| �S r5 )�abs�loc��x�r] r r �<lambda>� � z4get_individual_pred_interpretation.<locals>.<lambda>T)�key�reverseNc s67 � j | S r5 )ri rj rl r r rm � rn c s g | ]}t |� ��qS r )rY )r rN )rL r r r � s z6get_individual_pred_interpretation.<locals>.<listcomp>)� feat_name�SHAP_val�feat_val�colorrq rr rt )rt z SHAP Values�rbr g�������?皙�����?rs rG z.2frC � �?�Male�FemalerB �Std_mean�Age_mean�����rE z: �centerr\ )ZhaZvaZfontsizezfeature: valuec S s g | ]\}}t ||d ��qS ))Z facecolor�labelr )r r~ rt r r r r � s z Somatic Mut.z68CNA eventsz
Mutation Sig.zAge/SexrR rS rT rU zFeature Groups)Zhandles�title�/z.svgz.pdf) �pltZrcParams�updateZsubplotsZspinesZ set_visibler/ r# �sortedr& r' ri ZbarhZ69set_xlabelZ set_titleZ70set_yticks�open�pickle�loadZget_xlim� enumerater% �int�textZlegendZ tight_layoutZsavefigZshow)r] r^ rL rM r_ r` ra rb rc rd ZfigZaxZ all_featuresZsorted_featuresZ top_featuresZ top_feats_df�fp�combined_cohort_age_statsZ left_margin�i�name�valueZ71value_textZlegend_elementsr )rL r] r �"get_individual_pred_interpretation� sL � 727374 " 757677$$r� � )�preds_df�shapsrX r �nr c C s� | | }i }|� � D ]�\}}|�|�jdd�} | j�|�}78g } | jD ]4} |�| �}
tj||
|79 || gd�j}| �|� qDt� | �}| �80� |d�||< q|S )aX 81 Get top n predictions for each sample in the given DataFrame.82 Args:83 preds_df: DataFrame containing predictions.84 shaps: Array of SHAP values.85 features: List of features.86 n: Number of top predictions to return.87 cancer_types: List of cancer types.88 Returns:89 Dictionary containing top n predictions and SHAP values for each sample.90 F)Z ascendingr )Ztop_n_predictionsZshap_values_df) Ziterrows�nlargestZ sort_valuesr Zget_locr& r' �TrK �concatZto_dict)r� r� rX r r� Z df_chosenZ$sample_id_to_top_n_pred_and_shaps_df� sample_idZrowZ sorted_top_3Zrow_idxZshap_dfs� pred_cancer�pred_cancer_idx�shap_pred_cancer_dfZ
shap_total_dfr r r �get_top_n_pred_and_shap� s$ 9192� 93� r� ) � query_idsr� r� �df_features_genie�cancer_types_to_considerra rb rd r c
C s� i }| D ]�} |j | df }94|j | df } |�| �} |j| }
tj|| |j|jd�}|j| }d| � d| � d|95d��}t|j�\}}t||
|||t| �|d||d �96}|97| |d98�|| < q|S ) a9 99 Get OncoNPC predictions and generate SHAP-based explanation plots for multiple query IDs.100 101 Args:102 query_ids: List of IDs of the tumor samples to query.103 preds_df: DataFrame containing predictions.104 shaps: Array of SHAP values.105 df_features_genie: DataFrame containing features.106 cancer_types_to_consider: List of cancer types considered in the prediction.107 filepath: Path to save the explanation plots.108 save_plot: Whether to save the explanation plots.109 Returns:110 List of dictionaries containing prediction details and explanation plots for each query ID.111 r r" r z SAMPLE_ID: z
112Prediction: z113Prediction probability: z.3fr\ )r_ r` ra rc rb rd )� pred_probr� Zexplanation_plot) �atr ri r& r' r rO r� �str)r� r� r� r� r� ra rb rd Z results_dictZquery_idr� r� r� r^ r� r] r_ rL rM Z
full_filenamer r r �#get_onconpc_prediction_explanations s: 114115116�117� � r� �data/features_onconpc.pkl�"data/combined_cohort_age_stats.pkl�)data/mutation_signatures/sigProfiler*.csv)�df_patients_chosen�df_samples_chosen�df_mutations_chosen�
df_cna_chosen�features_onconpc_path� combined_cohort_age_stats_path�mut_sig_weights_filepathr c C s� |� dg�j�� }tj| |ddd�}t|ddddd d118d �} t| |d �}119t|||120|�\} } t|d
��}
t �121|
�}W 5 Q R X t | |�}t |j
�|_
t|d
��}
t �122|
�}W 5 Q R X |d |d |d |d< || fS )a\ 123 Process raw GENIE data for OncoNPC prediction inference.124 125 Args:126 df_patients_chosen: DataFrame containing chosen patient data.127 df_samples_chosen: DataFrame containing chosen sample data.128 df_mutations_chosen: DataFrame containing chosen mutation data.129 df_cna_chosen: DataFrame containing chosen copy number alteration data.130 features_onconpc_path: Path to OncoNPC features file.131 combined_cohort_age_stats_path: Path to combined cohort age statistics file.132 weights_filepath: Path to mutation signatures weights file.133 134 Returns:135 Processed DataFrame ready for OncoNPC prediction inference.136 � Hugo_Symbolrf Z137PATIENT_ID)�howZon�Tumor_Sample_BarcodeZ138ChromosomeZStart_PositionZReference_AlleleZTumor_Seq_Allele2�genie)�
sample_id_col�chromosome_col�
start_pos_col�ref_allele_col�alt_allele_col�config)�weights_filepathru rB r{ rz )� set_indexr� �copyr& �merge�get_snv_in_trinuc_context�obtain_mutation_signatures�pre_process_features_genier� r� r� �zero_pad_missing_features�standardize_feat_namesr )r� r� r� r� r� r� r� Zdf_cna_reshapedZdf_patients_mergedZdf_trinuc_feats_genieZdf_mut_sigs_genier� Zdf_labels_genier� Zfeatures_onconpcZdf_features_genie_finalr� r r r �"get_onconpc_features_from_raw_dataI s0 � � 139 r� r� )� df_mutationsr� r� r� r� r� r� r c
C s |||||g}| | � � }|dkr8d|| �t� ||< ||| j�d� }|| �d�||< d} t�t��8 ttjtj �� tj�140|�}141W 5 Q R X t� | �} W 5 Q R X t�t��< ttjtj �� tj� | |142|||||��} W 5 Q R X W 5 Q R X |dk�r| j
�d�| _
| S )a� 143 Processes the given mutations DataFrame to get SNV in trinucleotide context.144 145 Args:146 df_mutations: DataFrame containing mutation data.147 sample_id_col: Column name for sample IDs.148 chromosome_col: Column name for chromosome.149 start_pos_col: Column name for start position.150 ref_allele_col: Column name for reference allele.151 alt_allele_col: Column name for alternative allele.152 Returns:153 DataFrame with SNV in trinucleotide context.154 r� �chrzGL|chrMTr� a~ 155 function(mutationData, sample_id, chr, pos, ref, alt) {156 library(deconstructSigs)157 sigs.input <- mut.to.sigs.input(mut.ref = mutationData, 158 sample.id = sample_id,159 chr = chr, 160 pos = pos, 161 ref = ref, 162 alt = alt)163 164 # Filter samples with low mutations165 sigs.input <- sigs.input[rowSums(sigs.input) >= 1,]166 return(sigs.input)167 }168 Z profile_dfci�float)r� Zastyper� �containsr r r169 �robjectsr Z converterZpy2rpy�rZrpy2pyr )
r� r� r� r� r� r� r� Zcolumns_chosenZdf_mutations_chosen_trinucZr_codeZr_df_mutationsZ170r_function�df_trinuc_featsr r r r� � s. 171�172r� �+./data/mutation_signatures/sigProfiler*.csv)r� r� r c C s� t � |�}t|�dkr td��dd� |D �}i }tt|��D �]}t�|| �}|jdd� g }t|jj |j173j �D ].\}} | d d | d | d }174|� |175� qr||d176< |j d177dd� d �
|j�} d | kr�||| d
j |d< n*d| k�r||| d j |d< ntd��t�|jj �} t| d �dk�r6td��|||| < q<tjg | jd�}
|�� D ]\\}}tt|j�t| j�@ �}t|�dk�r�td��t�| | j |j| jj �}||
|< �q\|
S )z�Transforms tri-nucleotide features into mutation signature based features.178 179 Args:180 df_trinuc_feats: DataFrame containing tri-nucleotide features.181 Returns:182 mut_sig_based_df: DataFrame containing mutation signature based features.183 r z No mutation signatures data.c S s" g | ]}|� d �d dd� �qS )�_�����N�����)�split)r � file_namer r r r � s z.obtain_mutation_signatures.<locals>.<listcomp>T��inplace�[�]� ZType_new� ZGRCh38Z_GRCh38� values_oiZGRCh37Z_GRCh37z$No human-based experimental results.rw rv zValues do not sum up to 1.�r �` z6Number of substitution features does not sum up to 96.)�glob�len�184ValueError�ranger& Zread_csvZdropnar% ZTyper# ZSubtyperK r� �joinr r. r/ r� rh r' r rV rJ �set�matmulri )r� r� Z185file_namesZfile_names_keyZsbs_conversion_df_dictr� �dfZ
new_name_listZchangeZsubtype�new_nameZmerged_colum_namesZsum_valsZmut_sig_based_dfro Zmut_dfZcommon_substitute_featsZ mut_sig_valsr r r r� � sD 186 187 � r� � SAMPLE_ID� CANCER_TYPE)r� �df_cna�df_mutation_signatures� df_patientsr � id_column�cancer_type_columnr c s� || j }| j| j�|� }t�|jj �} tjd| |d�}188|ddg j D ]:} | d } | d }
|
|189j krH|190j191| |
f d |192j193| |
f< qHdd� | D �|194_ |jt t |j
�t |�@ � }|jddd � |�d195�}|d d g �� }g }g }g }t|j
|j �D ]�\} \}}|d
k�r|�d� n |dk�r*|�d� n196|�d� t�|��rV|�| � |�d� q�dt|�k�r||�t|dd� �� q�dt|�k�r�|�t|dd� �� q�|�t|�� q�||d < ||d < ddg|_ t t |197j
�t |j
�@ t |j
�@ t |j
�@ �}|198|||g}t||�}|jt t |j
�t |� � }|j|j
| }� dk �rztj� fdd�|j D �dg|j
d�}|j |d< ||fS ||fS dS )a Pre-process genetics data to create feature df for GENIE.199 200 Args:201 df_mutations: DataFrame containing mutation data.202 df_cna: DataFrame containing CNA data.203 df_mutation_signatures: DataFrame containing mutation signature data.204 df_patients: DataFrame containing patient data.205 cancer_types: List of cancer types.206 id_column: Column name for sample IDs.207 cancer_type_column: Column name for cancer types.208 Returns:209 df_features_merged_final: DataFrame containing merged features.210 df_labels_final: DataFrame containing labels.211 r �r r r� r� r c S s g | ] }|d �qS ��_mutr �r �gener r r r s z.pre_process_features_genie.<locals>.<listcomp>Tr� r� ZSEXZAGE_AT_SEQ_REPORTrx ry r� z>89Nz<18rC rB c s g | ]}� � |��qS r r� �r �valr! r r r A s � cancer_labelr" )r# ri r� �isinr. �uniquer� r& r' r r� rJ r� r �fillnar� r� r% rK Zisnullr� r� � merge_dfs) r� r� r� r� r r� r� �212sample_idsr� �mutation_gene_names�df_mutation_feature�entryr� r� r� �df_patients_id_indexed�df_sex_age_feature�sex_listZage_nan_indicesZage_list�sex�age�213common_ids�dfs_list�df_features_mergedZdf_features_merged_final� df_labels�df_labels_finalr r! r r� � sb 214215 216217 218 219 220 221 � 222 223 �224r� �UNIQUE_SAMPLE_ID)r� r� r� r� r �cup_samples_idsr� r c sf || j }| j| | �|� }t�|jj �} tjd| |d�}225|ddg j D ]:} | d } | d }
|
|226jkrJ|227j | |
f d |228j | |
f< qJdd� | D �|229_|jt230t |j �t |�@ � }|j
ddd � |�|�}|d231d g �� }g }t|j |j �D ]B\} \}}|d k�r|�d� q�|d
k�r$|�d� q�|�d� q�||d232< d233d g|_t234t |235j �t |j �@ t |j �@ t |j �@ �}|236|||g}t||�}i }d|d< d|d< d|d< d|d< d|d< d|d< d|d < d |d < d |d < d!|d"< |j|dd#� |jt237t |j �t |� � }|jt238t |j �t |�@ � }|j|j d$ }tj� fd%d�|j D �d&g|j d�}|j |d'< |||fS )(a� Pre-process genetics data to create feature df for DFCI data.239 240 Args:241 df_mutations: DataFrame containing mutation data.242 df_cna: DataFrame containing CNA data.243 df_mutation_signatures: DataFrame containing mutation signature data.244 df_patients: DataFrame containing patient data.245 cancer_types: List of cancer types.246 cup_samples_ids: List of CUP sample IDs.247 id_column: Column name for sample IDs.248 Returns:249 df_features_merged_ckps: DataFrame containing merged features for CKP samples.250 df_labels_final: DataFrame containing labels for CKP samples.251 df_features_merged_cups: DataFrame containing merged features for CUP samples.252 r r� r �CANONICAL_GENEr c S s g | ] }|d �qS r� r r� r r r r j s z-pre_process_features_dfci.<locals>.<listcomp>Tr� rC rB ZMALEZFEMALEr� ZFAAP100ZC17ORF70Z FAAP100_mutZ C17orf70_mutZFAAP24ZC19ORF40Z253FAAP24_mutZ C19orf40_mutZ BMS1P20_mutZ LOC96610_mutzMEF2BNB-MEF2B_mutzLOC729991-MEF2B_mutZ254FAAP20_mutZ C1orf86_mutZFAAP20ZC1ORF86Z RACK1_mutZ255GNB2L1_mutZRACK1ZGNB2L1)r r� r� c s g | ]}� � |��qS r r� r� r! r r r � s r� r" )r# ri r� r. r� r r& r' r r� rJ r� r r� r� r� r% rK r� �rename) r� r� r� r� r r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� Z profile_old_new_gene_mappingZdf_features_merged_ckpsZdf_features_merged_cupsr� r r r! r �pre_process_features_dfciH sr 256 257 258259 260 261�� 262��� �263r )r� � feature_listr c s8 � fdd�|D �}t j� t j|� jd�gdd�� � | S )z�264 Zero pad missing features.265 Args:266 df: DataFrame containing features.267 feature_list: List of features to include.268 Returns:269 df: DataFrame containing zero-padded features.270 c s& i | ] }|� j kr|d gt� � �qS )r )r r� )r Zfeature�r� r r �271<dictcomp>� s 272 z-zero_pad_missing_features.<locals>.<dictcomp>r� r r )r& r� r' r )r� r Z new_columnsr r r r� � s r� )�df_list�273ids_commonr c C sD t | �D ]6\}}|dkr$|j| }qtj||j| dddd�}q|S )z�Merges a list of dataframes using common sample IDs.274 275 Args:276 df_list: List of dataframes to merge.277 ids_common: Common sample IDs.278 Returns:279 df_merged_to_return: Merged dataframe.280 r �leftT)r� Z281left_indexZ right_index)r� ri r& r� )r r282 Zidxr� Zdf_merged_to_returnr r r r� � s283 r� )�curr_feat_namesr c C s� g }| D ]z}d|kr(|� |�dd�� qd|ks8d|krD|� d� qd|ksTd|kr`|� d� qd|krt|� |� q|� |d � q|S ) z�284 Standardize feature names.285 Args:286 curr_feat_names: list of feature names287 Returns:288 new_feat_names: list of standardized feature names289 r� r� ZAGErB ZGENDERrC r@ z CNA)rK �replace)r Znew_feat_namesrN r r r r� � s r� )NNrZ r[ r\ F)r� )rZ r[ F)r� r� r� )r� )r� )Nr� r� )Nr )<r� �typingr r r r r rH �osr� Zmatplotlib.patchesr Z
rpy2.robjectsr r290 Zmatplotlib.pyplotZpyplotr� Znumpyr. Zpandasr& r9 r� r Zrpy2.robjects.conversionr Zxgboostr- Z291adjustTextr
ZsklearnZ
XGBClassifierr' r� r, �coreZBoosterr0 Zndarrayr= r> rO rY �dictr� �boolr� Zasarrayr� Zarrayr� r� r� r� r� r r� r� r� r r r r �<module> s � � � � �� T � � . �� > �� > � � E �� 7 �292 � U �293� Q �