aravagarwal/CodeCloakPII
0
1o
2 ��eS � @ s� d dl mZ d dlZd dlmZ d dlmZ d dlm Z d dl3m Z m Z m
Z
mZ d dlmZmZ d dlmZmZmZ d d lmZmZ d4d � Zd Ze�� ���Z e� d
� e� d� e� d� e� � � ej edd�Z W d � n1 ssw Y e� d� e� � � ej!e"e�#� �g d�dd�Z$W d � n1 s�w Y e� d� e�%d��� e�&d�Z'e� � � ej(ddd�Z)W d � n1 s�w Y e� d� e�&d �Z*g Z+g Z,e-e �D ]/Z.ej dd �� Z/ej d d �Z0e+�1e0� ej(d d �Z2e,�1e2� W d � n 1 �sw Y q�e3d e4e,�� e3d e4e+�� d!d"� Z5d#d$� Z6e'j7e6e e$ge)gd%� e'j7e5e e$ge+d%� d&d'� Z8e*j7e8e+e,d%� W d � n 1 �sNw Y e�%d(��k e� d)� d*d+� Z9e� � �9 e�:� � e� d,� ej d d �Z;e� d-� ej d d �Z<W d � n 1 �s�w Y ej(d d �Z=W d � n 1 �s�w Y ej&d.dd �Z>e>j7e9e$e ge<e=e;gd/� W d � n 1 �s�w Y W d � n 1 �s�w Y e j?dd0� dS )1� )�deepcopyN)�OperatorConfig)�Differ)�5difference)�6MAX_LENGTH�MAX_STRINGS_IN_SAMPLE�analyzer�7anonymizer)�generate_constant_id�generate_hash_id)� syntax_aware_replace_with_lambda� visualize_tree_sitter_matches� TREE_SITTER_QUERY_NAME_TO_QUERY)�anonymize_string�analyze_stringc C sB ddg}|D ] }t | |tt�}qdg}|D ] }t ||tt�}q|S )N�(comment) @comment� (string_content) @string_content�(identifier) @identifier)r r8 r r )� source_text�queries�query�text_to_redact�queries2� r �[/home/arava/update_feedback_data/text_pii_removal/anoninterface/anoninterface/gradiodemo.py�visualize_redact s r z�9# My name is Inigo Montoya10Luke = 111Luke += 112Jane = 113Jane += 214"""15Monty16Python17Holy18Grail19"""20real_user_string = "My name is John Smith"21 22z9## Welcome to CodeCloak, an Interactive PII Removal Tool.zjWe will first pick out the sections of code we want to anonymize using Tree-Sitter, an incremental parser.z6Please place the Python code you want to analyze here:z Source Text)�value�labelz0Please select the elements you want to identify:)�comment�string�nameT)�choicesr � multiselectz}You can either **analyze** text to see what the tool detects as PII, or **anonymize** to get a diff of what the tool changes.�AnalyzezLocate useful sectionsz23Simple Vis)r �combine_adjacentz7Here are the strings we shall run our NER tool against:z Identify PII��visibleF�inputs�outputsc C s` t | |�\}}tttdd� |D ���dd� d�}t|�add� |D �tjdd�gtt|� S ) Nc S s g | ]24}|d j �d��qS )r �utf8)�text�decode��.0�elemr r r �25<listcomp>b s z6wrap_visualize_tree_sitter_matches.<locals>.<listcomp>c S s t | �S �N)�len)�xr r r �<lambda>b s z4wrap_visualize_tree_sitter_matches.<locals>.<lambda>)�keyc S s g | ] }t j|d d��qS )Tr% )�gr�Coder, r r r r/ e s Fr% ) r
�sorted�list�setr1 �TOTAL_ITEM_COUNTr5 r6 r )�inputv�26query_list�
labelled_text� capture_list�elems_to_capturer r r �"wrap_visualize_tree_sitter_matches` s "(r@ c C s t | |�\}}|S r0 )r
)r; r< r= r> r r r �#wrap_visualize_tree_sitter_matches2g s rA )�fnr' r( c g s� � t dd� | D ��}tjdgdd�g| tjdd�gt| }tt | ��D ](}t | | �dkrCt| | �}tj|dd�||< |V q&tjdd�||< |V q&| d S )Nc S s g | ]}|d kr|�qS )� r r, r r r r/ r s z(wrap_model_inference.<locals>.<listcomp>)zNo PIINTr% Fr )r1 r5 �HighlightedTextr �ranger )�input_textboxes�num_good_boxes�iterative_boxesr. �analysisr r r �wrap_model_inferenceq s �, rJ � AnonymizezQNow, we will apply the redactions to these matches, and return the redacted text:c C sv ddg}dg}dd� }|}| D ]}t | }t||||v r tntt�}qtj|dd�tj|||�ddd�tj|dd�fS ) Nr r r c S s t � }dd� |�| |�D �S )Nc S s0 g | ]}|d d� |d dkr|d ndf�qS )� Nr � r )r- �tokenr r r r/ � s "��z9wrap_final_output.<locals>.diff_texts.<locals>.<listcomp>)r �compare)�text1�text2�dr r r �27diff_texts� s 28�z%wrap_final_output.<locals>.diff_textsTr% )r& r$ )r r r29 r r r5 r6 rD )r< �inp�queries_constant� queries_hashrS r r r r r �wrap_final_output� s 0rW zBefore:zAfter:zAnonymize and Compare)r' r( )�share)@�copyr �gradior5 �presidio_anonymizerr �difflibr � code_diffr �
anoninterfacer r r r �anoninterface.redactionr30 r �anoninterface.treeutilsr r
r � anoninterface.identificationr r r �simple_example�Blocks�demo�Markdown�Rowr6 rT �Dropdownr8 �keysr< �Tab�Button�btnrD �stageoneoutput�btn2rF �output_highlightsrE �i�lookatme�t�append�output_t�printr1 r@ rA �clickrJ rW �Column� initial_text�31final_text�32final_diff�btn3�launchr r r r �<module> s�
33343536 �3738 �39 4041 �4243 44 �� � =45464748 49 � � �� �r