Team Ai
Apppublic

Kashtan/Detect_Edits_in_AI-Generated_Text

sourceHugging Facemitupdated 2y agoView on Hugging Face
0likes
rootsrc__pycache__PrepareSentenceContext.cpython-310.pyc
PrepareSentenceContext.cpython-310.pyc56 linesDownload Raw Back to __pycache__
1o

2s�e��@s@ddlZddlZddlZddlZddlmZGdd�de�ZdS)�N)�SentenceParserc@s2eZdZdZddd�Zdd�Zdd	�Zd3d�ZdS)
�PrepareSentenceContexta4    Parse text and extract length and context information5 6    This information is needed for evaluating log-perplexity of the text with respect to a language model7    and later on to test the likelihood that the sentence was sampled from the model with the relevant context.8    �spacyNcCsL|dkrtjdgd�d�|_|dkrt�d�t�|_||_||_||_dS)NrZen_core_web_sm)�taggerZattribute_rulerZ9lemmatizerZner)�disable�regexzKRegex-based parser is not good at breaking sentences like 'Dr. Stone', etc.)	r�load�nlp�logging�warningrZsentence_parser_name�context_policy�context)�selfZsentence_parserrr
�r�Fd:\.Idan\תואר שני\תזה\detectLM\src\PrepareSentenceContext.py�__init__s1011zPrepareSentenceContext.__init__cCs12|�|�S)N)�parse_sentences)r�textrrr�__call__s13zPrepareSentenceContext.__call__c14s�d}d}d}g�g�g�g������fdd�}d}��|�}|jD�]}|}	t�||j�}15t�||j�}t�d|j�}t�d|j���t�d|���t�d	|16���t�d17|���t|18�dkr�t|�dkr�t�d�|	dur�t|d�|kr�t�d
�||ddt|�d�d}|	dkr�t|19d�|kr�t�dt�����||20d|	t|�d�d}|	dkr�t�d�d}t|d�|kr�||ddt|�d�q"t|�dkr�d}|	dus�Jdt������t|d�|kr�|d}
||
|t|�d�q"t|21�dk�r,d}|	dk�sJdt������t|22d�|k�r)||23d|	t|�d�d}q"||j|t|��q"����t	�24ddt���d�S)Nz(.*?)</edit>z<edit>(.*?)�csr��|���|���|��jdur�j}n�jdur d}n�jdkr0t��dkr0�d}nd}��|�dS)N�previous_sentencer�����)�appendr
r�len)�	sent_text�tag�sent_lengthr
��contexts�lengthsr�tags�textsrr�update_sent(s252627282930z;PrepareSentenceContext.parse_sentences.<locals>.update_sentz<edit>(.*?)</edit>zCurrent sentence: zMatches open: zMatches close: zMatches between: rz8Found an opening and a closing tag in the same sentence.z7Openning followed by closing with some text in between.z<edit>�z>Wierd case: closing/openning followed by openning in sentence �z</edit>zClosing followed by openning.z<Found an opening tag without a closing tag in sentence num. z<Found a closing tag without an opening tag in sentence num. �r�lengthr
rZ
number_in_par)r	�sents�re�findallrr31�debugrr�np�arange)rrZ
pattern_closeZpattern_openZ
MIN_TOKEN_LENr"�curr_tag�parsed�sZprev_tagZ
matches_closeZmatches_openZmatches_betweenrrrrrsh32333435���z&PrepareSentenceContext.parse_sentencescCs�g}g}g}g}g}d}t�dd|�}|�|�}d}	d}36t|j�D]�\}}t�dt|��}
t|
�dkrBt�	d|�d|
���t37d�t|
�dkrz|
d}|dd�d	krg|38durdt�d39|�d|���q!d}40q!|41durwt�d|�d|���q!|}42q!t|�}t|�}|�|�|	d7}	|�|	�|�|43�|�|�|j
dur�|j
}n|jdur�d}n|jdkr�|}|}nd}|�|�q!|||||d
�S)Nz(</?[a-zA-Z0-9 ]+>\.?)\s+z\1.\nrz(</?[a-zA-Z0-9 ]+>)r$zMore than one tag in sentence z: r#z</z/Closing tag without an opening tag in sentence z.Opening tag without a closing tag in sentence rr%)r(�subr	�	enumerater'r)�strrr44�error�exitrrr
r)rrr!rrr Z45num_in_par�previousr.Zrunning_sent_numr-�i�sentZall_tagsrrrr
rrr�REMOVE_parse_sentencesfsV4647484950515253�z-PrepareSentenceContext.REMOVE_parse_sentences)rNN)�__name__�54__module__�__qualname__�__doc__rrrr8rrrrrs55Hr)	r56rr(�numpyr+Zsrc.SentenceParserr�objectrrrrr�<module>s