Kashtan/Detect_Edits_in_AI-Generated_Text
0
1o
2 �@�e� � @ sL d dl Zd dlZd dlmZ d dlmZ d dlZdd� ZG dd� de �Z3dS )� N)� MultiTest)�tqdmc C s d� | �� d |� �S )N� )�join�split)�textZ
max_no_tokens� r �8 d:\.Idan\תואר שני\תזה\detectLM\src\DetectLM.py�truncae_to_max_no_tokens s r4 c @ s5 e Zd Z d,dd�Zd-d ed6efd d �Zd-d
efdd�Zd
efdd�Zdede fdd�Z7de de d8e fdd�Z
de de d9e fd d �Zde de d10e fd d �Zde d11efd d!�Zde d12efd"d#�Zd$e d%e d13e fd&d'�Zd.d$e d%e d14efd(d)�Zd.d$e d%e d15efd*d+�ZdS )/�DetectLM� �d �stbl�truncateFc C s@ || _ || _|| _|| _|| _|| _|dkrd| _dS d| _dS )an 16 Test for the presence of sentences of irregular origin as reflected by the17 sentence_detection_function. The test is based on the sentence detection function18 and the P-values obtained from the survival function of the detector's responses.19 20 Args:21 ----22 :sentence_detection_function: a function returning the response of the text 23 under the detector. Typically, the response is a logloss value under some language model.24 :survival_function_per_length: survival_function_per_length(l, x) is the probability of the language25 model to produce a sentence value as extreme as x or more when the sentence s is the input to26 the detector. The function is defined for every sentence length l.27 The detector can also recieve a context c, in which case the input is the pair (s, c).28 :length_limit_policy: When a sentence exceeds ``max_len``, we can:29 'truncate': truncate sentence to the maximal length :max_len30 'ignore': do not evaluate the response and P-value for this sentence31 'max_available': use the logloss function of the maximal available length32 :ignore_first_sentence: whether to ignore the first sentence in the document or not. Useful when assuming33 context of the form previous sentence.34 r TFN)� survival_function_per_length�sentence_detector�min_len�max_len�length_limit_policy�ignore_first_sentence�HC_stbl)�selfZsentence_detection_functionr r r �HC_typer r r r r �__init__
s zDetectLM.__init__N�sent�returnc C s t | �||��S �N)�floatr )r r �contextr r r �_logperp- � zDetectLM._logperp�sentencec C s | � ||�S r )r )r r! r r r r �_test_sentence0 � zDetectLM._test_sentencec C s t |�� �S r )�lenr )r r! r r r � _get_length3 r# zDetectLM._get_length�response�lengthc C s� | j |krJd}|| jkr4| jdkr | j}d| j� d�}n| jdkr*d}tjtj|fS | jdkr4d}| j}| �||�}|d ksBJ d35��t||||d �S d }t|tj||d �S )
z�36 Args:37 response: sentence logloss38 length: sentence length in tokens39 40 Returns:41 pvals: P-value of the logloss of the sentence42 comments: comment on the P-value43 �OKr z
truncated to z tokens�ignorez ignored (above maximum limit)Z
max_availablez9exceeding length limit; resorting to max-available lengthr z%Negative P-value. Something is wrong.)r& �pvaluer' �commentz ignored (below minimal length))r r r �np�nanr �dict)r r& r' r+ Zpvalr r r �_test_response6 s2 44 4546474849 ��zDetectLM._test_response� responses�lengthsc C sP g }g }t ||�D ]\}}| �||�}|�t|d �� |�|d � q ||fS )Nr* r+ )�zipr/ �appendr )r r0 r1 �pvals�commentsr& r' �rr r r �50_get_pvalsY s zDetectLM._get_pvals� sentences�contextsc C s� t |�t |�ks51J �g }g }tt||��D ]P\}}t�d|� d|� �� | �|�}| jdkr4t|| j�}|dkrHt� d|� d�� |�52t j � qz |�53| �
||�� W n ddl}|�� Y |�54|� q||fS ) zA55 Compute response and length of a text sentence 56 zTesting sentence: z | context: r � z Sentence z is too short. Skipping.r N)r$ r r2 �logging�debugr% r r57 r �warningr3 r, r- r" �pdb� set_trace) r r8 r9 r0 r1 r �ctxr' r> r r r �_get_responsesc s$ 5859 zDetectLM._get_responsesc C s> t |�t |�ks60J �| �||�\}}| �||�\}}|||fS )z@61 logloss test of every (sentence, context) pair62 )r$ rA r7 )r r8 r9 r0 r1 r4 r5 r r r � get_pvals| s 63zDetectLM.get_pvalsc C s2 t �| �|�d �}t|| jd�}|jdd�d S )Nr: �r 皙�����?��gammar )r, �arrayrB r r �hc�r r8 r4 �mtr r r �testHC� s zDetectLM.testHCc C s@ t �| �|�d �}t|� t|| jd�}ttddg|�� ��S )Nr: rC �Fnr* ) r, rG rB �printr r r. r2 �fisherrI r r r �64testFisher� s zDetectLM.testFisher� lo_chunks� lo_contextsc C s� | � ||�\}}}| jrtj|d< t�d� d|d< tj|||||d�tt |��d�}||j65� � }|j r?t�
d� d |fS t|j66| jd�|fS )Nr z Ignoring the first sentence.zignored (first sentence))r! r& r* r r+ )�indexzNo valid chunks to test.rC )rB r r, r- r; �info�pd� DataFrame�ranger$ r* �isna�emptyr= r r )r rP rQ r4 r0 r5 �dfZdf_testr r r �_test_chunked_doc� s 676869�70�71zDetectLM._test_chunked_docc C s� | � ||�\}}|d u rtj}tjtjf}tj|d< n|jdd�\}}|�� }|d |k|d< |r7|jdd� t|||d |d d�S )N�maskrD rE r* r r: )r8 �HCrN �
fisher_pvalue) rZ r, r- rT �NArH rN Z hc_dashboardr. ) r rP rQ � dashboardrJ rY rH rN Zhctr r r �test_chunked_doc� s zDetectLM.test_chunked_docc C s | j |||d�S )N)r_ )r` )r rP rQ r_ r r r �__call__� r zDetectLM.__call__)r r
r r Fr )F)�__name__�72__module__� __qualname__r �strr r r" r% �intr/ �list�tupler7 rA rB rK r. rO rZ r` ra r r r r r s 73� #74 r ) �numpyr, �pandasrT Z multitestr r r; r75 �objectr r r r r �<module> s