Kashtan/Detect_Edits_in_AI-Generated_Text

1o

2�щeC�@sdd
lZGdd�de
�Zd
S)�Nc@s*eZ
dZd	dd�
Zd3dd�
Zd4dd�
ZdS)�PerplexityEvaluator�����cCs|
|_||_
||_dS�
N)�model�	tokenizer�ignore_index)�selfrrr�r	�Cd:\.Idan\תואר שני\תזה\detectLM\src\PerplexityEvaluator.py�__init__s

5
zPerplexityEvaluator.__init__NcCs|�|
|�Sr)
�log_perplexity)r�text�contextr	r	r6�__call__	s
zPerplexityEvaluator.__call__c	Cs�|jj
}|j|
d
d�}|r9|j|d
d�}tj|d|dgdd�}tjt�|d�
|j|dgdd�}td�

n|d}|}|j|�|�
|�|�
d�j	}|�7�����S)z�8        Evaluate log perplexity of text with respect to the language model9        based on the context10 11        :param text:12        :param context:13        :return:14        �pt)
�return_tensors�	input_ids�
)
�axisz:Warning, need to remove context length when reporting lppx)r�labels)
r�devicer�torch�concatenate�	ones_liker�print�to�loss�cpu�detach�numpy)	rr
rrZtext_idsZcontext_idsrrrr	r	r15rs	





�
�16

z"PerplexityEvaluator.log_perplexity)
rr)�__name__�17__module__�__qualname__rrrr	r	r	r18rs19
20r)r�objectrr	r	r	r21�<module>
s