Kashtan/Detect_Edits_in_AI-Generated_Text
0
1U
2 V��e& � @ s, d dl Zd dlZd dlZG dd� de�ZdS )� Nc @ s. e Zd ZdZd
dd�Zdd d3�Zdd d �ZdS )�PrepareArticlesa 4 Parse preprocessed data from csv5 6 This information is needed for evaluating log-perplexity of the text with respect to a language model7 and later on to test the likelihood that the sentence was sampled from the model with the relevant context.8 F�9 �d Nc C s" || _ || _|| _|| _|| _d S �N)�10input_file�11min_tokens�12max_tokens� get_edits�
max_sentences)�selfr r r r r13 � r �4/home/dsvmu/detectLM/detectLM/src/PrepareArticles.py�__init__ s14 zPrepareArticles.__init__Tc C s15 | � |�S r )�
parse_dataset)r �combinedr r r
�__call__ s zPrepareArticles.__call__c
C s� t �| j�}g }g }g }g }t|jd �D �]}t�|j| d �}g } g }16g } g } |d D �]t}
|
d D �]d}t|d � � �}|| j17kr�|| j kr�| � |d � |18� t|d � � �� | � d|kr�|d nd � | � d� | j
�rhd|k�rht|d � � �| j19k�rht|d � � �| j k�rh| � |d � |20� t|d � � �� | � d |k�rX|d nd � | � d21� | jrjt| �| jkrjtj|td �tj|td �tj|td �tj|td �t�d d t|� �d
� S qjq\|�r�|| }||22 }|| }|| }q*|� t�| �� |� t�|23�� |� t�| �� |� t�| �� q*tj|td �tj|td �tj|td �tj|td �t�d d t|� �d
�S )Nr Z article_jsonZ24sub_titles� sentences�sentence�contextzno edits� alternativeZalternative_contextz<edit>)�dtype� )�text�lengthr �tag�
number_in_par)�pd�read_csvr �range�shape�json�loads�iloc�len�splitr r �appendr r25 �np�array�object�arange)r r Zarticles_dataset�texts�lengths�contexts�tags�iZ article_objZ
current_textsZcurrent_lengthsZcurrent_contextsZ current_tagsZ sub_titler Z
sentence_sizer r r
r sP 26B 270�280�z PrepareArticles.parse_dataset)Fr r N)T)T)�__name__�29__module__� __qualname__�__doc__r r r r r r r
r s 3031r )�pandasr �numpyr& r r( r r r r r
�<module> s