David310/Detect_AI-generated_Image
4
1a
2 � �c � @ s| d dl Z d dlZd dlZd dlmZ d dlZd dlZe� dd� �Ze� dd� �Z dd� Z3d d4� Z d d � Z G d
d� de
�ZdS )� N)� lru_cachec C s t j�t j�t j�t��d�S )Nz bpe_simple_vocab_16e6.txt.gz)�os�path�join�dirname�abspath�__file__� r r �Y/nobackup2/yuheng-data/forensic_diffusion/linear_probe/my_models/clip/simple_tokenizer.py� default_bpe5 s r c C s� t ttd�td�d ��t ttd�td�d �� t ttd�td�d �� } | dd� }d }td6�D ],}|| vrf| �|� |�d7| � |d7 }qfd d � |D �}tt| |��S )
a9 8 Returns list of utf-8 byte and a corresponding list of unicode strings.9 The reversible bpe codes work on unicode strings.10 This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.11 When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.12 This is a signficant percentage of your normal, say, 32K bpe vocab.13 To avoid that, we want lookup tables between utf-8 bytes and unicode strings.14 And avoids mapping to whitespace/control characters the bpe code barfs on.15 �!�~� � ¡� ¬� ®� ÿNr � c S s g | ] }t |��qS r )�chr)�.0�nr r r16 �17<listcomp>" � z$bytes_to_unicode.<locals>.<listcomp>)�list�range�ord�append�dict�zip)�bs�csr �br r r18 �bytes_to_unicode s N 1920r" c C s6 t � }| d }| dd� D ]}|�||f� |}q|S )zReturn set of symbol pairs in a word.21 Word is represented as tuple of symbols (symbols being variable-length strings).22 r r N)�set�add)�word�pairsZ prev_char�charr r r23 � get_pairs& s r( c C s" t �| �} t�t�| ��} | �� S �N)�ftfyZfix_text�html�unescape�strip��textr r r24 � basic_clean2 s 25r0 c C s t �dd| �} | �� } | S )Nz\s+� )�re�subr- r. r r r26 �whitespace_clean8 s r4 c @ s8 e Zd Ze� fed�dd�Zdd� Zdd� Zdd � Zd27S ) �SimpleTokenizer)�bpe_pathc C s t � | _dd� | j�� D �| _t�|��� �d��d�}|dd� }dd� |D �}t t � �28� �}|d d� |D � }|D ]}|� d29� |�� qv|�
d d g� tt|tt|����| _d
d� | j�� D �| _tt|tt|����| _d d d�| _t�dtj�| _d S )Nc S s i | ]\}}||�qS r r �r �k�vr r r30 �31<dictcomp>A r z,SimpleTokenizer.__init__.<locals>.<dictcomp>�utf-8�32r i�� c S s g | ]}t |�� ��qS r )�tuple�split)r �merger r r33 r D r z,SimpleTokenizer.__init__.<locals>.<listcomp>c S s g | ] }|d �qS )�</w>r )r r9 r r r34 r F r � �<|startoftext|>�
<|endoftext|>c S s i | ]\}}||�qS r r r7 r r r35 r: K r )rB rC z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r" � byte_encoder�items� byte_decoder�gzip�open�read�decoder> r �valuesr r �extendr r r �len�encoder�decoder� bpe_ranks�cacher2 �compile�36IGNORECASE�pat)�selfr6 ZmergesZvocabr? r r r37 �__init__? s zSimpleTokenizer.__init__c 38 sv |� j v r� j | S t|d d� �|d d f }t|�}|sF|d S t|� fdd�d�}|� jvrh�q^|\}}g }d}|t|�k �r4z&|�||�} |�||| � � | }W n$ |�||d � � Y �q4Y n0 || |k�r |t|�d k �r ||d |k�r |�|| � |d7 }qx|�|| � |d7 }qxt|�}|}t|�dk�rT�q^qFt|�}qFd � |�}|� j |< |S )39N�����r@ c s � j �| td��S )N�inf)rP �get�float)�pair�rU r r40 �<lambda>Z r z%SimpleTokenizer.bpe.<locals>.<lambda>)�keyr r � r1 )41rQ r= r( �minrP rM �indexrL r r )42rU �tokenr% r& Zbigram�first�secondZnew_word�i�jr r\ r43 �bpeP sB 4445 46 24748495051zSimpleTokenizer.bpec sn g }t t|���� }t�� j|�D ]F}d�� fdd�|�d�D ��}|�� fdd�� � |��52d�D �� q"|S )NrA c 3 s | ]}� j | V qd S r) )rD )r r! r\ r r53 � <genexpr>} r z)SimpleTokenizer.encode.<locals>.<genexpr>r; c 3 s | ]}� j | V qd S r) )rN )r Z bpe_tokenr\ r r54 rh ~ r r1 ) r4 r0 �lowerr2 �findallrT r �encoderL rg r> )rU r/ Z55bpe_tokensrb r r\ r56 rk y s &zSimpleTokenizer.encodec sD d� � fdd�|D ��}t� fdd�|D ��jddd��dd �}|S )57NrA c s g | ]}� j | �qS r )rO )r rb r\ r r58 r � r z*SimpleTokenizer.decode.<locals>.<listcomp>c s g | ]}� j | �qS r )rF )r �cr\ r r59 r � r r; �replace)�errorsr@ r1 )r � bytearrayrJ rm )rU �tokensr/ r r\ r60 rJ � s (zSimpleTokenizer.decodeN) �__name__�61__module__� __qualname__r �strrV rg rk rJ r r r r62 r5 > s )r5 )rG r+ r � functoolsr r* �regexr2 r r" r( r0 r4 �objectr5 r r r r63 �<module> s 6465