David310/Detect_AI-generated_Image
4
1U
2 ��^f � @ s| d dl Z d dlZd dlZd dlmZ d dlZd dlZe� dd� �Ze� dd� �Z dd� Z3d d4� Z d d � Z G d
d� de
�ZdS )� N)� lru_cachec C s t j�t j�t j�t��d�S )Nz bpe_simple_vocab_16e6.txt.gz)�os�path�join�dirname�abspath�__file__� r r �D/root/autodl-tmp/UniversalFakeDetect/models/clip/simple_tokenizer.py� default_bpe5 s r c C s� t ttd�td�d ��t ttd�td�d �� t ttd�td�d �� } | dd� }d }td6�D ],}|| krf| �|� |�d7| � |d7 }qfd d � |D �}tt| |��S )
a9 8 Returns list of utf-8 byte and a corresponding list of unicode strings.9 The reversible bpe codes work on unicode strings.10 This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.11 When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.12 This is a signficant percentage of your normal, say, 32K bpe vocab.13 To avoid that, we want lookup tables between utf-8 bytes and unicode strings.14 And avoids mapping to whitespace/control characters the bpe code barfs on.15 �!�~� � ¡� ¬� ®� ÿNr � c S s g | ] }t |��qS r )�chr)�.0�nr r r16 �17<listcomp>" s z$bytes_to_unicode.<locals>.<listcomp>)�list�range�ord�append�dict�zip)�bs�csr �br r r18 �bytes_to_unicode s N 1920r! c C s6 t � }| d }| dd� D ]}|�||f� |}q|S )zReturn set of symbol pairs in a word.21 Word is represented as tuple of symbols (symbols being variable-length strings).22 r r N)�set�add)�word�pairsZ prev_char�charr r r23 � get_pairs& s r' c C s" t �| �} t�t�| ��} | �� S �N)�ftfyZfix_text�html�unescape�strip��textr r r24 � basic_clean2 s 25r/ c C s t �dd| �} | �� } | S )Nz\s+� )�re�subr, r- r r r26 �whitespace_clean8 s r3 c @ s8 e Zd Ze� fed�dd�Zdd� Zdd� Zdd � Zd27S ) �SimpleTokenizer)�bpe_pathc C s t � | _dd� | j�� D �| _t�|��� �d��d�}|dd� }dd� |D �}t t � �28� �}|d d� |D � }|D ]}|� d29� |�� qv|�
d d g� tt|tt|����| _d
d� | j�� D �| _tt|tt|����| _d d d�| _t�dtj�| _d S )Nc S s i | ]\}}||�qS r r �r �k�vr r r30 �31<dictcomp>A s z,SimpleTokenizer.__init__.<locals>.<dictcomp>�utf-8�32r i�� c S s g | ]}t |�� ��qS r )�tuple�split)r �merger r r33 r D s z,SimpleTokenizer.__init__.<locals>.<listcomp>c S s g | ] }|d �qS )�</w>r )r r8 r r r34 r F s � �<|startoftext|>�
<|endoftext|>c S s i | ]\}}||�qS r r r6 r r r35 r9 K s )rA rB z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r! � byte_encoder�items� byte_decoder�gzip�open�read�decoder= r �valuesr r �extendr r r �len�encoder�decoder� bpe_ranks�cacher1 �compile�36IGNORECASE�pat)�selfr5 ZmergesZvocabr> r r r37 �__init__? s zSimpleTokenizer.__init__c 38 sv |� j kr� j | S t|d d� �|d d f }t|�}|sF|d S t|� fdd�d�}|� jkrh�q^|\}}g }d}|t|�k �r4z&|�||�} |�||| � � | }W n$ |�||d � � Y �q4Y nX || |k�r |t|�d k �r ||d |k�r |�|| � |d7 }qx|�|| � |d7 }qxt|�}|}t|�dk�rT�q^qFt|�}qFd � |�}|� j |< |S )39N�����r? c s � j �| td��S )N�inf)rO �get�float)�pair�rT r r40 �<lambda>Z � z%SimpleTokenizer.bpe.<locals>.<lambda>)�keyr r � r0 )41rP r< r' �minrO rL �indexrK r r )42rT �tokenr$ r% Zbigram�first�secondZnew_word�i�jr r[ r43 �bpeP sB 4445 46 24748495051zSimpleTokenizer.bpec sn g }t t|���� }t�� j|�D ]F}d�� fdd�|�d�D ��}|�� fdd�� � |��52d�D �� q"|S )Nr@ c 3 s | ]}� j | V qd S r( )rC )r r r[ r r53 � <genexpr>} s z)SimpleTokenizer.encode.<locals>.<genexpr>r: c 3 s | ]}� j | V qd S r( )rM )r Z bpe_tokenr[ r r54 rh ~ s r0 ) r3 r/ �lowerr1 �findallrS r �encoderK rg r= )rT r. Z55bpe_tokensrb r r[ r56 rk y s &zSimpleTokenizer.encodec sD d� � fdd�|D ��}t� fdd�|D ��jddd��dd �}|S )57Nr@ c s g | ]}� j | �qS r )rN )r rb r[ r r58 r � s z*SimpleTokenizer.decode.<locals>.<listcomp>c s g | ]}� j | �qS r )rE )r �cr[ r r59 r � s r: �replace)�errorsr? r0 )r � bytearrayrI rm )rT �tokensr. r r[ r60 rI � s (zSimpleTokenizer.decodeN) �__name__�61__module__� __qualname__r �strrU rg rk rI r r r r62 r4 > s )r4 )rF r* r � functoolsr r) �regexr1 r r! r' r/ r3 �objectr4 r r r r63 �<module> s 6465