David310/Detect_AI-generated_Image
4
1o
2 �N�d � @ s| d dl Z d dlZd dlZd dlmZ d dlZd dlZe� dd� �Ze� dd� �Z dd� Z3d d4� Z d d � Z G d
d� de
�ZdS )� N)� lru_cachec C s t j�t j�t j�t��d�S )Nz bpe_simple_vocab_16e6.txt.gz)�os�path�join�dirname�abspath�__file__� r r �F/nobackup2/utkarsh/UniversalFakeDetect/models/clip/simple_tokenizer.py� default_bpe5 s r c C s� t ttd�td�d ��t ttd�td�d �� t ttd�td�d �� } | dd� }d }td6�D ]}|| vrI| �|� |�d7| � |d7 }q3d d � |D �}tt| |��S )
a9 8 Returns list of utf-8 byte and a corresponding list of unicode strings.9 The reversible bpe codes work on unicode strings.10 This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.11 When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.12 This is a signficant percentage of your normal, say, 32K bpe vocab.13 To avoid that, we want lookup tables between utf-8 bytes and unicode strings.14 And avoids mapping to whitespace/control characters the bpe code barfs on.15 �!�~� � ¡� ¬� ®� ÿNr � c S s g | ]}t |��qS r )�chr)�.0�nr r r16 �17<listcomp>" � z$bytes_to_unicode.<locals>.<listcomp>)�list�range�ord�append�dict�zip)�bs�csr �br r r18 �bytes_to_unicode s N 19�r" c C s6 t � }| d }| dd� D ] }|�||f� |}q
|S )zReturn set of symbol pairs in a word.20 Word is represented as tuple of symbols (symbols being variable-length strings).21 r r N)�set�add)�word�pairsZ prev_char�charr r r22 � get_pairs& s r( c C s" t �| �} t�t�| ��} | �� S �N)�ftfyZfix_text�html�unescape�strip��textr r r23 � basic_clean2 s 24r0 c C s t �dd| �} | �� } | S )Nz\s+� )�re�subr- r. r r r25 �whitespace_clean8 s r4 c @ s8 e Zd Ze� fdefdd�Zdd� Zdd� Zdd � Zd26S ) �SimpleTokenizer�bpe_pathc C s t � | _dd� | j�� D �| _t�|��� �d��d�}|dd� }dd� |D �}t t � �27� �}|d d� |D � }|D ]28}|� d29� |�� q;|�
d d g� tt|tt|����| _d
d� | j�� D �| _tt|tt|����| _d d d�| _t�dtj�| _d S )Nc S � i | ]\}}||�qS r r �r �k�vr r r30 �31<dictcomp>A � z,SimpleTokenizer.__init__.<locals>.<dictcomp>�utf-8�32r i�� c S s g | ]}t |�� ��qS r )�tuple�split)r �merger r r33 r D s z,SimpleTokenizer.__init__.<locals>.<listcomp>c S s g | ]}|d �qS )�</w>r )r r: r r r34 r F r � �<|startoftext|>�
<|endoftext|>c S r7 r r r8 r r r35 r; K r< )rD rE z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r" � byte_encoder�items� byte_decoder�gzip�open�read�decoder@ r �valuesr r �extendr r r �len�encoder�decoder� bpe_ranks�cacher2 �compile�36IGNORECASE�pat)�selfr6 ZmergesZvocabrA r r r37 �__init__? s zSimpleTokenizer.__init__c 38 sj |� j v r39� j | S t|d d� �|d d f }t|�}|s#|d S t|� fdd�d�}|� jvr4nu|\}}g }d}|t|�k r�z|�||�} |�||| � � | }W n |�||d � � Y n3|| |kr�|t|�d k r�||d |kr�|�|| � |d 7 }n |�|| � |d7 }|t|�k sBt|�}|}t|�dkr�nt|�}q$d40� |�}|� j |< |S ) N�����rB Tc s � j �| td��S )N�inf)rR �get�float)�pair�rW r r41 �<lambda>Z s z%SimpleTokenizer.bpe.<locals>.<lambda>)�keyr r � r1 )42rS r? r( �minrR rO �indexrN r r )43rW �tokenr% r& Zbigram�first�secondZnew_word�i�jr r^ r44 �bpeP sH 4546 47 ,48 � �49 50zSimpleTokenizer.bpec sn g }t t|���� }t�� j|�D ]#}d�� fdd�|�d�D ��}|�� fdd�� � |��51d�D �� q|S )NrC c 3 � � | ]}� j | V qd S r) )rF )r r! r^ r r52 � <genexpr>} � � z)SimpleTokenizer.encode.<locals>.<genexpr>r= c 3 rj r) )rP )r Z bpe_tokenr^ r r53 rk ~ rl r1 ) r4 r0 �lowerr2 �findallrV r �encoderN ri r@ )rW r/ Z54bpe_tokensrd r r^ r55 ro y s &zSimpleTokenizer.encodec sD d� � fdd�|D ��}t� fdd�|D ��jddd��dd �}|S )56NrC c � g | ]}� j | �qS r )rQ )r rd r^ r r57 r � r< z*SimpleTokenizer.decode.<locals>.<listcomp>c rp r )rH )r �cr^ r r58 r � r< r= �replace)�errorsrB r1 )r � bytearrayrL rr )rW �tokensr/ r r^ r59 rL � s (zSimpleTokenizer.decodeN) �__name__�60__module__� __qualname__r �strrX ri ro rL r r r r61 r5 > s62 ) r5 )rI r+ r � functoolsr r* �regexr2 r r" r( r0 r4 �objectr5 r r r r63 �<module> s 6465