Apppublic

xdecoder/Instruct-X-Decoder

sourceHugging Faceafl-3.0updated 3y agoView on Hugging Face

163likes

rootxdecoderlanguageLangEncoder__pycache__build.cpython-38.pyc

build.cpython-38.pyc11 linesDownload Raw Back to __pycache__

1U

2E$�c	�@sPddlZddlmZmZddlmZddlmZddlmZdd�Zd	d3�Z	dS)�N)�
CLIPTokenizer�CLIPTokenizerFast)�
AutoTokenizer�)�
lang_encoders)�is_lang_encodercKs2|d}t|�std|����t|�|||f|�S)N�NAMEzUnkown model: )r�4ValueErrorr)�config_encoder�	tokenizer�verbose�kwargs�5model_name�r�</data/arXiv/demo/Demo/xdecoder/language/LangEncoder/build.py�build_lang_encoder6srcCs|d}dtjd<|ddkrB|�dd�}t�|�}|�d|ji�n6|ddkrj|�dd�}tj|d	d7�}nt�|d�}|S)N�trueZTOKENIZERS_PARALLELISM�	TOKENIZER�clipZPRETRAINED_TOKENIZERzopenai/clip-vit-base-patch32�	cls_tokenz	clip-fastT)Z	from_slow)	�os�environ�getr�from_pretrainedZadd_special_tokensZ	eos_tokenrr)r8rZpretrained_tokenizerrrr�build_tokenizers"9�10�r)11r�transformersrrr�registryrrrrrrrr�<module>s