xdecoder/Instruct-X-Decoder
163
1U
2 E$�c� � @ s� d dl mZ d dlmZmZ d dlZd dlZd dlZd dl Z d dl3m m Z
d dl m Z d dlmZmZ ddlmZ d dlmZ d d lmZ e�e�ZeG d4d � d e j��ZG d d
� d
e j�Z G dd� de j�Z G dd� de j�Z edd� �Z dS )� )� OrderedDict)�Tuple�UnionN)�nn)�DropPath�
trunc_normal_� )�register_lang_encoder)�is_main_process)�register_norm_modulec s& e Zd Zd� fdd� Zdd� Z� ZS )� LayerNorm��-���q=c s< t t| ��� t�t�|��| _t�t�|��| _ || _5dS )zWConstruct a layernorm module in the TF style (epsilon inside the square root).6 N) �superr �__init__r � Parameter�torch�ones�weight�zeros�bias�variance_epsilon)�self� hidden_size�eps�� __class__� �B/data/arXiv/demo/Demo/xdecoder/language/LangEncoder/transformer.pyr s zLayerNorm.__init__c C sb |j }|�� }|jddd�}|| �d�jddd�}|| t�|| j � }| j|�|� | j S )N�����T)�keepdim� )7�dtype�float�mean�powr �sqrtr r �tor )r �xZpdtype�u�sr r r �forward s zLayerNorm.forward)r
)�__name__�8__module__� __qualname__r r* �
__classcell__r r r r r s r c @ s e Zd Zejd�dd�ZdS )� QuickGELU�r' c C s |t �d| � S )Ng� Zd;�?)r �sigmoid)r r' r r r r* ( s zQuickGELU.forwardN)r+ r, r- r �Tensorr* r r r r r/ ' s r/ c sX e Zd Zd eeejed�� fdd�
Zd ejejd�dd�Zd
ejejd�d d9�Z � Z10S )�ResidualAttentionBlockN� )�d_model�n_head� attn_mask� drop_pathc 11 s� t � �� t�||�| _t|�| _t�tdt� ||d �fdt12� fdt� |d |�fg��| _ t|�| _ || _
|dkr~t|�nt�� | _d S )NZc_fc� �geluZc_projr4 )r r r �MultiheadAttention�attnr �ln_1�13Sequentialr �Linearr/ �mlp�ln_2r7 r �Identityr8 )r r5 r6 r7 r8 r r r r - s 1415�1617z ResidualAttentionBlock.__init__)r' �key_padding_maskc C s@ | j d k r | j j|j|jd�nd | _ | j||||d| j d�d S )N)r! �deviceF)rC � need_weightsr7 r )r7 r& r! rD r< �r r' rC r r r � attention? s �� ��z ResidualAttentionBlock.attentionc C s< || � | j| �|�|d�� }|| � | �| �|��� }|S )N)rC )r8 rG r= r@ rA rF r r r r* K s z ResidualAttentionBlock.forward)Nr4 )N)N) r+ r, r- �intr r2 r" r rG r* r. r r r r r3 , s �� r3 c sv e Zd Zdeeeeeeed�� fdd�
Zedd� �Zdd � Z d18d � Z19d g dfd
d�Z e j
jdd� �Zddd�Z� ZS )� Transformerr4 T)�context_length�20vocab_size�width�layers�headsr8 � autogressivec s� t � �� t�|��| _|| _t�t�| j���| _ �| _21|| _ || _ |rR| �
� nd � dd� t�d||�D ��t�� ���fdd�t|�D ��| _t��| _t| j dd� t| jjdd� | �| j� d S )Nc S s g | ] }|� � �qS r )�item)�.0r' r r r �22<listcomp>g s z(Transformer.__init__.<locals>.<listcomp>r c s g | ]}t ��� �| ��qS r )r3 )rQ �i�r7 �dprrN rL r r rR i s ��{�G�z�?��std)r r r � Embedding�token_embeddingrJ r r �empty�positional_embeddingrL rM rO �build_attention_mask�linspace�23ModuleList�range� resblocksr �ln_finalr r �apply�
_init_weights)r rJ rK rL rM rN r8 rO r rT r r R s( 24 ���25zTransformer.__init__c C s | j S )N)rL �r r r r �dim_outv s zTransformer.dim_outc C s, t �| j| j�}|�td�� |�d� |S )Nz-infr )r r[ rJ �fill_r" �triu_)r �maskr r r r] z s 26z Transformer.build_attention_maskc C s� t |tjtjf�r\t� r"t�d� t|jdd� |j d k r~t� rJt�d� tj27� |j d� n"t |tj tj
f�r~tj28� |j d� d S )Nz/=> init weight of Linear/Conv2d from trunc normrV rW z&=> init bias of Linear/Conv2d to zerosr )�29isinstancer r? �Conv2dr30 �logger�infor r r �init� constant_r � BatchNorm2d)r �mr r r rd � s 313233zTransformer._init_weights� c s� t j�|��r�tj|dd�}t�d|� �� | �� � dd� �� �fdd�|�� D �}i }|�� D �] \}}|� d�d |kp�|d d34k}|r`|r�t35�d |� d |� �� d
|k�rx|� � � | � � k�rx|} � | }36| � � \} } |37� � \}
}| |k�rt38�d|� d�� np| |
k�rxt39�d� | | f|
|f�� | �
� }|jd d��d dd�}tjjj||
dd�}|�d dd�jd d�}|}|||< q`| j|dd� d S )N�cpu)� map_locationz => loading pretrained model c S s | � d�r| dd � S | S )Nz
lang_encoder.�
)�40startswithr0 r r r �<lambda>� � z-Transformer.load_pretrained.<locals>.<lambda>c s* i | ]"\}}�|�� � � kr�|�|�qS r )�keys)rQ �k�v��41model_dictZ stripped_keyr r �42<dictcomp>� s � z/Transformer.load_pretrained.<locals>.<dictcomp>�.r �*z=> init z from r\ zError in loading z , passingz-=> load_pretrained: resized variant: {} to {})�dimr r �linear)�size�modeF)�strict)�os�path�isfiler �load�loggingrm �43state_dict�items�splitrl r� �formatr" � unsqueeze�permuter �44functional� interpolate�squeeze�load_state_dict)r �45pretrained�pretrained_layers�verbose�pretrained_dict�need_init_state_dictrz r{ � need_initZ positional_embedding_pretrainedZ positional_embedding_current�L1�nH1�L2�nH2�posemb� posemb_gridr r| r �load_pretrained� sJ �46� 4748 ��49zTransformer.load_pretrainedc C s ddhS )Nr\ rZ r re r r r �no_weight_decay� s �zTransformer.no_weight_decayNc C sr | j s|d k r|dknd }| �|�}|| j }|�ddd�}| jD ]}|||�}qB|�ddd�}| �|�}d|iS )Nr r r �last_hidden_state)rO rZ r\ r� ra rb )r � input_ids�attention_maskrC r' �blockr r r r* � s 505152 53zTransformer.forward)r4 T)N)r+ r, r- rH r" �boolr �propertyrf r] rd r� r �jit�ignorer� r* r. r r r r rI Q s& ��$54 -55rI c 56 K sX t | d |j| d | d | d | �dd�d�}| �dd �rT|�| d57 | �d d g�� |S )
N�CONTEXT_LENGTH�WIDTHZLAYERSZHEADSZ AUTOGRESSIVET)rJ rK rL rM rN rO �LOAD_PRETRAINEDF�58PRETRAINED�PRETRAINED_LAYERSr� )rI rK �getr� )�config_encoder� tokenizerr� �kwargs� transformerr r r � lang_encoder� s 59� r� ) � collectionsr �typingr r r� r� �numpy�npr Ztorch.nn.functionalr r� �F�timm.models.layersr r �registryr �utils.distributedr60 Z utils.modelr � getLoggerr+ rl �Moduler r/ r3 rI r� r r r r �<module> s( 61%