xdecoder/Instruct-X-Decoder
163
1U
2 E$�c�! � @ s� d dl Z d dlmZ d dlZd dlm mZ d dlm Z3 d dl m Z m
Z
d dlmZ e� Zdd� Zdd� Zd d4� Zd d � Zd dd�Zd dd�Zdd� Zdd� Zd dd�Zd!dd�Zd"dd�Zd d � Z dS )#� N)�log)� rearrange�repeat)�SoftTargetCrossEntropyc C s5 t j�� S )N)�torch� distributed�is_initialized� r r �//data/arXiv/demo/Demo/xdecoder/language/loss.py�is_dist_initialized
s r c C s t � rtj�� S dS )N� )r r r �get_world_sizer r r r6 r
s 7r
c C s t � rt�� S dS )Nr )r �dist�get_rankr r r r8 r s r c sP t � dkrL� fdd�tt � �D �}tj�|� � � |tj�� < tj|dd�� � S )Nr c s g | ]}t �� ��qS r )r �9zeros_like��.0�_��xr r10 �11<listcomp> s z#all_gather_grad.<locals>.<listcomp>r ��dim)r
�ranger r �12all_gatherr �cat)r �all_xr r r13 �all_gather_grad s 14r r c C s� | t |d� }|t | d� }| jd }| jd }|jd }t t�|�|�d� d�}t t�|�|�d� d�} t | d�}15t |d�} |16t| ��� } | t|17��� }
tjtj ||||tj18|19j d�t � t
� d��|20j�}|t |d �ttj||21j|22j d�d23� 9 }t |d �}tjtj ||||tj24| j d�t � t
� d��| j�}|t | d �ttj|| j|25j d�d
� 9 }t |d�}|�� jdd�}t|| |�}t||
|�}d|| }|S )z�26 Args:27 image_feat (torch.Tensor): shape [B, L1, C] # B: batch_size, L1: 1, C: 25628 text_feat (torch.Tensor): shape [B, L2, C] # B:batch_size, L2: number of selected nouns, C: 25629 30 Returns:31 zb l c -> b c lr r zb l2 l1 -> b l1 l2zb l1 l2 -> b l2 l1zb l c -> (b l) c)�dtype�device)� num_classeszb l1 l2 -> b l1 1 l2 1zb1 b2 -> b1 1 b2 1 1z"b1 l1 b2 l2 w -> (b1 l1) (w b2 l2)zb l2 l1 -> b l2 1 l1 1zb2 b1 -> b2 1 b1 1 1z"b2 l2 b1 l1 w -> (b2 l2) (w b1 l1)�d ��maxg �?)r �shaper � ones_like�sizer �t�F�one_hot�ones�longr r r
�tor r �eye�exp�clamp�soft_cross_entropy)�32image_feat� text_feat� temperatureZ dist_per_imgZ
dist_per_text�batchZimg_len�text_lenZpos_labels_batch_imgZpos_labels_batch_textZimage_xZtext_xZlogits_per_imgZlogits_per_textZlabels_per_imgZlabels_per_text� logit_scale�loss_img� loss_text�lossr r r33 � vl_multilabel_contrastive_loss" sL 3435363738 �� �39 �� �40 r: c C sv t | �} t |�}t�| |�� �}|�� jdd�}tj|jd |jd�}t �41|| |�}t �42||�� |�}|| d S �Nr! r" r �r � ) r r �matmulr' r. r/ �aranger$ r r( �
cross_entropy�r1 r2 r3 �logitsr6 �gt�loss1�loss2r r r43 �vl_contrastive_loss] s rF c C s( t � }|dkr| gS t�| �}tj�|�}t�|��|�}t�|� � g��44� }dd� t |�D �}t �
||� dd� |D �}t|�}g } |D ] }45| �tj|fd��46� � q�||kr�tj|| fd��47� } tj|| fdd�}t �
| |� g } t|| �D ]0\}
}|�� �� �� d|
� }| �t�|�� q�| S ) z�48 Run all_gather on arbitrary picklable data (not necessarily tensors)49 Args:50 data: any picklable object51 Returns:52 list[data]: list of data gathered from each rank53 r c S s g | ]}t �d g��� �qS )r )r �54LongTensor�cudar r r r55 r � s z%all_gather_pickle.<locals>.<listcomp>c S s g | ]}t |�� ��qS r )�int�item)r r& r r r56 r � s )r& r r N)r
�pickle�dumpsr � ByteStorage� from_buffer�57ByteTensorr, rG �numelrH r r r r# �appendr �zip�cpu�numpy�tobytes�loads)�datar �58world_size�buffer�storage�tensor�59local_size� size_list�max_size� tensor_listr �padding� data_listr& r r r60 �all_gather_picklep s. 61 rb c sV t � dkrN| j� t| �� � �}� fdd�|D �}| |tj�� < tj|dd�}n| }|S )Nr c s g | ]}|� � ��qS r )r, )r r r< r r62 r � s z.all_gather_arbitary_tensor.<locals>.<listcomp>r r )r
r rb rS r r r r )r[ Z tensor_batchr r< r63 �all_gather_arbitary_tensor� s 64rc c C sv t | �} t |�}t�| |�� �}|�� jdd�}tj|jd |jd�}t �65|| |�}t �66||�� |�}|| d S r; ) rc r r> r' r. r/ r? r$ r r( r@ rA r r r67 �ql_contrastive_loss� s rd c C s( t �| |�� �}|�� jdd�| }|S )Nr! r" )r r> r' r. r/ )r1 r2 r3 rB r r r68 �
vl_similarity� s re c s� t | �} t � �� t|� j�}t�|��t����� }tj| jd t |�f� jd�}��� �t�69� �fdd�|D ��}t ��D ]\}} d|| |� | �< q|t�
| |�� �}70|71|�� jdd� }72t|73|�} t|74�� |�� |�� jdd d75� �} d | d | }
|
S )
Nr r< c s g | ]}� �� |� �qS r )�index)r �txt�r2 Z
text_hash_allr r76 r � s z-ql_multi_contrastive_loss.<locals>.<listcomp>r r! r" �����T)�keepdimgffffff�?g333333�?)rc rb r r r �unique�tolist�zerosr$ �len�stack� enumeraterf r> r' r. r/ r0 �sum)r1 r2 Z text_hashr3 Ztext_hash_batchZtext_hash_all_uniquerC Ztext_feat_unique�idxrg rB r7 r8 r9 r rh r77 �ql_multi_contrastive_loss� s 78 79$rs c C s� t | �� �}t |�� �}||jddd�d }||jddd�d }|j}t�||�� �}|�� jdd�}tj |j80d |j d�} t �
|| | �}81t �
||�� | �} |82| d S )83Nri T)r rj gH�����z>r! r" r r< r= )r �84contiguous�normr6 r r> r' r. r/ r? r$ r r( r@ ) Zimage_feat_inpZ
text_feat_inp�lang_enc�trainingr1 r2 r3 rB r6 rC rD rE r r r85 �!image_text_contrastive_loss_queue� s rx )r )r )r )r )r ) rK � distutilsr r Ztorch.nn.functional�nn�86functionalr( Ztorch.distributedr r Zeinopsr r Z timm.lossr r0 r r
r r r: rF rb rc rd re rs rx r r r r87 �<module> s& 88;89* 90
9192