xdecoder/Instruct-X-Decoder
163
1U
2 E$�c � @ s\ d dl Z d dlmZ d dlm mZ dd� Zdd� Zdd� Ze � � dd � �Z3d d4d �Z dS )
� Nc C s5 t j�� S )N)�torch� distributed�is_initialized� r r �6/data/arXiv/demo/Demo/xdecoder/utils/it_contrastive.py�is_dist_initialized s r c C s t � rtj�� S dS )N� )r r r �get_world_sizer r r r r s 6r c sP t � dkrL� fdd�tt � �D �}tj�|� � � |tj�� < tj|dd�� � S )Nr c s g | ]}t �� ��qS r )r �7zeros_like��.0�_��xr r �8<listcomp> s z#all_gather_grad.<locals>.<listcomp>r ��dim)r �ranger r �9all_gather�get_rank�cat)r Zall_xr r r �all_gather_grad
s 10r c sJ t � dkrF� fdd�ttj� � �D �}tjj|� dd� tj|dd�� � S )z�11 Performs all_gather operation on the provided tensors.12 *** Warning ***: torch.distributed.all_gather has no gradient.13 r c s g | ]}t �� ��qS r )r � ones_liker ��tensorr r r s �z%all_gather_nograd.<locals>.<listcomp>F)�async_opr r )r r r r r r )r Ztensors_gatherr r r �all_gather_nograd s 1415 �r c C s t | �} t |�}t�| |�� �}|| }|d krl|d krltj|jd |jd�}t�||�}t�|�� |�}n�t |�}t |�}|� d�|� d�k} |� d�|� d�k}16t�17| |18�}t� |tj |dd� � |� � }t� |�� tj |�� dd� � |� � }|| d t
� S )Nr )�device)�����r )r r r r � )r r �matmul�t�arange�shaper �F�
cross_entropy�reshape�19logical_or�sum� log_softmaxr ) Z20image_featZ text_feat� temperature�image_idZtext_id�logits�gtZloss1Zloss2Zgt_imageZgt_textr r r �image_text_contrastive_loss$ s "*r. )NN) r �torch.nn�nnZtorch.nn.functional�21functionalr$ r r r �no_gradr r. r r r r �<module> s 22