OneScience-Group/ESM
024
1�
2 75Nj� � �p � d dl mZ d dlZd dlmZ d dlmZ d dlmZm Z m3Z4m Z G d� dej � � Z
dS )� )�UnionN)�Alphabet)�ContactPredictionHead�ESM1bLayerNorm�
RobertaLMHead�TransformerLayerc �j � � e Zd Z ddededed eeef d5ef6� fd �
Zd � Z g d
d
fd�Z7d� Z � xZ S )�ESM2�! � � �ESM-1bT�8num_layers� embed_dim�attention_heads�alphabet�
token_dropoutc �� �� t � � � � � || _ || _ || _ t |t � � st
j |� � }|| _ t |� � | _9 |j | _ |j | _ |j
| _
|j | _ |j | _ |j | _ || _ | � � � d S )N)�super�__init__r r r �10isinstancer �from_architecturer �len�
alphabet_size� padding_idx�mask_idx�cls_idx�eos_idx� prepend_bos�11append_eosr �_init_submodules)�selfr r r r r � __class__s ��G/root/private_data/gaozhan/onescience_reconstruct/ESM/model/esm/esm2.pyr z
ESM2.__init__ s� �� � ��������$���"��� .�� ��(�H�-�-� <� �1�(�;�;�H� ��
� ��]�]���#�/��� �)��
� �'�� � �'�� �#�/���"�-��� *��� � � � � � � � � c �� � � d� _ t j � j � j � j �� � � _ t j � fd�t � j � � D � � � � � _12 t � j � j z � j
� j � j �� � � _ t# � j � � � _ t' � j � j � j j �� � � _ d S )N� )r c 13 �^ �� g | ])}t �j d �j z �j ddd�� � ��*S )� FT)� add_bias_kv�use_esm1b_layer_norm�use_rotary_embeddings)r r r )�.0�_r" s �r$ �14<listcomp>z)ESM2._init_submodules.<locals>.<listcomp>7 sX �� � 15
� 16
� 17
� � !��N����&��(� %�)-�*.�
� � �18
� 19
� 20
r% )r )r �21output_dim�weight)� embed_scale�nn� Embeddingr r r � embed_tokens�22ModuleList�ranger �layersr r r r r � contact_headr �emb_layer_norm_afterr r1 �lm_head)r" s `r$ r! zESM2._init_submodules. s� �� � ��� �L� � � �N� �(� 23� 24� 25��� �m�26
� 27
� 28
� 29
� �t��/�/�30
� 31
� 32
� 33� 34�� � 2� �O�d� 2� 2� � � �O��L� 35� 36� 37��� %3�4�>�$B�$B��!�$��n��)��$�+�38� 39� 40�� � � r% Fc � � |rd}|j dk sJ �|� | j � � }| j | � |� � z }| j r�|� || j k � d� � d� � d}| � d� � }|| j k � d� � �41 |j � � |z } |d|z42 z d| z43 d d �d d f z }|�.|d|� d� � � |� � z44 z }t |� � }i }45d|v r||46d< |rg } |� dd� � }|� � � sd }t! | j � � D ]d\ } }
|
|||�� � \ }}| dz |v r |� dd� � |47| dz < |r)| � |� dd� � � � �e| � |� � }|� dd� � }| dz |v r||48| dz < | � |� � }||49d �}|r�t+ j | d� � }|�Yd|� |� � z50 }|� d� � |� d� � z }||d d �d d d d �d d �f z }||d51< |r| � ||� � }||d < |S ) NT� �����g g� ��Q��?r' r )�self_attn_padding_mask�need_head_weights)�logits�representations�52attentions�contacts)�ndim�eqr r2 r5 r � masked_fill_r � unsqueeze�sum�to�dtype�type_as�set� transpose�any� enumerater8 �appendr: r; �torch�stackr9 )r" �tokens� repr_layersr@ �return_contacts� padding_mask�x�mask_ratio_train� src_lengths�mask_ratio_observed�hidden_representations� attn_weights� layer_idx�layer�attn�resultrC �attention_maskrD s r$ �forwardz ESM2.forwardR sJ � � � %� $� ��{�a� � � � � �y�y��!1�2�2� � � �t� 0� 0�� 8� 8� 8�� � � V�
�N�N�F�d�m� 3�>�>�r�B�B�C� H� H� H� )� � (�=�-�-�b�1�1�K�#)�T�]�#:�"?�"?��"C�"C�"F�"F�q�w�"O�"O�R]�"]� ��Q�)�)�*�a�2E�.E�q�q�q�$�PT�}�-U�U�A� � #��Q� �/�/��3�3�;�;�A�>�>�>�?�A��+�&�&� �!#� � � � � �()� "�1� %� � � �L�
�K�K��1� � ��� � �!�!� � �L� )�$�+� 6� 6� 53 :� 54 :� �I�u��e��'3�"3�� � �G�A�t�55 �A�
�+�-�-�89� � �A�q�8I�8I�&�y�1�}�5� �
:� �#�#�D�N�N�1�a�$8�$8�9�9�9�� � %� %�a� (� (��
�K�K��1� � ��
��M�k� )� )�45� "�9�q�=� 1� �L�L��O�O�� �2H�I�I�� � 56 .� ��\�1�5�5�J��'�!"�\�%9�%9�*�%E�%E�!E��!/�!9�!9�!�!<�!<�~�?W�?W�XY�?Z�?Z�!Z�� '�.����D�$����1�1�1�9L�*M� M�57�#-�F�<� � �
.� �,�,�V�Z�@�@��%-��z�"��
r% c �( � | |d�� � d S )NT)rV rD � )r" rT s r$ �predict_contactszESM2.predict_contacts� s � ��t�F�D�1�1�1�*�=�=r% )r r r
r T)
�__name__�58__module__� __qualname__�intr r �str�boolr r! rc rf �
__classcell__)r# s @r$ r59 r60 s� �� � � � � � � � !�)1� "�
� �� � � � � �61 ��#�
�&� � �
� � � � � �4"62� "63� "64�H +-��W\� C� C� C� C�J>� >� >� >� >� >� >r% r65 )�typingr rR �torch.nnr3 �onescience.datapipes.esmr �onescience.modules.esmr r r r �Moduler66 re r% r$ �<module>rs s� �� � � � � � � � � � � � � � � � � -� -� -� -� -� -�� � � � � � � � � � � �E>� E>� E>� E>� E>�2�9� E>� E>� E>� E>� E>r%