HCKLab/BiBert-MultiTask-1
06
1B
2 �Vc; � @ s� d dl Z d dlZd dl mZ d dlmZmZmZ d dlmZm Z m3Z4m Z d dlm Z d dlm
Z
mZmZ d dlmZ d dlmZ d d lmZmZmZmZmZmZmZm Z m Z d d5l m Z m Z m!Z! e!d e�G d d
� d
e��Z"dS )� N)�nn)�BCEWithLogitsLoss�CrossEntropyLoss�MSELoss)�List�Optional�Tuple�Union)�
BertTokenizer)�models�DataCollatorWithPadding�
AutoTokenizer)�SequenceClassifierOutput)�6BertConfig) �BertPreTrainedModel�BERT_INPUTS_DOCSTRING�_TOKENIZER_FOR_DOC�_CHECKPOINT_FOR_DOC�BERT_START_DOCSTRING�_CONFIG_FOR_DOC�_SEQ_CLASS_EXPECTED_OUTPUT�_SEQ_CLASS_EXPECTED_LOSS� BertModel)�add_code_sample_docstrings�%add_start_docstrings_to_model_forward�add_start_docstringsz�7 Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled8 output) e.g. for GLUE tasks.9 c s� e Zd Z� fdd�Zee�d��eee e10e e e
d�d eej eej eej eej eej eej eej ee ee ee eeej e11f d� dd���Z� ZS )12� BertForSequenceClassificationc s� t � �t�� � |�di �| _|| _t|�| _|j d k r>|j n|j13}t � |�| _
t �|j| jd j�| _t �|j| jd j�| _| �� d S )NZ tasks_mapr � )�super�__init__� transformers�PretrainedConfig�get�tasks�configr �bert�classifier_dropoutZhidden_dropout_probr �Dropout�dropout�Linear� hidden_size�14num_labels� classifier1� classifier2Z init_weights)�selfr$ �kwargsr& )� __class__� �?/content/BiBert-MultiTask-1/bert_for_sequence_classification.pyr & s 15 z&BertForSequenceClassification.__init__zbatch_size, sequence_length)�processor_class�16checkpoint� output_type� config_class�expected_output�
expected_lossN) � input_ids�attention_mask�token_type_ids� position_ids� head_mask�
inputs_embeds�labels�output_attentions�output_hidden_states� return_dict�returnc C s2 |17dk r |18n| j j}19| j|||||||| |20d� } | d }
| �|
�}
t�| ��� }g }d}x�|D ]z}d}| |k}|dkr�| �|
| �}n|dkr�| �|
| �}|dk r^t � }||�21d| j | j �|| �22d��}|�
|� q^W |r�t�|��� }|23�s |f| dd� }|dk �r|f| S |S t||| j| jd�S )a� 24 labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):25 Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ...,26 config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss),27 If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).28 N)r: r; r< r= r> r@ rA rB r r ������ )�loss�logits�
hidden_states�29attentions)r$ �use_return_dictr% r( �torch�unique�tolistr, r- r �viewr# r+ �append�stack�meanr rH rI )r. r9 r: r; r< r= r> r? r@ rA rB �task_ids�outputsZ
pooled_outputZunique_task_ids_list� loss_listrG Zunique_task_idrF Ztask_id_filterZloss_fct�outputr1 r1 r2 �forward= sJ 3031$z%BertForSequenceClassification.forward) NNNNNNNNNNN)�__name__�32__module__� __qualname__r r r �formatr r r r r r r r rK �Tensor�boolr r rV �
__classcell__r1 r1 )r0 r2 r s, 33 Lr )#rK r r �torch.nnr r r �typingr r r r r34 r r r
Z transformers.modeling_outputsr Z+transformers.models.bert.configuration_bertr Z&transformers.models.bert.modeling_bertr r r r r r r r r Ztransformers.file_utilsr r r r r1 r1 r1 r2 �<module> s ,