OpenXAIProject/pnpxai-image-classification
3
1o
2 w �f� � @ s� d dl mZmZ d dlZd dlZd dlZd dlZd dlmZ d dl m3Z4 d dl m Z d dl
mZ d dlZd dlZd dlmZ d dlmZ d d lmZmZmZ d d5lmZmZ d d lmZm Z d d l m Z d d
l m Z G dd� de�Z d7de!deee! fdd�Z"G dd� de�Z#d8dd�Z$e e� G dd � d e�Z%d d � Z&d d � Z'G d!d"� d"e�Z(d#d$� Z)G d%d&� d&e�Z*d'd(� Z+d)d*� Z,d+d,� Z-d9d-d.�Z.d/d0� Z/d1d2� Z0d:d3d4�Z1d5d6� Z2dS );� )�Optional�ListN)�BytesIO)�Path)�disable_warnings)�InsecureRequestWarning)�Tensor)�Module)�Dataset�Subset�6DataLoader)�
BertTokenizer� BertForSequenceClassification)�ViltForQuestionAnswering�
ViltProcessor)�tqdm)�Imagec @ s. e Zd Zd7dd�Zdd� Zdd� Zdd � ZdS ) �ImageNetDatasetNc C s� || _ tj�| j d�| _tj�| j d�| _t| j��}t�|�| _ W d � n1 s+w Y t�8| j�| _ | j � � || _
d S )Nzsamples/zimagenet_class_index.json)�root_dir�os�path�join�img_dirZ label_dir�open�json�load�
idx_to_labels�listdir� img_names�sort� transform)�selfr r � json_data� r# �/home/user/app/helpers.py�__init__ s �910zImageNetDataset.__init__c C �11 t | j�S �N)�lenr �r! r# r# r$ �__len__( � 12zImageNetDataset.__len__c C sB t j�| j| j| �}t�|��d�}|}| jr | �|�}||fS )N�RGB) r r r r r r r �convertr )r! �idx�img_path�image�labelr# r# r$ � __getitem__+ s 13zImageNetDataset.__getitem__c C s | j t|� d S )N� )r �str�r! r. r# r# r$ � idx_to_label5 s z ImageNetDataset.idx_to_labelr' )�__name__�14__module__� __qualname__r% r* r2 r6 r# r# r# r$ r s15 16
17r �d �./data/ImageNet� subset_size�indicesc C sX t �tt�j� t|| d�}|d urt||d�S ttt |���}t||d |� d�}|S )N)r r )r= )18r �chdirr �__file__�parentr r �list�ranger( )r r< r r= �dataset�subsetr# r# r$ �get_imagenet_dataset8 s rE c s. e Zd Zd� fdd� Zdd� Zdd� Z� ZS ) � IMDBDataset�testc s, t � �� t|d�}dd� t|�D �| _d S )N��splitc S s g | ]19\}}||d f�qS �r3 r# )�.0r1 �liner# r# r$ �20<listcomp>K s z(IMDBDataset.__init__.<locals>.<listcomp>)�superr% ZIMDBr � annotations)r! rI � data_iter�� __class__r# r$ r% H s 2122zIMDBDataset.__init__c C r& r' �r( rO r) r# r# r$ r* M r+ zIMDBDataset.__len__c C s23 | j | S r' )rO r5 r# r# r$ r2 P r+ zIMDBDataset.__getitem__�rG �r7 r8 r9 r% r* r2 �
__classcell__r# r# rQ r$ rF G s rF rG c C s24 t | d�S )NrH )rF rH r# r# r$ �get_imdb_datasetT r+ rW c s, e Zd Z� fdd�Zdd� Zdd� Z� ZS )�25VQADatasetc s$ t � �� t�d�}t|j�| _d S )Nz'https://visualqa.org/balanced_data.json)rN r% �requests�get�eval�textrO )r! �resrQ r# r$ r% [ s 2627zVQADataset.__init__c C r& r' rS r) r# r# r$ r* ` r+ zVQADataset.__len__c C sr | j | }t|d t�r.td|d � d�� tj|d dd�}t�t|j ���28d�}||d< |d |d |d fS ) N�original_imagez Requesting z...F)�verifyr, �questionZoriginal_answer) rO �29isinstancer4 �printrY rZ r r r �contentr- )r! r. �datar] �imgr# r# r$ r2 d s 30zVQADataset.__getitem__rU r# r# rQ r$ rX Z s rX c C s t � S r' )rX r# r# r# r$ �get_vqa_datasetn s rf c C s2 t j�| �j}t jj| |d��� }|�� }||fS )N)�weights)� torchvision�models�get_model_weights�DEFAULT� get_modelr[ �31transforms)�32model_namerg �modelr r# r# r$ �get_torchvision_modelt s rp c � e Zd Z� fdd�Z� ZS )�Bertc s t � j|||d�jS )N)� input_ids�token_type_ids�attention_mask�rN �forward�logits)r! rs rt ru rQ r# r$ rw | s ��z Bert.forward�r7 r8 r9 rw rV r# r# rQ r$ rr { � rr c C s t j| |d�S )N��33num_labels)rr �from_pretrained)rn r| r# r# r$ �get_bert_model� s r~ c rq )�Viltc s t � j|||||d�jS )N)rs rt ru � pixel_values�34pixel_maskrv )r! r� rs rt ru r� rQ r# r$ rw � s ��z Vilt.forwardry r# r# rQ r$ r � rz r c C �35 t �| �S r' )r r} �rn r# r# r$ �get_vilt_model� r+ r� c C s | � ddd��� �� S )Nr3 � r )�permute�detach�numpy)re r# r# r$ �<lambda>� s r� c C s4 t | t|�d d �d d f t|�d d �d d f �S r' )� img_to_npr )�inputs�mean�stdr# r# r$ �denormalize_image� s ���r� c C s> |dd� | D �dddd�}t �dd� | D ��}t|�� �|fS )Nc S � g | ]}|d �qS �r r# �rK �dr# r# r$ rM � � z#bert_collate_fn.<locals>.<listcomp>T�pt)�padding�36truncation�return_tensorsc S r� rJ r# r� r# r# r$ rM � r� )�torch�tensor�tuple�values)�batch� tokenizerr� �labelsr# r# r$ �bert_collate_fn� s �r� c C r� r' )r
r} r� r# r# r$ �get_bert_tokenizer� r+ r� c C r� r' )r r} r� r# r# r$ �get_vilt_processor� r+ r� c sj dd� | D �}dd� | D �}|||dddd�}t �� fdd�| D ��}|d |d |d37 |d |d |fS )
Nc S r� r� r# r� r# r# r$ rM � r� z#vilt_collate_fn.<locals>.<listcomp>c S r� rJ r# r� r# r# r$ rM � r� Tr� )�imagesr\ r� r� r� c s g | ]}� |d �qS )r� r# r� ��label2idr# r$ rM � s r� rs rt ru r� )r� r� )r� � processorr� �imgsZqstsr� r� r# r� r$ �vilt_collate_fn� s" ��r� c C s | dkr#t d�\}}|�|�}|�� t|�}t|ddd�}|||fS | dkrQtddd �}|�|�}|�� td38d �}td�}t|ddtj t39|d �d
�}|||fS | dkr~t d�}|�|� |�� t � }t
d�}t|ddtj t||jjd�d
�}|||fS d S )Nr0 �resnet18� F)�40batch_size�shuffler\ z!fabriceyhc/bert-base-uncased-imdbr� r{ rG rH )r� )r� r� �41collate_fn)r0 r\ z dandelin/vilt-b32-finetuned-vqa)r� r� )rp �tor[ rE r r~ rW r� � functools�partialr� r� rf r� r� �configr� )�modality�devicero r rC �loaderr� r� r# r# r$ �&load_model_and_dataloader_for_tutorial� sL 4243 4445 �4647��48 49�r� )r: r; NrT r' )NN)3�typingr r r r rY r� �ior �pathlibr �urllib3r Zurllib3.exceptionsr r� rh r �torch.nn.modulesr �torch.utils.datar50 r r � transformersr
r r r r �PILr r �intrE rF rW rX rf rp rr r~ r r� r� r� r� r� r� r� r� r# r# r# r$ �<module> sX ��5152�53
54 55