sumanthd/IndicTrans-MultilingualTranslation
6
1U
2 ��,cw � @ s� d dl mZ d dlmZ d dlmZ d dlmZ d dlmZmZ d dl Z d dl3m4Z5 d dl m Z d d l m
Z
d d6lmZ d d lmZ d d lmZ d d
l mZ d dlmZ dddddddddddg Zdd� Zd d � Zd d � Zd d!� ZG d"d#� d#�Z dS )$� )�truncate)�MosesPunctNormalizer)�MosesTokenizer)�MosesDetokenizer)�BPE�read_vocabularyN)�tqdm)�indic_tokenize)�indic_detokenize)�indic_normalize)�unicode_transliterate)�MosesSentenceSplitter)�sentence_tokenize)�7Translator�asZbn�gu�hi�kn�ml�mr�or�pa�ta�tec 8 C sJ |dkr0t |��}|| g�W 5 Q R � S Q R X n|tkrFtj| |d�S d S )N�en��lang)r
�INDICr Zsentence_split)� paragraph�language�splitter� r! �+D:\ai4bharat\indicTrans\inference\engine.py�split_sentences s9 10 r# c C sB g }|D ]&\}}d| d | d }|� |� qd�|�d | S )z�add special tokens specified by tag_infos to each element in list11 12 tag_infos: list of tuples (tag_type,tag)13 14 each tag_info results in a token of the form: __{tag_type}__{tag}__15 16 �__� )�append�join)�sentZ tag_infos�tokensZtag_type�tag�tokenr! r! r" � add_token s17 r, c C s6 g }| D ](}t |�� d|fd|fg�}|�|� q|S )N�src�tgt)r, �stripr& )�sents�src_lang�tgt_lang� tagged_sentsr( Z tagged_sentr! r! r" �apply_lang_tags. s18 r4 c C s� d}g }| D ]p}|� � }t|�}||krrd�|d d� �d d�|dd � � }d�|d |� �}td|� d�� |�|� q |S )N�� r% � z .... �����zWARNING: Sentence z; truncated to 200 tokens as it exceeds maximum length limit)�split�lenr' �printr& )r0 Z MAX_SEQ_LENZ new_sentsr( �wordsZ num_wordsZ print_strr! r! r" �truncate_long_sentences6 s (19� r< c @ sF e Zd Zdd� Zdd� Zdd� Zdd� Zd d20� Zdd d
�Zdd� Z dS )�Modelc C s� || _ tdd�| _t� | _tdd�| _t�� | _ t21d� t t j
|� d�dd�d�| _tt j
|� d�dd�d d22| jd �| _t23d � t|� d �|� d
�dd�| _d S )Nr r zInitializing vocab and bpez/vocab/vocab.SRCzutf-8)�encodingr6 z/vocab/bpe_codes.32k.SRC�����z@@z"Initializing model for translationz24/final_binz/model/checkpoint_best.pt�d )Z25batch_size)�expdirr �en_tokr �
en_normalizerr �en_detokr �UnicodeIndicTransliterator�26xliteratorr: r �codecs�openZ27vocabularyr �bper �28translator)�selfrA r! r! r" �__init__J s. 29 �� �zModel.__init__c C sV t |t�st�| j||d�}| �|�}t|||�}t|�}| j�|�}| � ||�}|S )Nr )30�31isinstance�list�AssertionError�32preprocess� apply_bper4 r< rJ � translate� postprocess) rK �batchr1 r2 Zpreprocessed_sentsZ bpe_sentsr3 Z translations�postprocessed_sentsr! r! r" �batch_translatec s 33 zModel.batch_translatec C s4 t |t�st�t||�}| �|||�}d�|�}|S )Nr% )rM �strrO r# rV r' )rK r r1 r2 r0 rU Ztranslated_paragraphr! r! r" �translate_paragraphq s34 3536zModel.translate_paragraphc 37 C s^ |dkr*d� | jj| j�|�� �dd��S tj�d� t �38|�|�� �|��|d�� dd�S d S )Nr r% F)�escaper u ् u ्) r' rB �tokenizerC � normalizer/ r rE �
transliterater Ztrivial_tokenize�replace)rK r( �39normalizerr r! r! r" �preprocess_sent| s( �� ��� �zModel.preprocess_sentc sR � dkr"� �fdd�t |�D �}n,t�� }|�� ��� ��fdd�t |�D �}|S )zr40 Normalize, tokenize and script convert(for Indic)41 return number of sentences input file42 43 r c s g | ]}�� |d � ��qS )N�r_ ��.0�line)r rK r! r" �44<listcomp>� s z$Model.preprocess.<locals>.<listcomp>c s g | ]}�� |�� ��qS r! r` ra �r r^ rK r! r" rd � s )r r ZIndicNormalizerFactoryZget_normalizer)rK r0 r Zprocessed_sentsZ normfactoryr! re r" rP � s �45�zModel.preprocessr c C s` g }|dkr0|D ] }|� | j�|�d��� qn,|D ]&}t�| j�|||�|�}|� |� q4|S )al 46 parse fairseq interactive output, convert script back to native Indic script (in case of Indic languages) and detokenize.47 48 infname: fairseq log file49 outfname: output file of translation (sentences not translated contain the dummy string 'DUMMY_OUTPUT'50 input_size: expected number of output sentences51 lang: language52 r r% )r& rD Z53detokenizer8 r54 Ztrivial_detokenizerF r\ )rK r0 r Z common_langrU r( �outstrr! r! r" rS � s � zModel.postprocessc s � fdd�|D �S )Nc s g | ]}� j �|��qS r! )rI Z process_line)rb r( �rK r! r" rd � s z#Model.apply_bpe.<locals>.<listcomp>r! )rK r0 r! rg r" rQ � s zModel.apply_bpeN)r )55�__name__�56__module__� __qualname__rL rV rX r_ rP rS rQ r! r! r! r" r= I s 57r= ) �osr Z58sacremosesr r r Zsubword_nmt.apply_bper r rG r Zindicnlp.tokenizer r59 Zindicnlp.normalizer Zindicnlp.transliterater Zmosestokenizerr
r Z inference.custom_interactiver r r# r, r4 r< r= r! r! r! r" �<module> s&