CalebKoster/Translation_Note_Alignment
0
1a
2 ��e � @ sl d dl mZ d dlZd dlZd dlmZ d dlm Z g d�Z3dd� e4D �Z5dd d6�Z dd d �Z dd
d�Z
dS )� )�TfidfVectorizerN)�islice)�uroman)A� i� i�7 i i i� ip i� i/ iZ i" iB% i( i�+ i�. i0 i�1 iJ2 ix6 iW@ i�D i=E iHJ i�O i6P i/U i�V iYW i�W i4X iIX iyX i�X iIY i~Y i�Y iwZ i�Z i�^ i�a if iqi i_m io i�p i�q iZr i�r i]s i�s it iDt i�t iu i6u iOu i~v i�v iSw i�w i�w ix ix i.x i�y c C s g | ] }|d �qS )r � )�.0�xr r �:c:\Users\caleb\Bible Translation Project\guidance\tfidf.py�8<listcomp>M � r9 Fc C s� t | ddd��f}t|�D ]L\}}||kr|rJt|�� � W d � S |�� W d � S qW d � n1 sz0 Y d S )N�r�utf-8��encoding)�open� enumerater �strip)� file_path� line_number�romanize�file�i�liner r r �extract_interested_verseP s :r c C s� g }g }t | ddd���}t|dd�D ]F\}}|tv r\|r\d�|�}|rNt|�}|�|� g }|�|�� � q$|r�d�|�}|r�t|�}|�|� W d � n1 s�0 Y |S )Nr r
r r )�start� )r r �verses�joinr �appendr )r r � documentsZcurrent_documentr r r Zjoined_doc_stringr r r �segment_corpus\ s$ 101112(r c s� t | |d�}tdd�}|�|�}|�� }t� fdd�tt�D ��d }t�|| � � ��13� }t t ||���t
| � d |d�} | r�tdd�}14|15�| g� |16�� } �fdd�| D �} t t| �� d d17� d d ��}
|
S d
S d S )N)r )� � )Z ngram_rangec 3 s | ]\}}|� kr|V qd S )Nr )r r �v)�interested_liner r � <genexpr>y r z*analyze_verse_in_corpus.<locals>.<genexpr>r c s i | ]}|� � |d ��qS )r )�get)r �ngram)� scores_dictr r �18<dictcomp>� r z+analyze_verse_in_corpus.<locals>.<dictcomp>c S s | d S )Nr r )�itemr r r �<lambda>� r z)analyze_verse_in_corpus.<locals>.<lambda>T)�key�reversezVerse not found.)r r Z
fit_transformZget_feature_names_out�nextr r �np�array�todense�flatten�dict�zipr �fit�sorted�items)r r$ r r Ztfidf_vectorizerZ tfidf_matrixZ
feature_namesZdocument_index�scoresZinterested_verseZtfidf_vectorizer_verseZ verse_ngrams� verse_scoresZsorted_verse_scoresr )r$ r( r �analyze_verse_in_corpusr s 1920 21 r: )F)F)F)Z sklearn.feature_extraction.textr �pandas�pd�numpyr/ � itertoolsr r r r r r r: r r r r �<module> s E22 23