Th3BossC/TranscriptApi
1
1o
2 j��dN � @ s d dl Z d dlZd dlZd dlmZ d dlmZ d dl m3Z4 d dl Z d dl Z d dl
Z
d dlmZ d dlmZ d dlmZ e
j�� rCdndZd Zd5d � Zd d
� Zd)dd�Zdd� Zde j ded�fdd�Z dd� Z d*dd�Z d+d d �Z d e!fd d!�Z"d,d#d$�Z#d%e!d&e!fd'd(�Z$dS )-� N)�YouTube)�
VideoFileClip��pipeline)�tqdm)� PdfReader�cudaZcpuz/Th3BossC/SummarizationModel_t5-small_opeai_tldrc C s t d| �jS )Nz https://www.youtube.com/watch?v=)r �title)�video_id� r �Bd:\Coding\Main_Project\TranscriptApi\TranscriptApi\common\utils.pyr s r c C s& t � d�}t �|j�}|d d }|S )Nz/http://www.youtube.com/watch?v=z_AbfPXTKms&NR=1�vr )�urlparse�parse_qs�query)� video_urlZurl_datar �videor r r � get_video_id s 6 r �audioc C st t j�|�s t �|� || d }|| d }td� t| �jjdd��� j |d�}t7|�}td� |j � |� |S )Nz.mp4�.mp3z[INFO] downloading video...Zmp4)�file_extension)�filenamez%[INFO] extracting audio from video...)
�os�path�exists�makedirs�printr �streams�filter�firstZdownloadr r Zwrite_audiofile)r �locationr Zvideo_filename�audio_filenamer r r r � get_video$ s 8 r" c C s� t j�|�s t �|� tj| dd�\}}tj||d�}t|| �d }td|� d�� g }t |�D ]1}|| | } |d | | }9|| |10� } t11� t j� |d|� d��| |� |�
|d|� d� � q1t|� |S ) NiD� )�sr)�yr# � z[INFO] Chunking z12 chunks...Zsegment_r )r r �isdir�mkdir�librosa�loadZ get_duration�intr �range�sf�write�join�append) r �segment_lengthZ13output_dirr r# ZdurationZ num_segments� audio_files�i�start�end�segmentr r r � chunk_audio6 s 14 r6 �base)�devicec C s� t d� g }|�t� | D ]}|�|�}|�|d � q
|d urBt|d��}|D ] }|�|d � q(W d � |S 1 s=w Y |S )Nz"[INFO] converting audio to text...�text�w�15)r �tor8 Z16transcriber/ �openr- )r1 Z output_file�modelZ transcriptsZ17audio_file�response�f�18transcriptr r r �transcribe_audioK s 1920 �21��rB c 22 C s� t dt� �� |� d�}|� d�}|� d�}|� d�}d}tj�|�r-t�|� t�|� t| |�}t |||�}t23||�} g }24| D ] } |25| � d�7 }26qAdd � |27D �} t | �t
| � }
td28|
�}t
|29�| t
|30�| d k }t d t
| �� d
|
� d|� d|� d� � tdttddd�}g }tt|��D ] }d�|31|| ||d � �}||dd�d d }|�|� q�d�|�}t|d��
}|�|� W d � n1 s�w Y t|d�� }|�� }W d � n1 s�w Y ||d �S ) Nz[INFO] running on z /raw_audio/z/chunks/z/transcripts.txtz /summary.txtiX �.c S � g | ]}t |��qS r ��len��.0Zsentencer r r �32<listcomp>v � z+summarize_youtube_video.<locals>.<listcomp>�@ r � 33 [INFO] sentences_len : �#34 [INFO] sentence_mean_length : �&,35 [INFO] num_sentences_per_step : �,36 [INFO] num_steps : �37 �
summarization�� T)r> � tokenizer�38max_length�39truncation� r% F�Z do_sample� summary_textr: �r)rA �summary)r r8 r r r �shutil�rmtreer' r" r6 rB �split�sumrF r* r �40checkpointr r+ r. r/ r= r- �read)r Z outputs_dirZ
raw_audio_dirZ41chunks_dirZtranscripts_fileZ summary_filer0 r! Zchunked_audio_filesZtranscriptions� sentencesrA �
sentences_len�sentence_mean_length�num_sentences_per_step� num_steps�42summarizer� summariesr2 �chunkrZ �complete_summaryr@ Zcomplete_transcriptr r r �summarize_youtube_video` sT 43444546 474849 50 ���51� 52 � 53 �54rj �#TranscriptApi/static/files/temp.pdfc C s( t | �}d}|jD ]}||�� 7 }q |S )N� )r �pagesZ extract_text)�
file_location�readerr9 �pager r r �extract_text_pdf� s55 56rq �#TranscriptApi/static/files/temp.txtc C s8 t | d��
}|�� }W d � |S 1 sw Y |S )NrY )r= r` )rn r@ r9 r r r �extract_text_txt� s 5758��rs r9 c 59 C s� | � d�}tdttdddd�}dd� |D �}t|�t|� }td | �}t|�| t|�| dk }td60t|�� d |� d |� d
|� d� � g }tt|��D ] }d� ||| ||d � �} || dd�d d }61|�62|63� qMd� |�} | S )NrC rQ rR Tr )r> rS rT rU r8 c S rD r rE rG r r r rI � rJ z$summarize_string.<locals>.<listcomp>rK rL rM rN rO rP rV r% FrW rX ) r] r r_ r^ rF r* r r r+ r. r/ ) r9 ra rf rb rc rd re rg r2 rh rZ ri r r r �summarize_string� s, 64 ���65� 66rt �TranscriptApi/static/filesc C sX d}|dkr t | �}n |dkrt| �}ndS tj�|�r!t�|� t�|� |t|�gS )Nrl �pdf�txtz[ERROR]) rq rs r r r r[ r\ r' rt )rn r Z working_dirr9 r r r �summarize_file� s 6768 6970 rx �question�contextc C s t ddd�}|| |d�d S )Nzquestion-answeringzdeepset/roberta-base-squad2)�taskr> )ry rz �answerr )ry rz Zqar r r r| � s r| )r )rk )rr )ru )%r r( Z soundfiler, Zpytuber � urllib.parse�parser Zmoviepy.editorr r[ ZwhisperZtorchZ transformersr Z tqdm.autor ZPyPDF2r r Z is_availabler8 r_ r r r" r6 Z71load_modelrB rj rq rs �strrt rx r| r r r r �<module> s2 727397475