afulara/PythonicRAG-FastAPI-React
0
1�
2 X��g� � � � d dl Z d dlmZ d dlZ G d� d� Z G d� d� Z G d� d� Zed k( r� ed3� Zej � e� Z4e5j ej � Z
e ee
� � ee
d � ed � ee
d � ed � ee
d
� ed � ee
d � yy)� N)�Listc �6 � e Zd Zd dedefd�Zd� Zd� Zd� Zd� Zy)6�TextFileLoader�path�encodingc �. � g | _ || _ || _ y �N)� documentsr r )�selfr r s �w/Users/aqsa/AI Projects/AIE6 Class Projects/AIE6-DeployPythonicRAG-Fastapi-react-app/backend/aimakerspace/text_utils.py�__init__zTextFileLoader.__init__ s � ������ � ��
� c �6 � t j j | j � r| j � y t j j | j � r,| j j d� r| j
� y t d� �)N�.txtz;Provided path is neither a valid directory nor a .txt file.)�osr �isdir�load_directory�isfile�endswith� load_file�7ValueError�r s r �loadzTextFileLoader.load sa � �
�7�7�=�=���� #� � � � !�
�W�W�^�^�D�I�I�
&�4�9�9�+=�+=�f�+E� �N�N� � �M��
r c �� � t | j d| j �� 5 }| j j |j � � d d d � y # 1 sw Y y xY w)N�r�r )�openr r r8 �append�read)r �fs r r zTextFileLoader.load_file sE � �
�$�)�)�S�4�=�=�
9� ,�Q� �N�N� !� !�!�&�&�(� +� ,� ,� ,�s �*A�A c �n � t j | j � D ]� \ }}}|D ]| }|j d� s�t t j j ||� d| j �� 5 }| j j |j � � d d d � �~ �� y # 1 sw Y ��xY w)Nr r r )9r �walkr r r �joinr r10 r r )r �root�_�files�filer s r r z TextFileLoader.load_directory s� � � �g�g�d�i�i� 0� 8�N�D�!�U� �
8���=�=��(� ���� � �T�4�0�#��
�
�� 8�� ���-�-�a�f�f�h�7�8� 8�
8� 8�8� 8�s �4*B+�+B4 c �: � | j � | j S r �r r11 r s r �load_documentsz TextFileLoader.load_documents# � � � � � � ��~�~� r N)zutf-8� �__name__�12__module__� __qualname__�strr
r r r r* � r r r r s) � �!�S� !�C� !�13�,�8� r r c �T � e Zd Z d14dedefd�Zdedee fd�Zdee dee fd�Zy ) �CharacterTextSplitter�15chunk_size�
chunk_overlapc �8 � ||kD sJ d� �|| _ || _ y )Nz-Chunk size must be greater than chunk overlap)r4 r5 )r r4 r5 s r r
z CharacterTextSplitter.__init__) s- � �
� � &� ;� :� ;� &� %��� *��r �text�returnc � � g }t dt |� | j | j z16 � D ]# }|j |||| j z � �% |S )Nr )�range�lenr4 r5 r )r r7 �chunks�is r �splitzCharacterTextSplitter.split5 sS � ����q�#�d�)�T�_�_�t�7I�7I�%I�J� 9�A� �M�M�$�q�1�t���#6�7� 8� 9��
r �textsc �X � g }|D ]" }|j | j |� � �$ |S r )�extendr> )r r? r< r7 s r � split_textsz!CharacterTextSplitter.split_texts; s0 � ���� ,�D� �M�M�$�*�*�T�*� +� ,��
r N)i� �� ) r- r. r/ �intr
r0 r r> rB r1 r r r3 r3 ( sS � � � �17+��18+� �19+��#� �$�s�)� � ��c�� �t�C�y� r r3 c �0 � e Zd Zdefd�Zd� Zd� Zd� Zd� Zy)� PDFLoaderr c �P � g | _ || _ t d| j � �� y )Nz!PDFLoader initialized with path: )r20 r �print)r r s r r
zPDFLoader.__init__C s$ � ������ �
�1�$�)�)��=�>r c �8 � t d| j � �� t dt j j | j � � �� t dt j j | j � � �� t dt j j | j � � �� t dt
t j | j � j � dd � �� t | j d� 5 } d d d � | j � y # 1 sw Y �xY w# t $ r)}t d| j � d t |� � �� �d }~wt $ r)}t d21| j � d t |� � �� �d }~ww xY w) NzLoading PDF from path: z
Path exists: z Is file: zIs directory: zFile permissions: ������rbzCannot access file at 'z': zError processing file at ')rH r r �existsr r �oct�stat�st_moder r �IOErrorr r0 � Exception)r � test_file�es r r zPDFLoader.loadH sB � �
�'�� � �{�3�4�
�
�b�g�g�n�n�T�Y�Y� 7� 8�9�:�
� �"�'�'�.�.����3�4�5�6�
��r�w�w�}�}�T�Y�Y� 7� 8�9�:�
�"�3�r�w�w�t�y�y�'9�'A�'A�#B�2�3�#G�"H�I�J� R��d�i�i��&�
�)��
�
�N�N� �
�
�� � O� � 6�t�y�y�k��S��V�H� M�N� N��� R� � 9�$�)�)��C��A��x� P�Q� Q�� R�s< �:D7 �D+�D7 �+D4�0D7 �7 F� $E$�$ F�0$F�Fc � � t | j d� 5 }t j |� }d}|j D ] }||j � dz z
}� | j j |� d d d � y # 1 sw Y y xY w)NrK � �22)r r �PyPDF2� PdfReader�pages� extract_textr23 r )r r' �24pdf_readerr7 �pages r r zPDFLoader.load_file\ s � �
�$�)�)�T�
"� (�d� �)�)�$�/�J� �D�"�(�(�
3����)�)�+�d�2�2��
3�
�N�N� !� !�$� '� (� (� (�s �AA:�:Bc �� � t j | j � D ]� \ }}}|D ]� }|j � j d� s�#t j j ||� }t
|d� 5 }t j |� }d}|j D ] } || j � dz z
}� | j j |� d d d � �� �� y # 1 sw Y ��xY w)Nz.pdfrK rU rV )
r r" r �lowerr r# r rW rX rY rZ r25 r )26r r$ r% r& r' � file_pathr r[ r7 r\ s27 r r zPDFLoader.load_directoryh s� � � �g�g�d�i�i� 0� 4�N�D�!�U� �
4���:�:�<�(�(��0� "��� � �T�4� 8�I� �i��.� 4�!�%+�%5�%5�a�%8�28� "��$.�$4�$4� ?�D� �D�$5�$5�$7�$�$>� >�D�?� ���-�-�d�3�4� 4�
4� 4�4� 4�s
�8AC � C( c �: � | j � | j S r r) r s r r* zPDFLoader.load_documentsw r+ r Nr, r1 r r rF rF B s# � �?�S� ?�29R�(30(�
4� r rF �__main__zdata/KingLear.txtz--------� ����������)r �typingr rW r r3 rF r- �loaderr �splitterrB r31 r< rH r; r1 r r �<module>rh s� �� � �
� � �D� �47 � 7 �t �z��
� /�
0�F�32�K�K�M�$�&�H�
�
!�
!�&�"2�"2�
3�F� �#�f�+�� �&��)�� �*�� �&��)�� �*�� �&��*�� �*�� �&��*�� r