afulara/PythonicRAG-FastAPI-React
0
1�
2 X��g� � � � S SK r S SKJr S SKr " S S5 r " S S5 r " S S5 r\S :X a� \" S35 r\R 5 \" 5 r4\5R \R 5 r
\" \" \
5 5 \" \
S 5 \" S 5 \" \
S 5 \" S 5 \" \
S
5 \" S 5 \" \
S 5 gg)� N)�Listc �B � \ rS rSrS S\S\4S jjrS rS rS rS r S r6g7) �TextFileLoader� �path�encodingc �* � / U l Xl X l g �N)� documentsr r )�selfr r s �w/Users/aqsa/AI Projects/AIE6 Class Projects/AIE6-DeployPythonicRAG/fastapi-react-app/backend/aimakerspace/text_utils.py�__init__�TextFileLoader.__init__ s � ����� � �
� c �T � [ R R U R 5 ( a U R 5 g [ R R U R 5 ( a1 U R R S5 ( a U R
5 g [ S5 e)N�.txtz;Provided path is neither a valid directory nor a .txt file.)�osr �isdir�load_directory�isfile�endswith� load_file�8ValueError�r s r
�load�TextFileLoader.load sj � �
�7�7�=�=���� #� #� � � � !�
�W�W�^�^�D�I�I�
&�
&�4�9�9�+=�+=�f�+E�+E� �N�N� � �M��
r c �� � [ U R SU R S9 nU R R UR 5 5 S S S 5 g ! , ( d f g = f)N�r�r )�openr r r �append�read)r �fs r
r �TextFileLoader.load_file s< � �
�$�)�)�S�4�=�=�
9�Q� �N�N� !� !�!�&�&�(� +� :�
9�
9�s �*A�9A!c � � [ R " U R 5 H� u pnU H nUR S5 ( d M [ [ R R X5 SU R S9 nU R R UR 5 5 S S S 5 M� M� g ! , ( d f M� = f)Nr r r )10r �walkr r r �joinr r r! r" )r �root�_�files�filer# s r
r � TextFileLoader.load_directory s� � � �g�g�d�i�i� 0�N�D�U� ���=�=��(�(� ���� � �T�0�#��
�
��� ���-�-�a�f�f�h�7�� � � 1�� �s �6*B1�111C c �: � U R 5 U R $ r12 �r r r s r
�load_documents� TextFileLoader.load_documents# � � � � � � ��~�~� r )r r r N)zutf-8� �__name__�13__module__� __qualname__�__firstlineno__�strr r r r r/ �__static_attributes__� r r
r r s) � �!�S� !�C� !�14�,�8� r r c �d � \ rS rSr S S\S\4S jjrS\S\\ 4S jrS\\ S\\ 4S jr S15r16g )
�CharacterTextSplitter�( �17chunk_size�
chunk_overlapc �4 � X:� d S5 eXl X l g )Nz-Chunk size must be greater than chunk overlap)r= r> )r r= r> s r
r � CharacterTextSplitter.__init__) s' � �
� &� ;� :� ;� &� %�� *�r �text�returnc � � / n[ S[ U5 U R U R -18 5 H" nUR XX0R - 5 M$ U$ )Nr )�range�lenr= r> r! )r rA �chunks�is r
�split�CharacterTextSplitter.split5 sK � ����q�#�d�)�T�_�_�t�7I�7I�%I�J�A� �M�M�$�1���#6�7� 8� K��
r �textsc �\ � / nU H# nUR U R U5 5 M% U$ r19 )�extendrH )r rJ rF rA s r
� split_texts�!CharacterTextSplitter.split_texts; s+ � ����D� �M�M�$�*�*�T�*� +� ��
r )r> r= N)i� �� ) r3 r4 r5 r6 �intr r7 r rH rM r8 r9 r r
r; r; ( sS � � � �20+��21+� �22+��#� �$�s�)� � ��c�� �t�C�y� r r; c �: � \ rS rSrS\4S jrS rS rS rS r Sr23g )24� PDFLoader�B r c �N � / U l Xl [ SU R 35 g )Nz!PDFLoader initialized with path: )r r �print)r r s r
r �PDFLoader.__init__C s" � ����� �
�1�$�)�)��=�>r c �H � [ SU R 35 [ S[ R R U R 5 35 [ S[ R R U R 5 35 [ S[ R R U R 5 35 [ S[
[ R " U R 5 R 5 SS 35 [ U R S5 n S S S 5 U R 5 g ! , ( d f N = f! [ a) n[ SU R S [ U5 35 eS nAf[ a) n[ S25U R S [ U5 35 eS nAff = f) NzLoading PDF from path: z
Path exists: z Is file: zIs directory: zFile permissions: ������rbzCannot access file at 'z': zError processing file at ')rU r r �existsr r �oct�stat�st_moder r �IOErrorr r7 � Exception)r � test_file�es r
r �PDFLoader.loadH s; � �
�'�� � �{�3�4�
�
�b�g�g�n�n�T�Y�Y� 7� 8�9�:�
� �"�'�'�.�.����3�4�5�6�
��r�w�w�}�}�T�Y�Y� 7� 8�9�:�
�"�3�r�w�w�t�y�y�'9�'A�'A�#B�2�3�#G�"H�I�J� R��d�i�i��&�)�� '�
�N�N� � '�&�� � O� � 6�t�y�y�k��S��V�H� M�N� N��� R� � 9�$�)�)��C��A��x� P�Q� Q�� R�s< �;D= �D,�D= �,26D:�6D= �=27F!�$E+�+
F!�8$F � F!c � � [ U R S5 n[ R " U5 nSnUR H nX4R 5 S- -
nM U R R U5 S S S 5 g ! , ( d f g = f)NrY � �28)r r �PyPDF2� PdfReader�pages� extract_textr r! )r r+ �29pdf_readerrA �pages r
r �PDFLoader.load_file\ so � �
�$�)�)�T�
"�d� �)�)�$�/�J� �D�"�(�(���)�)�+�d�2�2�� )�
�N�N� !� !�$� '� #�
"�
"�s �A A<�<30B31c �� � [ R " U R 5 H� u pnU H� nUR 5 R S5 ( d M) [ R R X5 n[
US5 n[ R " U5 nSnUR H n X�R 5 S- -
nM U R R U5 S S S 5 M� M� g ! , ( d f M� = f)Nz.pdfrY rd re )
r r& r �lowerr r' r rf rg rh ri r r! )32r r( r) r* r+ � file_pathr# rj rA rk s33 r
r �PDFLoader.load_directoryh s� � � �g�g�d�i�i� 0�N�D�U� ���:�:�<�(�(��0�0� "��� � �T� 8�I� �i��.�!�%+�%5�%5�a�%8�34� "��$.�$4�$4�D� �$5�$5�$7�$�$>� >�D� %5� ���-�-�d�3� /�.� � 1� /�.�s
�=A C*�*35C9 c �: � U R 5 U R $ r36 r. r s r
r/ �PDFLoader.load_documentsw r1 r )r r Nr2 r9 r r
rR rR B s# � �?�S� ?�37R�(38(�
4� r rR �__main__zdata/KingLear.txtz--------� ����������)r �typingr rf r r; rR r3 �loaderr �splitterrM r rF rU rE r9 r r
�<module>rz s� �� � �
� � �D� �47 � 7 �t �z��
� /�
0�F�39�K�K�M�$�&�H�
�
!�
!�&�"2�"2�
3�F� �#�f�+�� �&��)�� �*�� �&��)�� �*�� �&��*�� �*�� �&��*�� r