MachineLearningReply/docwhiz
0
1�
2 ��f� � �� � d dl Zd dlmZmZ d dlmZ d dlmZ d dl m3Z4 d dl m Z m Z m
Z
d dlmZmZmZmZ d dlmZ ej d �5� � d � � � Z ej d �6� � d efd
�� � Z ej d �7� � de8fd�� � Z ej d �9� � d� � � Z ej d �10� � de11dedefd�� � Z ej d �12� � de13defd�� � Z d� Z dd�Z dS )� N)�document_store_configs�
model_configs)�Pipeline)�Answer)�BaseDocumentStore)�InMemoryDocumentStore�OpenSearchDocumentStore�WeaviateDocumentStore)�EmbeddingRetriever�14FARMReader�15PromptNode� PreProcessor)�MilvusDocumentStoreF)� show_spinnerc �N � t d� � t dddddd�� � } | S )Nz initializing preprocessor nodeT�word�d )�clean_empty_lines�clean_whitespace�clean_header_footer�split_by� split_length� split_respect_sentence_boundary)�printr )� processors �;/Users/a.madhigirigopinath/search_mlReply/utils/haystack.py�start_preprocessor_noder
s? � � �16*�+�+�+� � � � ���(,�17� 18� 19�I� �� �typec 20 �( � t d� � | dk rt dd�� � }n�| dk rdt t d t d t d t d21 t d t d t d
�� � }n�| dk r@t t d t d t d t d �� � }n:| dk r4t t d t d t d d�� � }|S )Nzinitializing document store�inmemoryTi� )�use_bm25�
embedding_dim�22opensearch�OPENSEARCH_SCHEME�OPENSEARCH_USERNAME�OPENSEARCH_PASSWORD�OPENSEARCH_HOST�OPENSEARCH_PORT�OPENSEARCH_INDEX�OPENSEARCH_EMBEDDING_DIM)�scheme�username�password�host�port�indexr# �weaviate�
WEAVIATE_HOST�
WEAVIATE_PORT�WEAVIATE_INDEX�WEAVIATE_EMBEDDING_DIM)r/ r0 r1 r# �milvus�23MILVUS_URI� MILVUS_INDEX�MILVUS_EMBEDDING_DIM)�urir1 r# �return_embedding)r r r r r24 r )r �document_stores r �start_document_storer> s* � � 25�26'�(�(�(� �z���.��C�P�P�P�� � 27� � � �0�:P�Qd�:e�<R�Sh�<i�<R�Sh�<i�8N�O`�8a�8N�O`�8a�9O�Pb�9c�AW�Xr�As�
u� u� u��� 28�� � �.�6L�_�6]�7M�o�7^�8N�O_�8`�@V�Wo�@p�r� r� r��� 29�� � �,�3I�,�3W�4J�>�4Z�<R�Si�<j�=A�C� C� C�� �r �_document_storec �^ � t d� � t | t d d�� � }|S )Nzinitializing retriever�EMBEDDING_MODEL� )r= �embedding_model�top_k)r r r )r? � retrievers r �start_retrieverrF F s; � � �30"�#�#�#�"�/�3@�AR�3S�)*�,� ,� ,�I� �r c �Z � t d� � t t d �� � } | S )Nzinitializing reader�EXTRACTIVE_MODEL)�model_name_or_path)r r r )�readers r � start_readerrK R s, � � �31 � � � �
�=�9K�+L�
M�
M�
M�F� �Mr �32_retriever�_readerc � � t d� � t � � }|� |ddg�� � |� |ddg�� � |S )Nzinitializing pipeline� Retriever�Query�� component�name�inputs�Reader)r r �add_node)r? rL rM �pipes r �start_haystack_extractiverX [ sS � � �33!�"�"�"� �:�:�D��M�M�J�[�'��M�K�K�K��M�M�W�8�[�M�M�J�J�J� �Kr c �� � t dt d |d�� � }t � � }|� |ddg�� � |� |ddg�� � |S ) Nzdeepset/question-answering�GENERATIVE_MODELi� )�default_prompt_templaterI �api_key�34max_lengthrO rP rQ r
)r
r r rV )r? rL �35openai_key� prompt_noderW s r �start_haystack_ragr` c sl � � �5Q�0=�>P�0Q�%/�(+�-� -� -�K� �:�:�D��M�M�J�[�'��M�K�K�K��M�M�K�l�K�=�M�Q�Q�Q� �Kr c �8 � i }| � ||�� � }|S )N)�params)�run)� _pipeline�questionrb �resultss r �queryrg q s! � �
�F��m�m�H�V�m�4�4�G� �Nr � c �b � | dk rt |||� � S | dk rt |||� � S d S )N�36extractive�rag)rX r` )�taskr= rE rJ r^ s r �initialize_pipelinerm v sA � � �|���(� ��F�K�K�K�
�� � �!�.�)�Z�H�H�H� 37� r )rh ) � streamlit�st� utils.configr r �haystackr �haystack.schemar �haystack.document_storesr r r r38 �haystack.nodesr r r
r �milvus_haystackr �cache_resourcer �strr> rF rK rX r` rg rm � r r �<module>ry s: �� � � � � >� >� >� >� >� >� >� >� � � � � � � "� "� "� "� "� "� 6� 6� 6� 6� 6� 6� j� j� j� j� j� j� j� j� j� j� S� S� S� S� S� S� S� S� S� S� S� S� /� /� /� /� /� /� ����&�&�&�
�
� '�&�
� ����&�&�&�$�s� $� $� $� '�&�$�N ����&�&�&��%6� � � � '�&�� ����&�&�&�� � '�&�� ����&�&�&��/@� �N`� �ku� � � � '�&�� ����&�&�&�39�(9� 40�GY� 41� 42� 43� '�&�44�� � �45I� I� I� I� I� Ir