marcduda/langchain_tutorial

1{"id": "7958c2e23e09-0", "text": ".rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nEcosystem\nAdditional Resources\nWelcome to LangChain#\nLangChain is a framework for developing applications powered by language models. We believe that the most powerful and differentiated applications will not only call out to a language model, but will also be:\nData-aware: connect a language model to other sources of data\nAgentic: allow a language model to interact with its environment\nThe LangChain framework is designed around these principles.\nThis is the Python specific portion of the documentation. For a purely conceptual guide to LangChain, see here. For the JavaScript documentation, see here.\nGetting Started#\nHow to get started using LangChain to create an Language Model application.\nQuickstart Guide\nConcepts and terminology.\nConcepts and terminology\nTutorials created by community experts and presented on YouTube.\nTutorials\nModules#\nThese modules are the core abstractions which we view as the building blocks of any LLM-powered application.\nFor each module LangChain provides standard, extendable interfaces. LangChain also provides external integrations and even end-to-end implementations for off-the-shelf use.\nThe docs for each module contain quickstart examples, how-to guides, reference docs, and conceptual guides.\nThe modules are (from least to most complex):\nModels: Supported model types and integrations.\nPrompts: Prompt management, optimization, and serialization.\nMemory: Memory refers to state that is persisted between calls of a chain/agent.\nIndexes: Language models become much more powerful when combined with application-specific data - this module contains interfaces and integrations for loading, querying and updating external data.\nChains: Chains are structured sequences of calls (to an LLM or to a different utility).", "source": "https://python.langchain.com/en/latest/index.html"}2{"id": "7958c2e23e09-1", "text": "Chains: Chains are structured sequences of calls (to an LLM or to a different utility).\nAgents: An agent is a Chain in which an LLM, given a high-level directive and a set of tools, repeatedly decides an action, executes the action and observes the outcome until the high-level directive is complete.\nCallbacks: Callbacks let you log and stream the intermediate steps of any chain, making it easy to observe, debug, and evaluate the internals of an application.\nUse Cases#\nBest practices and built-in implementations for common LangChain use cases:\nAutonomous Agents: Autonomous agents are long-running agents that take many steps in an attempt to accomplish an objective. Examples include AutoGPT and BabyAGI.\nAgent Simulations: Putting agents in a sandbox and observing how they interact with each other and react to events can be an effective way to evaluate their long-range reasoning and planning abilities.\nPersonal Assistants: One of the primary LangChain use cases. Personal assistants need to take actions, remember interactions, and have knowledge about your data.\nQuestion Answering: Another common LangChain use case. Answering questions over specific documents, only utilizing the information in those documents to construct an answer.\nChatbots: Language models love to chat, making this a very natural use of them.\nQuerying Tabular Data: Recommended reading if you want to use language models to query structured data (CSVs, SQL, dataframes, etc).\nCode Understanding: Recommended reading if you want to use language models to analyze code.\nInteracting with APIs: Enabling language models to interact with APIs is extremely powerful. It gives them access to up-to-date information and allows them to take actions.\nExtraction: Extract structured information from text.\nSummarization: Compressing longer documents. A type of Data-Augmented Generation.", "source": "https://python.langchain.com/en/latest/index.html"}3{"id": "7958c2e23e09-2", "text": "Summarization: Compressing longer documents. A type of Data-Augmented Generation.\nEvaluation: Generative models are hard to evaluate with traditional metrics. One promising approach is to use language models themselves to do the evaluation.\nReference Docs#\nFull documentation on all methods, classes, installation methods, and integration setups for LangChain.\nLangChain Installation\nReference Documentation\nEcosystem#\nLangChain integrates a lot of different LLMs, systems, and products.\nFrom the other side, many systems and products depend on LangChain.\nIt creates a vibrant and thriving ecosystem.\nIntegrations: Guides for how other products can be used with LangChain.\nDependents: List of repositories that use LangChain.\nDeployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps.\nAdditional Resources#\nAdditional resources we think may be useful as you develop your application!\nLangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents.\nGallery: A collection of great projects that use Langchain, compiled by the folks at Kyrolabs. Useful for finding inspiration and example implementations.\nTracing: A guide on using tracing in LangChain to visualize the execution of chains and agents.\nModel Laboratory: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so.\nDiscord: Join us on our Discord to discuss all things LangChain!\nYouTube: A collection of the LangChain tutorials and videos.\nProduction Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support. Please fill out this form and we\u2019ll set up a dedicated support Slack channel.\nnext\nQuickstart Guide\n Contents\n  \nGetting Started\nModules\nUse Cases\nReference Docs\nEcosystem\nAdditional Resources\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/index.html"}4{"id": "7958c2e23e09-3", "text": "Getting Started\nModules\nUse Cases\nReference Docs\nEcosystem\nAdditional Resources\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/index.html"}5{"id": "8ea51f3894e1-0", "text": ".md\n.pdf\nDependents\nDependents#\nDependents stats for hwchase17/langchain\n[update: 2023-05-17; only dependent repositories with Stars > 100]\nRepository\nStars\nopenai/openai-cookbook\n35401\nLAION-AI/Open-Assistant\n32861\nmicrosoft/TaskMatrix\n32766\nhpcaitech/ColossalAI\n29560\nreworkd/AgentGPT\n22315\nimartinez/privateGPT\n17474\nopenai/chatgpt-retrieval-plugin\n16923\nmindsdb/mindsdb\n16112\njerryjliu/llama_index\n15407\nmlflow/mlflow\n14345\nGaiZhenbiao/ChuanhuChatGPT\n10372\ndatabrickslabs/dolly\n9919\nAIGC-Audio/AudioGPT\n8177\nlogspace-ai/langflow\n6807\nimClumsyPanda/langchain-ChatGLM\n6087\narc53/DocsGPT\n5292\ne2b-dev/e2b\n4622\nnsarrazin/serge\n4076\nmadawei2699/myGPTReader\n3952\nzauberzeug/nicegui\n3952\ngo-skynet/LocalAI\n3762\nGreyDGL/PentestGPT\n3388\nmmabrouk/chatgpt-wrapper\n3243\nzilliztech/GPTCache\n3189\nwenda-LLM/wenda\n3050\nmarqo-ai/marqo\n2930\ngkamradt/langchain-tutorials\n2710\nPrefectHQ/marvin\n2545\nproject-baize/baize-chatbot\n2479\nwhitead/paper-qa\n2399\nlanggenius/dify", "source": "https://python.langchain.com/en/latest/dependents.html"}6{"id": "8ea51f3894e1-1", "text": "2479\nwhitead/paper-qa\n2399\nlanggenius/dify\n2344\nGerevAI/gerev\n2283\nhwchase17/chat-langchain\n2266\nguangzhengli/ChatFiles\n1903\nAzure-Samples/azure-search-openai-demo\n1884\nOpenBMB/BMTools\n1860\nFarama-Foundation/PettingZoo\n1813\nOpenGVLab/Ask-Anything\n1571\nIntelligenzaArtificiale/Free-Auto-GPT\n1480\nhwchase17/notion-qa\n1464\nNVIDIA/NeMo-Guardrails\n1419\nUnstructured-IO/unstructured\n1410\nKav-K/GPTDiscord\n1363\npaulpierre/RasaGPT\n1344\nStanGirard/quivr\n1330\nlunasec-io/lunasec\n1318\nvocodedev/vocode-python\n1286\nagiresearch/OpenAGI\n1156\nh2oai/h2ogpt\n1141\njina-ai/thinkgpt\n1106\nyanqiangmiffy/Chinese-LangChain\n1072\nttengwang/Caption-Anything\n1064\njina-ai/dev-gpt\n1057\njuncongmoo/chatllama\n1003\ngreshake/llm-security\n1002\nvisual-openllm/visual-openllm\n957\nrichardyc/Chrome-GPT\n918\nirgolic/AutoPR\n886\nmmz-001/knowledge_gpt\n867\nthomas-yanxin/LangChain-ChatGLM-Webui\n850\nmicrosoft/X-Decoder\n837\npeterw/Chat-with-Github-Repo\n826\ncirediatpl/FigmaChain\n782\nhashintel/hash", "source": "https://python.langchain.com/en/latest/dependents.html"}7{"id": "8ea51f3894e1-2", "text": "826\ncirediatpl/FigmaChain\n782\nhashintel/hash\n778\nseanpixel/Teenage-AGI\n773\njina-ai/langchain-serve\n738\ncorca-ai/EVAL\n737\nai-sidekick/sidekick\n717\nrlancemartin/auto-evaluator\n703\npoe-platform/api-bot-tutorial\n689\nSamurAIGPT/Camel-AutoGPT\n666\neyurtsev/kor\n608\nrun-llama/llama-lab\n559\nnamuan/dr-doc-search\n544\npieroit/cheshire-cat\n520\ngriptape-ai/griptape\n514\ngetmetal/motorhead\n481\nhwchase17/chat-your-data\n462\nlangchain-ai/langchain-aiplugin\n452\njina-ai/agentchain\n439\nSamurAIGPT/ChatGPT-Developer-Plugins\n437\nalexanderatallah/window.ai\n433\nmichaelthwan/searchGPT\n427\nmpaepper/content-chatbot\n425\nmckaywrigley/repo-chat\n422\nwhyiyhw/chatgpt-wechat\n421\nfreddyaboulton/gradio-tools\n407\njonra1993/fastapi-alembic-sqlmodel-async\n395\nyeagerai/yeagerai-agent\n383\nakshata29/chatpdf\n374\nOpenGVLab/InternGPT\n368\nruoccofabrizio/azure-open-ai-embeddings-qna\n358\n101dotxyz/GPTeam\n357\nmtenenholtz/chat-twitter\n354\namosjyng/langchain-visualizer\n343\nmsoedov/langcorn\n334\nshowlab/VLog\n330\ncontinuum-llms/chatgpt-memory\n324\nsteamship-core/steamship-langchain\n323", "source": "https://python.langchain.com/en/latest/dependents.html"}8{"id": "8ea51f3894e1-3", "text": "324\nsteamship-core/steamship-langchain\n323\ndaodao97/chatdoc\n320\nxuwenhao/geektime-ai-course\n308\nStevenGrove/GPT4Tools\n301\nlogan-markewich/llama_index_starter_pack\n300\nandylokandy/gpt-4-search\n299\nAnil-matcha/ChatPDF\n287\nitamargol/openai\n273\nBlackHC/llm-strategy\n267\nmomegas/megabots\n259\nbborn/howdoi.ai\n238\nCheems-Seminar/grounded-segment-any-parts\n232\nur-whitelab/exmol\n227\nsullivan-sean/chat-langchainjs\n227\nexplosion/spacy-llm\n226\nrecalign/RecAlign\n218\njupyterlab/jupyter-ai\n218\nalvarosevilla95/autolang\n215\nconceptofmind/toolformer\n213\nMagnivOrg/prompt-layer-library\n209\nJohnSnowLabs/nlptest\n208\nairobotlab/KoChatGPT\n197\nlangchain-ai/auto-evaluator\n195\nyvann-hub/Robby-chatbot\n195\nalejandro-ao/langchain-ask-pdf\n192\ndaveebbelaar/langchain-experiments\n189\nNimbleBoxAI/ChainFury\n187\nkaleido-lab/dolphin\n184\nAnil-matcha/Website-to-Chatbot\n183\nplchld/InsightFlow\n180\nOpenBMB/AgentVerse\n166\nbenthecoder/ClassGPT\n166\njbrukh/gpt-jargon\n161\nhardbyte/qabot\n160\nshaman-ai/agent-actors\n153\nradi-cho/datasetGPT\n153\npoe-platform/poe-protocol\n152", "source": "https://python.langchain.com/en/latest/dependents.html"}9{"id": "8ea51f3894e1-4", "text": "radi-cho/datasetGPT\n153\npoe-platform/poe-protocol\n152\npaolorechia/learn-langchain\n149\najndkr/lanarky\n149\nfengyuli-dev/multimedia-gpt\n147\nyasyf/compress-gpt\n144\nhomanp/superagent\n143\nrealminchoi/babyagi-ui\n141\nethanyanjiali/minChatGPT\n141\nccurme/yolopandas\n139\nhwchase17/langchain-streamlit-template\n138\nJaseci-Labs/jaseci\n136\nhirokidaichi/wanna\n135\nHaste171/langchain-chatbot\n134\njmpaz/promptlib\n130\nKlingefjord/chatgpt-telegram\n130\nfilip-michalsky/SalesGPT\n128\nhandrew/browserpilot\n128\nshauryr/S2QA\n127\nsteamship-core/vercel-examples\n127\nyasyf/summ\n127\ngia-guar/JARVIS-ChatGPT\n126\njerlendds/osintbuddy\n125\nibiscp/LLM-IMDB\n124\nTeahouse-Studios/akari-bot\n124\nhwchase17/chroma-langchain\n124\nmenloparklab/langchain-cohere-qdrant-doc-retrieval\n123\npeterw/StoryStorm\n123\nchakkaradeep/pyCodeAGI\n123\npetehunt/langchain-github-bot\n115\nsu77ungr/CASALIOY\n113\neunomia-bpf/GPTtrace\n113\nzenml-io/zenml-projects\n112\npablomarin/GPT-Azure-Search-Engine\n111\nshamspias/customizable-gpt-chatbot\n109\nWongSaang/chatgpt-ui-server\n108", "source": "https://python.langchain.com/en/latest/dependents.html"}10{"id": "8ea51f3894e1-5", "text": "109\nWongSaang/chatgpt-ui-server\n108\ndavila7/file-gpt\n104\nenhancedocs/enhancedocs\n102\naurelio-labs/arxiv-bot\n101\nGenerated by github-dependents-info\n[github-dependents-info \u2013repo hwchase17/langchain \u2013markdownfile dependents.md \u2013minstars 100 \u2013sort stars]\nprevious\nZilliz\nnext\nDeployments\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/dependents.html"}11{"id": "94d1235acb1c-0", "text": ".rst\n.pdf\nAPI References\nAPI References#\nFull documentation on all methods, classes, and APIs in LangChain.\nModels\nPrompts\nIndexes\nMemory\nChains\nAgents\nUtilities\nExperimental Modules\nprevious\nInstallation\nnext\nModels\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/reference.html"}12{"id": "a08f07fe3e75-0", "text": ".rst\n.pdf\nIntegrations\n Contents \nIntegrations by Module\nAll Integrations\nIntegrations#\nLangChain integrates with many LLMs, systems, and products.\nIntegrations by Module#\nIntegrations grouped by the core LangChain module they map to:\nLLM Providers\nChat Model Providers\nText Embedding Model Providers\nDocument Loader Integrations\nText Splitter Integrations\nVectorstore Providers\nRetriever Providers\nTool Providers\nToolkit Integrations\nAll Integrations#\nA comprehensive list of LLMs, systems, and products integrated with LangChain:\nAI21 Labs\nAim\nAnalyticDB\nAnyscale\nApify\nAtlasDB\nBanana\nBeam\nCerebriumAI\nChroma\nClearML Integration\nCohere\nComet\nC Transformers\nDataberry\nDatabricks\nDeepInfra\nDeep Lake\nDocugami\nAdvantages vs Other Chunking Techniques\nForefrontAI\nGoogle Search\nGoogle Serper\nGooseAI\nGPT4All\nGraphsignal\nHazy Research\nHelicone\nHugging Face\nJina\nLanceDB\nLlama.cpp\nMetal\nMilvus\nMLflow\nModal\nMomento\nMyScale\nNLPCloud\nOpenAI\nOpenSearch\nOpenWeatherMap API\nPetals\nPGVector\nPinecone\nPipelineAI\nPrediction Guard\nPromptLayer\nPsychic\nAdvantages vs Other Document Loaders\nQdrant\nRebuff: Prompt Injection Detection with LangChain\nRedis\nReplicate\nRunhouse\nRWKV-4\nSearxNG Search API\nSerpAPI\nscikit-learn\nStochasticAI\nTair\nUnstructured\nVectara\nWeights & Biases\nWeaviate\nWhyLabs Integration\nWolfram Alpha Wrapper\nWriter\nYeager.ai\nZilliz\nprevious\nExperimental Modules", "source": "https://python.langchain.com/en/latest/integrations.html"}13{"id": "a08f07fe3e75-1", "text": "Wolfram Alpha Wrapper\nWriter\nYeager.ai\nZilliz\nprevious\nExperimental Modules\nnext\nAI21 Labs\n Contents\n  \nIntegrations by Module\nAll Integrations\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/integrations.html"}14{"id": "eddc1cc06a3b-0", "text": "Index\n_\n | A\n | B\n | C\n | D\n | E\n | F\n | G\n | H\n | I\n | J\n | K\n | L\n | M\n | N\n | O\n | P\n | Q\n | R\n | S\n | T\n | U\n | V\n | W\n | Y\n | Z\n_\n__call__() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)", "source": "https://python.langchain.com/en/latest/genindex.html"}15{"id": "eddc1cc06a3b-1", "text": "(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nA\naadd_documents() (langchain.retrievers.TimeWeightedVectorStoreRetriever method)\n(langchain.vectorstores.VectorStore method)\naadd_texts() (langchain.vectorstores.VectorStore method)\naapply() (langchain.chains.LLMChain method)\naapply_and_parse() (langchain.chains.LLMChain method)\nacall_actor() (langchain.utilities.ApifyWrapper method)\naccess_token (langchain.document_loaders.DocugamiLoader attribute)\naccount_sid (langchain.utilities.TwilioAPIWrapper attribute)\nacompress_documents() (langchain.retrievers.document_compressors.CohereRerank method)\n(langchain.retrievers.document_compressors.DocumentCompressorPipeline method)\n(langchain.retrievers.document_compressors.EmbeddingsFilter method)\n(langchain.retrievers.document_compressors.LLMChainExtractor method)\n(langchain.retrievers.document_compressors.LLMChainFilter method)\naction_id (langchain.tools.ZapierNLARunAction attribute)\nadd() (langchain.docstore.InMemoryDocstore method)\nadd_ai_message() (langchain.memory.CassandraChatMessageHistory method)", "source": "https://python.langchain.com/en/latest/genindex.html"}16{"id": "eddc1cc06a3b-2", "text": "add_ai_message() (langchain.memory.CassandraChatMessageHistory method)\n(langchain.memory.ChatMessageHistory method)\n(langchain.memory.CosmosDBChatMessageHistory method)\n(langchain.memory.DynamoDBChatMessageHistory method)\n(langchain.memory.FileChatMessageHistory method)\n(langchain.memory.MomentoChatMessageHistory method)\n(langchain.memory.MongoDBChatMessageHistory method)\n(langchain.memory.PostgresChatMessageHistory method)\n(langchain.memory.RedisChatMessageHistory method)\nadd_documents() (langchain.retrievers.TimeWeightedVectorStoreRetriever method)\n(langchain.retrievers.WeaviateHybridSearchRetriever method)\n(langchain.vectorstores.VectorStore method)\nadd_embeddings() (langchain.vectorstores.FAISS method)\nadd_example() (langchain.prompts.example_selector.LengthBasedExampleSelector method)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector method)\nadd_memory() (langchain.experimental.GenerativeAgentMemory method)\nadd_texts() (langchain.retrievers.ElasticSearchBM25Retriever method)\n(langchain.retrievers.PineconeHybridSearchRetriever method)\n(langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Annoy method)\n(langchain.vectorstores.AtlasDB method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.ElasticVectorSearch method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.LanceDB method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.MyScale method)\n(langchain.vectorstores.OpenSearchVectorSearch method)\n(langchain.vectorstores.Pinecone method)\n(langchain.vectorstores.Qdrant method)\n(langchain.vectorstores.Redis method)\n(langchain.vectorstores.SKLearnVectorStore method)", "source": "https://python.langchain.com/en/latest/genindex.html"}17{"id": "eddc1cc06a3b-3", "text": "(langchain.vectorstores.Redis method)\n(langchain.vectorstores.SKLearnVectorStore method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.Tair method)\n(langchain.vectorstores.Typesense method)\n(langchain.vectorstores.Vectara method)\n(langchain.vectorstores.VectorStore method)\n(langchain.vectorstores.Weaviate method)\nadd_user_message() (langchain.memory.CassandraChatMessageHistory method)\n(langchain.memory.ChatMessageHistory method)\n(langchain.memory.CosmosDBChatMessageHistory method)\n(langchain.memory.DynamoDBChatMessageHistory method)\n(langchain.memory.FileChatMessageHistory method)\n(langchain.memory.MomentoChatMessageHistory method)\n(langchain.memory.MongoDBChatMessageHistory method)\n(langchain.memory.PostgresChatMessageHistory method)\n(langchain.memory.RedisChatMessageHistory method)\nadd_vectors() (langchain.vectorstores.SupabaseVectorStore method)\nadd_video_info (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\nadelete() (langchain.utilities.TextRequestsWrapper method)\nafrom_documents() (langchain.vectorstores.VectorStore class method)\nafrom_texts() (langchain.vectorstores.VectorStore class method)\nage (langchain.experimental.GenerativeAgent attribute)\nagenerate() (langchain.chains.LLMChain method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)", "source": "https://python.langchain.com/en/latest/genindex.html"}18{"id": "eddc1cc06a3b-4", "text": "(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nagenerate_prompt() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)", "source": "https://python.langchain.com/en/latest/genindex.html"}19{"id": "eddc1cc06a3b-5", "text": "(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)", "source": "https://python.langchain.com/en/latest/genindex.html"}20{"id": "eddc1cc06a3b-6", "text": "(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nagent (langchain.agents.AgentExecutor attribute)\nAgentType (class in langchain.agents)\naget() (langchain.utilities.TextRequestsWrapper method)\naget_relevant_documents() (langchain.retrievers.ArxivRetriever method)\n(langchain.retrievers.AzureCognitiveSearchRetriever method)\n(langchain.retrievers.ChatGPTPluginRetriever method)\n(langchain.retrievers.ContextualCompressionRetriever method)\n(langchain.retrievers.DataberryRetriever method)\n(langchain.retrievers.ElasticSearchBM25Retriever method)\n(langchain.retrievers.KNNRetriever method)\n(langchain.retrievers.MetalRetriever method)\n(langchain.retrievers.PineconeHybridSearchRetriever method)\n(langchain.retrievers.RemoteLangChainRetriever method)\n(langchain.retrievers.SelfQueryRetriever method)\n(langchain.retrievers.SVMRetriever method)\n(langchain.retrievers.TFIDFRetriever method)\n(langchain.retrievers.TimeWeightedVectorStoreRetriever method)\n(langchain.retrievers.VespaRetriever method)\n(langchain.retrievers.WeaviateHybridSearchRetriever method)\n(langchain.retrievers.WikipediaRetriever method)\n(langchain.retrievers.ZepRetriever method)\naget_table_info() (langchain.utilities.PowerBIDataset method)\naggregate_importance (langchain.experimental.GenerativeAgentMemory attribute)\nai_prefix (langchain.agents.ConversationalAgent attribute)\n(langchain.memory.ConversationBufferMemory attribute)\n(langchain.memory.ConversationBufferWindowMemory attribute)\n(langchain.memory.ConversationEntityMemory attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}21{"id": "eddc1cc06a3b-7", "text": "(langchain.memory.ConversationBufferWindowMemory attribute)\n(langchain.memory.ConversationEntityMemory attribute)\n(langchain.memory.ConversationKGMemory attribute)\n(langchain.memory.ConversationStringBufferMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory attribute)\naiosession (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\n(langchain.retrievers.ChatGPTPluginRetriever attribute)\n(langchain.serpapi.SerpAPIWrapper attribute)\n(langchain.utilities.GoogleSerperAPIWrapper attribute)\n(langchain.utilities.PowerBIDataset attribute)\n(langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\n(langchain.utilities.SerpAPIWrapper attribute)\n(langchain.utilities.TextRequestsWrapper attribute)\nAirbyteJSONLoader (class in langchain.document_loaders)\naleph_alpha_api_key (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\n(langchain.llms.AlephAlpha attribute)\nallowed_special (langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\nallowed_tools (langchain.agents.Agent attribute)\naload() (langchain.document_loaders.WebBaseLoader method)\nalpha (langchain.retrievers.PineconeHybridSearchRetriever attribute)\namax_marginal_relevance_search() (langchain.vectorstores.VectorStore method)\namax_marginal_relevance_search_by_vector() (langchain.vectorstores.VectorStore method)\nAnalyticDB (class in langchain.vectorstores)\nAnnoy (class in langchain.vectorstores)\nanswers (langchain.utilities.searx_search.SearxResults property)\napatch() (langchain.utilities.TextRequestsWrapper method)", "source": "https://python.langchain.com/en/latest/genindex.html"}22{"id": "eddc1cc06a3b-8", "text": "apatch() (langchain.utilities.TextRequestsWrapper method)\napi (langchain.document_loaders.DocugamiLoader attribute)\napi_answer_chain (langchain.chains.APIChain attribute)\napi_docs (langchain.chains.APIChain attribute)\napi_key (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\n(langchain.retrievers.DataberryRetriever attribute)\napi_operation (langchain.chains.OpenAPIEndpointChain attribute)\napi_request_chain (langchain.chains.APIChain attribute)\n(langchain.chains.OpenAPIEndpointChain attribute)\napi_resource (langchain.agents.agent_toolkits.GmailToolkit attribute)\napi_response_chain (langchain.chains.OpenAPIEndpointChain attribute)\napi_spec (langchain.tools.AIPluginTool attribute)\napi_token (langchain.llms.Databricks attribute)\napi_url (langchain.llms.StochasticAI attribute)\napi_version (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\napi_wrapper (langchain.tools.BingSearchResults attribute)\n(langchain.tools.BingSearchRun attribute)\n(langchain.tools.DuckDuckGoSearchResults attribute)\n(langchain.tools.DuckDuckGoSearchRun attribute)\n(langchain.tools.GooglePlacesTool attribute)\n(langchain.tools.GoogleSearchResults attribute)\n(langchain.tools.GoogleSearchRun attribute)\n(langchain.tools.GoogleSerperResults attribute)\n(langchain.tools.GoogleSerperRun attribute)\n(langchain.tools.MetaphorSearchResults attribute)\n(langchain.tools.OpenWeatherMapQueryRun attribute)\n(langchain.tools.SceneXplainTool attribute)\n(langchain.tools.WikipediaQueryRun attribute)\n(langchain.tools.WolframAlphaQueryRun attribute)\n(langchain.tools.ZapierNLAListActions attribute)\n(langchain.tools.ZapierNLARunAction attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}23{"id": "eddc1cc06a3b-9", "text": "(langchain.tools.ZapierNLARunAction attribute)\napify_client (langchain.document_loaders.ApifyDatasetLoader attribute)\n(langchain.utilities.ApifyWrapper attribute)\napify_client_async (langchain.utilities.ApifyWrapper attribute)\naplan() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\n(langchain.agents.LLMSingleActionAgent method)\napost() (langchain.utilities.TextRequestsWrapper method)\napp_creation() (langchain.llms.Beam method)\nappend() (langchain.memory.CassandraChatMessageHistory method)\n(langchain.memory.DynamoDBChatMessageHistory method)\n(langchain.memory.FileChatMessageHistory method)\n(langchain.memory.MongoDBChatMessageHistory method)\n(langchain.memory.PostgresChatMessageHistory method)\n(langchain.memory.RedisChatMessageHistory method)\napply() (langchain.chains.LLMChain method)\napply_and_parse() (langchain.chains.LLMChain method)\napredict() (langchain.chains.LLMChain method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)", "source": "https://python.langchain.com/en/latest/genindex.html"}24{"id": "eddc1cc06a3b-10", "text": "(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\napredict_and_parse() (langchain.chains.LLMChain method)\napredict_messages() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)", "source": "https://python.langchain.com/en/latest/genindex.html"}25{"id": "eddc1cc06a3b-11", "text": "(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\naprep_prompts() (langchain.chains.LLMChain method)", "source": "https://python.langchain.com/en/latest/genindex.html"}26{"id": "eddc1cc06a3b-12", "text": "aprep_prompts() (langchain.chains.LLMChain method)\naput() (langchain.utilities.TextRequestsWrapper method)\narbitrary_types_allowed (langchain.experimental.BabyAGI.Config attribute)\n(langchain.experimental.GenerativeAgent.Config attribute)\n(langchain.retrievers.WeaviateHybridSearchRetriever.Config attribute)\nare_all_true_prompt (langchain.chains.LLMSummarizationCheckerChain attribute)\naresults() (langchain.serpapi.SerpAPIWrapper method)\n(langchain.utilities.GoogleSerperAPIWrapper method)\n(langchain.utilities.searx_search.SearxSearchWrapper method)\n(langchain.utilities.SearxSearchWrapper method)\n(langchain.utilities.SerpAPIWrapper method)\nargs (langchain.agents.Tool property)\n(langchain.tools.BaseTool property)\n(langchain.tools.StructuredTool property)\n(langchain.tools.Tool property)\nargs_schema (langchain.tools.AIPluginTool attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.ClickTool attribute)\n(langchain.tools.CopyFileTool attribute)\n(langchain.tools.CurrentWebPageTool attribute)\n(langchain.tools.DeleteFileTool attribute)\n(langchain.tools.ExtractHyperlinksTool attribute)\n(langchain.tools.ExtractTextTool attribute)\n(langchain.tools.FileSearchTool attribute)\n(langchain.tools.GetElementsTool attribute)\n(langchain.tools.GmailCreateDraft attribute)\n(langchain.tools.GmailGetMessage attribute)\n(langchain.tools.GmailGetThread attribute)\n(langchain.tools.GmailSearch attribute)\n(langchain.tools.ListDirectoryTool attribute)\n(langchain.tools.MoveFileTool attribute)\n(langchain.tools.NavigateBackTool attribute)\n(langchain.tools.NavigateTool attribute)\n(langchain.tools.ReadFileTool attribute)\n(langchain.tools.ShellTool attribute)\n(langchain.tools.StructuredTool attribute)\n(langchain.tools.Tool attribute)\n(langchain.tools.WriteFileTool attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}27{"id": "eddc1cc06a3b-13", "text": "(langchain.tools.Tool attribute)\n(langchain.tools.WriteFileTool attribute)\narun() (langchain.serpapi.SerpAPIWrapper method)\n(langchain.tools.BaseTool method)\n(langchain.utilities.GoogleSerperAPIWrapper method)\n(langchain.utilities.PowerBIDataset method)\n(langchain.utilities.searx_search.SearxSearchWrapper method)\n(langchain.utilities.SearxSearchWrapper method)\n(langchain.utilities.SerpAPIWrapper method)\narxiv_exceptions (langchain.utilities.ArxivAPIWrapper attribute)\nArxivLoader (class in langchain.document_loaders)\nas_retriever() (langchain.vectorstores.Redis method)\n(langchain.vectorstores.Vectara method)\n(langchain.vectorstores.VectorStore method)\nasearch() (langchain.vectorstores.VectorStore method)\nasimilarity_search() (langchain.vectorstores.VectorStore method)\nasimilarity_search_by_vector() (langchain.vectorstores.VectorStore method)\nasimilarity_search_with_relevance_scores() (langchain.vectorstores.VectorStore method)\nasync_browser (langchain.agents.agent_toolkits.PlayWrightBrowserToolkit attribute)\nAtlasDB (class in langchain.vectorstores)\natransform_documents() (langchain.document_transformers.EmbeddingsRedundantFilter method)\n(langchain.text_splitter.TextSplitter method)\nauth_token (langchain.utilities.TwilioAPIWrapper attribute)\nauth_with_token (langchain.document_loaders.OneDriveLoader attribute)\nAutoGPT (class in langchain.experimental)\nawslambda_tool_description (langchain.utilities.LambdaWrapper attribute)\nawslambda_tool_name (langchain.utilities.LambdaWrapper attribute)\nAZLyricsLoader (class in langchain.document_loaders)\nAzureBlobStorageContainerLoader (class in langchain.document_loaders)\nAzureBlobStorageFileLoader (class in langchain.document_loaders)\nB", "source": "https://python.langchain.com/en/latest/genindex.html"}28{"id": "eddc1cc06a3b-14", "text": "AzureBlobStorageFileLoader (class in langchain.document_loaders)\nB\nBabyAGI (class in langchain.experimental)\nbad_words (langchain.llms.NLPCloud attribute)\nbase_compressor (langchain.retrievers.ContextualCompressionRetriever attribute)\nbase_embeddings (langchain.chains.HypotheticalDocumentEmbedder attribute)\nbase_prompt (langchain.tools.ZapierNLARunAction attribute)\nbase_retriever (langchain.retrievers.ContextualCompressionRetriever attribute)\nbase_url (langchain.document_loaders.BlackboardLoader attribute)\n(langchain.llms.AI21 attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.Writer attribute)\n(langchain.tools.APIOperation attribute)\n(langchain.tools.OpenAPISpec property)\nBashProcess (class in langchain.utilities)\nbatch_size (langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\nbearer_token (langchain.retrievers.ChatGPTPluginRetriever attribute)\nbest_of (langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Writer attribute)\nBibtexLoader (class in langchain.document_loaders)\nBigQueryLoader (class in langchain.document_loaders)\nBiliBiliLoader (class in langchain.document_loaders)\nbinary_location (langchain.document_loaders.SeleniumURLLoader attribute)\nbing_search_url (langchain.utilities.BingSearchAPIWrapper attribute)\nbing_subscription_key (langchain.utilities.BingSearchAPIWrapper attribute)\nBlackboardLoader (class in langchain.document_loaders)\nBlockchainDocumentLoader (class in langchain.document_loaders)", "source": "https://python.langchain.com/en/latest/genindex.html"}29{"id": "eddc1cc06a3b-15", "text": "BlockchainDocumentLoader (class in langchain.document_loaders)\nbody_params (langchain.tools.APIOperation property)\nbrowser (langchain.document_loaders.SeleniumURLLoader attribute)\nBSHTMLLoader (class in langchain.document_loaders)\nbuffer (langchain.memory.ConversationBufferMemory property)\n(langchain.memory.ConversationBufferWindowMemory property)\n(langchain.memory.ConversationEntityMemory property)\n(langchain.memory.ConversationStringBufferMemory attribute)\n(langchain.memory.ConversationSummaryBufferMemory property)\n(langchain.memory.ConversationSummaryMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory property)\nC\ncache_folder (langchain.embeddings.HuggingFaceEmbeddings attribute)\n(langchain.embeddings.HuggingFaceInstructEmbeddings attribute)\ncall_actor() (langchain.utilities.ApifyWrapper method)\ncallback_manager (langchain.agents.agent_toolkits.PowerBIToolkit attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.Tool attribute)\ncallbacks (langchain.tools.BaseTool attribute)\n(langchain.tools.Tool attribute)\ncaptions_language (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\nCassandraChatMessageHistory (class in langchain.memory)\ncategories (langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nchain (langchain.chains.ConstitutionalChain attribute)\nchains (langchain.chains.SequentialChain attribute)\n(langchain.chains.SimpleSequentialChain attribute)\nchannel_name (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\nCharacterTextSplitter (class in langchain.text_splitter)\nCHAT_CONVERSATIONAL_REACT_DESCRIPTION (langchain.agents.AgentType attribute)\nchat_history_key (langchain.memory.ConversationEntityMemory attribute)\nCHAT_ZERO_SHOT_REACT_DESCRIPTION (langchain.agents.AgentType attribute)\nChatGPTLoader (class in langchain.document_loaders)", "source": "https://python.langchain.com/en/latest/genindex.html"}30{"id": "eddc1cc06a3b-16", "text": "ChatGPTLoader (class in langchain.document_loaders)\ncheck_assertions_prompt (langchain.chains.LLMCheckerChain attribute)\n(langchain.chains.LLMSummarizationCheckerChain attribute)\ncheck_bs4() (langchain.document_loaders.BlackboardLoader method)\nChroma (class in langchain.vectorstores)\nCHUNK_LEN (langchain.llms.RWKV attribute)\nchunk_size (langchain.embeddings.OpenAIEmbeddings attribute)\nclean_pdf() (langchain.document_loaders.MathpixPDFLoader method)\nclear() (langchain.experimental.GenerativeAgentMemory method)\n(langchain.memory.CassandraChatMessageHistory method)\n(langchain.memory.ChatMessageHistory method)\n(langchain.memory.CombinedMemory method)\n(langchain.memory.ConversationEntityMemory method)\n(langchain.memory.ConversationKGMemory method)\n(langchain.memory.ConversationStringBufferMemory method)\n(langchain.memory.ConversationSummaryBufferMemory method)\n(langchain.memory.ConversationSummaryMemory method)\n(langchain.memory.CosmosDBChatMessageHistory method)\n(langchain.memory.DynamoDBChatMessageHistory method)\n(langchain.memory.FileChatMessageHistory method)\n(langchain.memory.InMemoryEntityStore method)\n(langchain.memory.MomentoChatMessageHistory method)\n(langchain.memory.MongoDBChatMessageHistory method)\n(langchain.memory.PostgresChatMessageHistory method)\n(langchain.memory.ReadOnlySharedMemory method)\n(langchain.memory.RedisChatMessageHistory method)\n(langchain.memory.RedisEntityStore method)\n(langchain.memory.SimpleMemory method)\n(langchain.memory.VectorStoreRetrieverMemory method)\nclient (langchain.llms.Petals attribute)\n(langchain.retrievers.document_compressors.CohereRerank attribute)\ncluster_driver_port (langchain.llms.Databricks attribute)\ncluster_id (langchain.llms.Databricks attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}31{"id": "eddc1cc06a3b-17", "text": "cluster_id (langchain.llms.Databricks attribute)\nCollegeConfidentialLoader (class in langchain.document_loaders)\ncolumn_map (langchain.vectorstores.MyScaleSettings attribute)\ncombine_docs_chain (langchain.chains.AnalyzeDocumentChain attribute)\ncombine_documents_chain (langchain.chains.MapReduceChain attribute)\ncombine_embeddings() (langchain.chains.HypotheticalDocumentEmbedder method)\ncompletion_bias_exclusion_first_token_only (langchain.llms.AlephAlpha attribute)\ncompletion_with_retry() (langchain.chat_models.ChatOpenAI method)\ncompress_documents() (langchain.retrievers.document_compressors.CohereRerank method)\n(langchain.retrievers.document_compressors.DocumentCompressorPipeline method)\n(langchain.retrievers.document_compressors.EmbeddingsFilter method)\n(langchain.retrievers.document_compressors.LLMChainExtractor method)\n(langchain.retrievers.document_compressors.LLMChainFilter method)\ncompress_to_size (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\nconfig (langchain.llms.CTransformers attribute)\nConfluenceLoader (class in langchain.document_loaders)\nCoNLLULoader (class in langchain.document_loaders)\nconnect() (langchain.vectorstores.AnalyticDB method)\nconnection_string_from_db_params() (langchain.vectorstores.AnalyticDB class method)\nconstitutional_principles (langchain.chains.ConstitutionalChain attribute)\nconstruct() (langchain.llms.AI21 class method)\n(langchain.llms.AlephAlpha class method)\n(langchain.llms.Anthropic class method)\n(langchain.llms.Anyscale class method)\n(langchain.llms.AzureOpenAI class method)\n(langchain.llms.Banana class method)\n(langchain.llms.Beam class method)\n(langchain.llms.CerebriumAI class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}32{"id": "eddc1cc06a3b-18", "text": "(langchain.llms.CerebriumAI class method)\n(langchain.llms.Cohere class method)\n(langchain.llms.CTransformers class method)\n(langchain.llms.Databricks class method)\n(langchain.llms.DeepInfra class method)\n(langchain.llms.FakeListLLM class method)\n(langchain.llms.ForefrontAI class method)\n(langchain.llms.GooglePalm class method)\n(langchain.llms.GooseAI class method)\n(langchain.llms.GPT4All class method)\n(langchain.llms.HuggingFaceEndpoint class method)\n(langchain.llms.HuggingFaceHub class method)\n(langchain.llms.HuggingFacePipeline class method)\n(langchain.llms.HuggingFaceTextGenInference class method)\n(langchain.llms.HumanInputLLM class method)\n(langchain.llms.LlamaCpp class method)\n(langchain.llms.Modal class method)\n(langchain.llms.MosaicML class method)\n(langchain.llms.NLPCloud class method)\n(langchain.llms.OpenAI class method)\n(langchain.llms.OpenAIChat class method)\n(langchain.llms.OpenLM class method)\n(langchain.llms.Petals class method)\n(langchain.llms.PipelineAI class method)\n(langchain.llms.PredictionGuard class method)\n(langchain.llms.PromptLayerOpenAI class method)\n(langchain.llms.PromptLayerOpenAIChat class method)\n(langchain.llms.Replicate class method)\n(langchain.llms.RWKV class method)\n(langchain.llms.SagemakerEndpoint class method)\n(langchain.llms.SelfHostedHuggingFaceLLM class method)\n(langchain.llms.SelfHostedPipeline class method)\n(langchain.llms.StochasticAI class method)\n(langchain.llms.VertexAI class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}33{"id": "eddc1cc06a3b-19", "text": "(langchain.llms.StochasticAI class method)\n(langchain.llms.VertexAI class method)\n(langchain.llms.Writer class method)\ncontent_handler (langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.SagemakerEndpoint attribute)\ncontent_key (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\nCONTENT_KEY (langchain.vectorstores.Qdrant attribute)\ncontext_erase (langchain.llms.GPT4All attribute)\ncontextual_control_threshold (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\n(langchain.llms.AlephAlpha attribute)\ncontinue_on_failure (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\n(langchain.document_loaders.PlaywrightURLLoader attribute)\n(langchain.document_loaders.SeleniumURLLoader attribute)\ncontrol_log_additive (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\n(langchain.llms.AlephAlpha attribute)\nCONVERSATIONAL_REACT_DESCRIPTION (langchain.agents.AgentType attribute)\ncopy() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)", "source": "https://python.langchain.com/en/latest/genindex.html"}34{"id": "eddc1cc06a3b-20", "text": "(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\ncoroutine (langchain.agents.Tool attribute)\n(langchain.tools.StructuredTool attribute)\n(langchain.tools.Tool attribute)\nCosmosDBChatMessageHistory (class in langchain.memory)\ncountPenalty (langchain.llms.AI21 attribute)\ncreate() (langchain.retrievers.ElasticSearchBM25Retriever class method)\ncreate_assertions_prompt (langchain.chains.LLMSummarizationCheckerChain attribute)\ncreate_collection() (langchain.vectorstores.AnalyticDB method)", "source": "https://python.langchain.com/en/latest/genindex.html"}35{"id": "eddc1cc06a3b-21", "text": "create_collection() (langchain.vectorstores.AnalyticDB method)\ncreate_csv_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_documents() (langchain.text_splitter.TextSplitter method)\ncreate_draft_answer_prompt (langchain.chains.LLMCheckerChain attribute)\ncreate_index() (langchain.vectorstores.AtlasDB method)\ncreate_index_if_not_exist() (langchain.vectorstores.Tair method)\ncreate_json_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_llm_result() (langchain.llms.AzureOpenAI method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\ncreate_openapi_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_outputs() (langchain.chains.LLMChain method)\ncreate_pandas_dataframe_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_pbi_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_pbi_chat_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_prompt() (langchain.agents.Agent class method)\n(langchain.agents.ConversationalAgent class method)\n(langchain.agents.ConversationalChatAgent class method)\n(langchain.agents.ReActTextWorldAgent class method)\n(langchain.agents.StructuredChatAgent class method)\n(langchain.agents.ZeroShotAgent class method)\ncreate_python_agent() (in module langchain.agents.agent_toolkits)\ncreate_spark_dataframe_agent() (in module langchain.agents)", "source": "https://python.langchain.com/en/latest/genindex.html"}36{"id": "eddc1cc06a3b-22", "text": "create_spark_dataframe_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_spark_sql_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_sql_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_tables_if_not_exists() (langchain.vectorstores.AnalyticDB method)\ncreate_vectorstore_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncreate_vectorstore_router_agent() (in module langchain.agents)\n(in module langchain.agents.agent_toolkits)\ncredential (langchain.utilities.PowerBIDataset attribute)\ncredentials (langchain.llms.VertexAI attribute)\ncredentials_path (langchain.document_loaders.GoogleApiClient attribute)\n(langchain.document_loaders.GoogleDriveLoader attribute)\ncredentials_profile_name (langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.SagemakerEndpoint attribute)\ncritique_chain (langchain.chains.ConstitutionalChain attribute)\nCSVLoader (class in langchain.document_loaders)\ncurrent_plan (langchain.experimental.GenerativeAgentMemory attribute)\ncustom_headers (langchain.utilities.GraphQLAPIWrapper attribute)\ncypher_generation_chain (langchain.chains.GraphCypherQAChain attribute)\nD\ndaily_summaries (langchain.experimental.GenerativeAgent attribute)\ndata (langchain.document_loaders.MathpixPDFLoader property)\ndatabase (langchain.chains.SQLDatabaseChain attribute)\n(langchain.vectorstores.MyScaleSettings attribute)\nDataberryRetriever (class in langchain.retrievers)\nDataFrameLoader (class in langchain.document_loaders)\ndataset_id (langchain.document_loaders.ApifyDatasetLoader attribute)\n(langchain.utilities.PowerBIDataset attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}37{"id": "eddc1cc06a3b-23", "text": "(langchain.utilities.PowerBIDataset attribute)\ndataset_mapping_function (langchain.document_loaders.ApifyDatasetLoader attribute)\ndatastore_url (langchain.retrievers.DataberryRetriever attribute)\ndb (langchain.agents.agent_toolkits.SparkSQLToolkit attribute)\n(langchain.agents.agent_toolkits.SQLDatabaseToolkit attribute)\ndecay_rate (langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\ndecider_chain (langchain.chains.SQLDatabaseSequentialChain attribute)\nDeepLake (class in langchain.vectorstores)\ndefault_output_key (langchain.output_parsers.RegexParser attribute)\ndefault_parser (langchain.document_loaders.WebBaseLoader attribute)\ndefault_request_timeout (langchain.llms.Anthropic attribute)\ndefault_salience (langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\ndelete() (langchain.memory.InMemoryEntityStore method)\n(langchain.memory.RedisEntityStore method)\n(langchain.utilities.TextRequestsWrapper method)\n(langchain.vectorstores.DeepLake method)\ndelete_collection() (langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Chroma method)\ndelete_dataset() (langchain.vectorstores.DeepLake method)\ndeployment_name (langchain.chat_models.AzureChatOpenAI attribute)\n(langchain.llms.AzureOpenAI attribute)\ndescription (langchain.agents.agent_toolkits.VectorStoreInfo attribute)\n(langchain.agents.Tool attribute)\n(langchain.output_parsers.ResponseSchema attribute)\n(langchain.tools.APIOperation attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.ClickTool attribute)\n(langchain.tools.CopyFileTool attribute)\n(langchain.tools.CurrentWebPageTool attribute)\n(langchain.tools.DeleteFileTool attribute)\n(langchain.tools.ExtractHyperlinksTool attribute)\n(langchain.tools.ExtractTextTool attribute)\n(langchain.tools.FileSearchTool attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}38{"id": "eddc1cc06a3b-24", "text": "(langchain.tools.ExtractTextTool attribute)\n(langchain.tools.FileSearchTool attribute)\n(langchain.tools.GetElementsTool attribute)\n(langchain.tools.GmailCreateDraft attribute)\n(langchain.tools.GmailGetMessage attribute)\n(langchain.tools.GmailGetThread attribute)\n(langchain.tools.GmailSearch attribute)\n(langchain.tools.GmailSendMessage attribute)\n(langchain.tools.ListDirectoryTool attribute)\n(langchain.tools.MoveFileTool attribute)\n(langchain.tools.NavigateBackTool attribute)\n(langchain.tools.NavigateTool attribute)\n(langchain.tools.ReadFileTool attribute)\n(langchain.tools.ShellTool attribute)\n(langchain.tools.StructuredTool attribute)\n(langchain.tools.Tool attribute)\n(langchain.tools.WriteFileTool attribute)\ndeserialize_json_input() (langchain.chains.OpenAPIEndpointChain method)\ndevice (langchain.llms.SelfHostedHuggingFaceLLM attribute)\ndialect (langchain.agents.agent_toolkits.SQLDatabaseToolkit property)\ndict() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\n(langchain.agents.LLMSingleActionAgent method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)", "source": "https://python.langchain.com/en/latest/genindex.html"}39{"id": "eddc1cc06a3b-25", "text": "(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\n(langchain.prompts.BasePromptTemplate method)\n(langchain.prompts.FewShotPromptTemplate method)\n(langchain.prompts.FewShotPromptWithTemplates method)\nDiffbotLoader (class in langchain.document_loaders)\nDirectoryLoader (class in langchain.document_loaders)\ndisallowed_special (langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}40{"id": "eddc1cc06a3b-26", "text": "(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\nDiscordChatLoader (class in langchain.document_loaders)\ndo_sample (langchain.llms.NLPCloud attribute)\n(langchain.llms.Petals attribute)\ndoc_content_chars_max (langchain.utilities.ArxivAPIWrapper attribute)\n(langchain.utilities.WikipediaAPIWrapper attribute)\nDocArrayHnswSearch (class in langchain.vectorstores)\nDocArrayInMemorySearch (class in langchain.vectorstores)\ndocs (langchain.retrievers.TFIDFRetriever attribute)\ndocset_id (langchain.document_loaders.DocugamiLoader attribute)\ndocument_ids (langchain.document_loaders.DocugamiLoader attribute)\n(langchain.document_loaders.GoogleDriveLoader attribute)\nDocx2txtLoader (class in langchain.document_loaders)\ndownload() (langchain.document_loaders.BlackboardLoader method)\ndrive_id (langchain.document_loaders.OneDriveLoader attribute)\ndrop() (langchain.vectorstores.MyScale method)\ndrop_index() (langchain.vectorstores.Redis static method)\n(langchain.vectorstores.Tair static method)\ndrop_tables() (langchain.vectorstores.AnalyticDB method)\nDuckDBLoader (class in langchain.document_loaders)\nDynamoDBChatMessageHistory (class in langchain.memory)\nE\nearly_stopping (langchain.llms.NLPCloud attribute)\nearly_stopping_method (langchain.agents.AgentExecutor attribute)\necho (langchain.llms.AlephAlpha attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nElasticSearchBM25Retriever (class in langchain.retrievers)", "source": "https://python.langchain.com/en/latest/genindex.html"}41{"id": "eddc1cc06a3b-27", "text": "ElasticSearchBM25Retriever (class in langchain.retrievers)\nElasticsearchEmbeddings (class in langchain.embeddings)\nElasticVectorSearch (class in langchain.vectorstores)\nembed_documents() (langchain.chains.HypotheticalDocumentEmbedder method)\n(langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding method)\n(langchain.embeddings.AlephAlphaSymmetricSemanticEmbedding method)\n(langchain.embeddings.CohereEmbeddings method)\n(langchain.embeddings.ElasticsearchEmbeddings method)\n(langchain.embeddings.FakeEmbeddings method)\n(langchain.embeddings.HuggingFaceEmbeddings method)\n(langchain.embeddings.HuggingFaceHubEmbeddings method)\n(langchain.embeddings.HuggingFaceInstructEmbeddings method)\n(langchain.embeddings.LlamaCppEmbeddings method)\n(langchain.embeddings.MiniMaxEmbeddings method)\n(langchain.embeddings.ModelScopeEmbeddings method)\n(langchain.embeddings.MosaicMLInstructorEmbeddings method)\n(langchain.embeddings.OpenAIEmbeddings method)\n(langchain.embeddings.SagemakerEndpointEmbeddings method)\n(langchain.embeddings.SelfHostedEmbeddings method)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings method)\n(langchain.embeddings.TensorflowHubEmbeddings method)\nembed_instruction (langchain.embeddings.HuggingFaceInstructEmbeddings attribute)\n(langchain.embeddings.MosaicMLInstructorEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings attribute)\nembed_query() (langchain.chains.HypotheticalDocumentEmbedder method)\n(langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding method)\n(langchain.embeddings.AlephAlphaSymmetricSemanticEmbedding method)\n(langchain.embeddings.CohereEmbeddings method)", "source": "https://python.langchain.com/en/latest/genindex.html"}42{"id": "eddc1cc06a3b-28", "text": "(langchain.embeddings.CohereEmbeddings method)\n(langchain.embeddings.ElasticsearchEmbeddings method)\n(langchain.embeddings.FakeEmbeddings method)\n(langchain.embeddings.HuggingFaceEmbeddings method)\n(langchain.embeddings.HuggingFaceHubEmbeddings method)\n(langchain.embeddings.HuggingFaceInstructEmbeddings method)\n(langchain.embeddings.LlamaCppEmbeddings method)\n(langchain.embeddings.MiniMaxEmbeddings method)\n(langchain.embeddings.ModelScopeEmbeddings method)\n(langchain.embeddings.MosaicMLInstructorEmbeddings method)\n(langchain.embeddings.OpenAIEmbeddings method)\n(langchain.embeddings.SagemakerEndpointEmbeddings method)\n(langchain.embeddings.SelfHostedEmbeddings method)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings method)\n(langchain.embeddings.TensorflowHubEmbeddings method)\nembed_type_db (langchain.embeddings.MiniMaxEmbeddings attribute)\nembed_type_query (langchain.embeddings.MiniMaxEmbeddings attribute)\nembedding (langchain.llms.GPT4All attribute)\nembeddings (langchain.document_transformers.EmbeddingsRedundantFilter attribute)\n(langchain.retrievers.document_compressors.EmbeddingsFilter attribute)\n(langchain.retrievers.KNNRetriever attribute)\n(langchain.retrievers.PineconeHybridSearchRetriever attribute)\n(langchain.retrievers.SVMRetriever attribute)\nencode_kwargs (langchain.embeddings.HuggingFaceEmbeddings attribute)\nendpoint_kwargs (langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.SagemakerEndpoint attribute)\nendpoint_name (langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.Databricks attribute)\n(langchain.llms.SagemakerEndpoint attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}43{"id": "eddc1cc06a3b-29", "text": "(langchain.llms.Databricks attribute)\n(langchain.llms.SagemakerEndpoint attribute)\nendpoint_url (langchain.embeddings.MiniMaxEmbeddings attribute)\n(langchain.embeddings.MosaicMLInstructorEmbeddings attribute)\n(langchain.llms.CerebriumAI attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.HuggingFaceEndpoint attribute)\n(langchain.llms.Modal attribute)\n(langchain.llms.MosaicML attribute)\nengines (langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nentity_cache (langchain.memory.ConversationEntityMemory attribute)\nentity_extraction_chain (langchain.chains.GraphQAChain attribute)\nentity_extraction_prompt (langchain.memory.ConversationEntityMemory attribute)\n(langchain.memory.ConversationKGMemory attribute)\nentity_store (langchain.memory.ConversationEntityMemory attribute)\nentity_summarization_prompt (langchain.memory.ConversationEntityMemory attribute)\nerror (langchain.chains.OpenAIModerationChain attribute)\nescape_str() (langchain.vectorstores.MyScale method)\nEverNoteLoader (class in langchain.document_loaders)\nexample_keys (langchain.prompts.example_selector.SemanticSimilarityExampleSelector attribute)\nexample_prompt (langchain.prompts.example_selector.LengthBasedExampleSelector attribute)\n(langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\nexample_selector (langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\nexample_separator (langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\nexamples (langchain.agents.agent_toolkits.PowerBIToolkit attribute)\n(langchain.prompts.example_selector.LengthBasedExampleSelector attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}44{"id": "eddc1cc06a3b-30", "text": "(langchain.prompts.example_selector.LengthBasedExampleSelector attribute)\n(langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\n(langchain.tools.QueryPowerBITool attribute)\nexecutable_path (langchain.document_loaders.SeleniumURLLoader attribute)\nexecute_task() (langchain.experimental.BabyAGI method)\nexists() (langchain.memory.InMemoryEntityStore method)\n(langchain.memory.RedisEntityStore method)\nextra (langchain.retrievers.WeaviateHybridSearchRetriever.Config attribute)\nextract_video_id() (langchain.document_loaders.YoutubeLoader static method)\nF\nf16_kv (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nFacebookChatLoader (class in langchain.document_loaders)\nFAISS (class in langchain.vectorstores)\nfetch_all() (langchain.document_loaders.WebBaseLoader method)\nfetch_data_from_telegram() (langchain.document_loaders.TelegramChatApiLoader method)\nfetch_k (langchain.prompts.example_selector.MaxMarginalRelevanceExampleSelector attribute)\nfetch_memories() (langchain.experimental.GenerativeAgentMemory method)\nfetch_place_details() (langchain.utilities.GooglePlacesAPIWrapper method)\nfile_ids (langchain.document_loaders.GoogleDriveLoader attribute)\nfile_paths (langchain.document_loaders.DocugamiLoader attribute)\nfile_types (langchain.document_loaders.GoogleDriveLoader attribute)\nFileChatMessageHistory (class in langchain.memory)\nfilter (langchain.retrievers.ChatGPTPluginRetriever attribute)\nfolder_id (langchain.document_loaders.GoogleDriveLoader attribute)\nfolder_path (langchain.document_loaders.BlackboardLoader attribute)\n(langchain.document_loaders.OneDriveLoader attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}45{"id": "eddc1cc06a3b-31", "text": "(langchain.document_loaders.OneDriveLoader attribute)\nforce_delete_by_path() (langchain.vectorstores.DeepLake class method)\nformat() (langchain.prompts.BaseChatPromptTemplate method)\n(langchain.prompts.BasePromptTemplate method)\n(langchain.prompts.ChatPromptTemplate method)\n(langchain.prompts.FewShotPromptTemplate method)\n(langchain.prompts.FewShotPromptWithTemplates method)\n(langchain.prompts.PromptTemplate method)\nformat_messages() (langchain.prompts.BaseChatPromptTemplate method)\n(langchain.prompts.ChatPromptTemplate method)\n(langchain.prompts.MessagesPlaceholder method)\nformat_place_details() (langchain.utilities.GooglePlacesAPIWrapper method)\nformat_prompt() (langchain.prompts.BaseChatPromptTemplate method)\n(langchain.prompts.BasePromptTemplate method)\n(langchain.prompts.StringPromptTemplate method)\nfrequency_penalty (langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\nfrequencyPenalty (langchain.llms.AI21 attribute)\nfrom_agent_and_tools() (langchain.agents.AgentExecutor class method)\nfrom_api_operation() (langchain.chains.OpenAPIEndpointChain class method)\nfrom_bearer_token() (langchain.document_loaders.TwitterTweetLoader class method)\nfrom_browser() (langchain.agents.agent_toolkits.PlayWrightBrowserToolkit class method)\nfrom_chains() (langchain.agents.MRKLChain class method)\nfrom_client_params() (langchain.memory.MomentoChatMessageHistory class method)\n(langchain.vectorstores.Typesense class method)\nfrom_colored_object_prompt() (langchain.chains.PALChain class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}46{"id": "eddc1cc06a3b-32", "text": "from_colored_object_prompt() (langchain.chains.PALChain class method)\nfrom_credentials() (langchain.embeddings.ElasticsearchEmbeddings class method)\nfrom_documents() (langchain.retrievers.TFIDFRetriever class method)\n(langchain.vectorstores.AnalyticDB class method)\n(langchain.vectorstores.AtlasDB class method)\n(langchain.vectorstores.Chroma class method)\n(langchain.vectorstores.Tair class method)\n(langchain.vectorstores.VectorStore class method)\nfrom_embeddings() (langchain.vectorstores.Annoy class method)\n(langchain.vectorstores.FAISS class method)\nfrom_examples() (langchain.prompts.example_selector.MaxMarginalRelevanceExampleSelector class method)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector class method)\n(langchain.prompts.PromptTemplate class method)\nfrom_existing_index() (langchain.vectorstores.Pinecone class method)\n(langchain.vectorstores.Redis class method)\n(langchain.vectorstores.Tair class method)\nfrom_file() (langchain.prompts.PromptTemplate class method)\n(langchain.tools.OpenAPISpec class method)\nfrom_function() (langchain.agents.Tool class method)\n(langchain.tools.StructuredTool class method)\n(langchain.tools.Tool class method)\nfrom_huggingface_tokenizer() (langchain.text_splitter.TextSplitter class method)\nfrom_jira_api_wrapper() (langchain.agents.agent_toolkits.JiraToolkit class method)\nfrom_llm() (langchain.agents.agent_toolkits.OpenAPIToolkit class method)\n(langchain.chains.ChatVectorDBChain class method)\n(langchain.chains.ConstitutionalChain class method)\n(langchain.chains.ConversationalRetrievalChain class method)\n(langchain.chains.FlareChain class method)\n(langchain.chains.GraphCypherQAChain class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}47{"id": "eddc1cc06a3b-33", "text": "(langchain.chains.GraphCypherQAChain class method)\n(langchain.chains.GraphQAChain class method)\n(langchain.chains.HypotheticalDocumentEmbedder class method)\n(langchain.chains.LLMBashChain class method)\n(langchain.chains.LLMCheckerChain class method)\n(langchain.chains.LLMMathChain class method)\n(langchain.chains.LLMSummarizationCheckerChain class method)\n(langchain.chains.QAGenerationChain class method)\n(langchain.chains.SQLDatabaseChain class method)\n(langchain.chains.SQLDatabaseSequentialChain class method)\n(langchain.experimental.BabyAGI class method)\n(langchain.output_parsers.OutputFixingParser class method)\n(langchain.output_parsers.RetryOutputParser class method)\n(langchain.output_parsers.RetryWithErrorOutputParser class method)\n(langchain.retrievers.document_compressors.LLMChainExtractor class method)\n(langchain.retrievers.document_compressors.LLMChainFilter class method)\n(langchain.retrievers.SelfQueryRetriever class method)\nfrom_llm_and_ai_plugin() (langchain.agents.agent_toolkits.NLAToolkit class method)\nfrom_llm_and_ai_plugin_url() (langchain.agents.agent_toolkits.NLAToolkit class method)\nfrom_llm_and_api_docs() (langchain.chains.APIChain class method)\nfrom_llm_and_spec() (langchain.agents.agent_toolkits.NLAToolkit class method)\nfrom_llm_and_tools() (langchain.agents.Agent class method)\n(langchain.agents.BaseSingleActionAgent class method)\n(langchain.agents.ConversationalAgent class method)\n(langchain.agents.ConversationalChatAgent class method)\n(langchain.agents.StructuredChatAgent class method)\n(langchain.agents.ZeroShotAgent class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}48{"id": "eddc1cc06a3b-34", "text": "(langchain.agents.ZeroShotAgent class method)\nfrom_llm_and_url() (langchain.agents.agent_toolkits.NLAToolkit class method)\nfrom_math_prompt() (langchain.chains.PALChain class method)\nfrom_messages() (langchain.memory.ConversationSummaryMemory class method)\nfrom_model_id() (langchain.llms.HuggingFacePipeline class method)\nfrom_number (langchain.utilities.TwilioAPIWrapper attribute)\nfrom_openapi_spec() (langchain.tools.APIOperation class method)\nfrom_openapi_url() (langchain.tools.APIOperation class method)\nfrom_params() (langchain.chains.MapReduceChain class method)\n(langchain.document_loaders.WeatherDataLoader class method)\n(langchain.retrievers.VespaRetriever class method)\n(langchain.vectorstores.DocArrayHnswSearch class method)\n(langchain.vectorstores.DocArrayInMemorySearch class method)\nfrom_pipeline() (langchain.llms.SelfHostedHuggingFaceLLM class method)\n(langchain.llms.SelfHostedPipeline class method)\nfrom_plugin_url() (langchain.tools.AIPluginTool class method)\nfrom_rail() (langchain.output_parsers.GuardrailsOutputParser class method)\nfrom_rail_string() (langchain.output_parsers.GuardrailsOutputParser class method)\nfrom_response_schemas() (langchain.output_parsers.StructuredOutputParser class method)\nfrom_secrets() (langchain.document_loaders.TwitterTweetLoader class method)\nfrom_spec_dict() (langchain.tools.OpenAPISpec class method)\nfrom_string() (langchain.chains.LLMChain class method)\nfrom_template() (langchain.prompts.PromptTemplate class method)\nfrom_text() (langchain.tools.OpenAPISpec class method)\nfrom_texts() (langchain.retrievers.KNNRetriever class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}49{"id": "eddc1cc06a3b-35", "text": "from_texts() (langchain.retrievers.KNNRetriever class method)\n(langchain.retrievers.SVMRetriever class method)\n(langchain.retrievers.TFIDFRetriever class method)\n(langchain.vectorstores.AnalyticDB class method)\n(langchain.vectorstores.Annoy class method)\n(langchain.vectorstores.AtlasDB class method)\n(langchain.vectorstores.Chroma class method)\n(langchain.vectorstores.DeepLake class method)\n(langchain.vectorstores.DocArrayHnswSearch class method)\n(langchain.vectorstores.DocArrayInMemorySearch class method)\n(langchain.vectorstores.ElasticVectorSearch class method)\n(langchain.vectorstores.FAISS class method)\n(langchain.vectorstores.LanceDB class method)\n(langchain.vectorstores.Milvus class method)\n(langchain.vectorstores.MyScale class method)\n(langchain.vectorstores.OpenSearchVectorSearch class method)\n(langchain.vectorstores.Pinecone class method)\n(langchain.vectorstores.Qdrant class method)\n(langchain.vectorstores.Redis class method)\n(langchain.vectorstores.SKLearnVectorStore class method)\n(langchain.vectorstores.SupabaseVectorStore class method)\n(langchain.vectorstores.Tair class method)\n(langchain.vectorstores.Typesense class method)\n(langchain.vectorstores.Vectara class method)\n(langchain.vectorstores.VectorStore class method)\n(langchain.vectorstores.Weaviate class method)\n(langchain.vectorstores.Zilliz class method)\nfrom_texts_return_keys() (langchain.vectorstores.Redis class method)\nfrom_tiktoken_encoder() (langchain.text_splitter.TextSplitter class method)\nfrom_uri() (langchain.utilities.SparkSQL class method)\nfrom_url() (langchain.tools.OpenAPISpec class method)\nfrom_url_and_method() (langchain.chains.OpenAPIEndpointChain class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}50{"id": "eddc1cc06a3b-36", "text": "from_url_and_method() (langchain.chains.OpenAPIEndpointChain class method)\nfrom_youtube_url() (langchain.document_loaders.YoutubeLoader class method)\nfrom_zapier_nla_wrapper() (langchain.agents.agent_toolkits.ZapierToolkit class method)\nFRONT_MATTER_REGEX (langchain.document_loaders.ObsidianLoader attribute)\nfull_key_prefix (langchain.memory.RedisEntityStore property)\nfunc (langchain.agents.Tool attribute)\n(langchain.tools.StructuredTool attribute)\n(langchain.tools.Tool attribute)\nfunction_name (langchain.utilities.LambdaWrapper attribute)\nG\nGCSDirectoryLoader (class in langchain.document_loaders)\nGCSFileLoader (class in langchain.document_loaders)\ngenerate() (langchain.chains.LLMChain method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)", "source": "https://python.langchain.com/en/latest/genindex.html"}51{"id": "eddc1cc06a3b-37", "text": "(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\ngenerate_dialogue_response() (langchain.experimental.GenerativeAgent method)\ngenerate_prompt() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)", "source": "https://python.langchain.com/en/latest/genindex.html"}52{"id": "eddc1cc06a3b-38", "text": "(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\ngenerate_reaction() (langchain.experimental.GenerativeAgent method)\nGenerativeAgent (class in langchain.experimental)\nGenerativeAgentMemory (class in langchain.experimental)\nget() (langchain.memory.InMemoryEntityStore method)\n(langchain.memory.RedisEntityStore method)\n(langchain.utilities.TextRequestsWrapper method)\n(langchain.vectorstores.Chroma method)\nget_all_tool_names() (in module langchain.agents)", "source": "https://python.langchain.com/en/latest/genindex.html"}53{"id": "eddc1cc06a3b-39", "text": "get_all_tool_names() (in module langchain.agents)\nget_allowed_tools() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\nget_answer_expr (langchain.chains.PALChain attribute)\nget_cleaned_operation_id() (langchain.tools.OpenAPISpec static method)\nget_collection() (langchain.vectorstores.AnalyticDB method)\nget_connection_string() (langchain.vectorstores.AnalyticDB class method)\nget_current_entities() (langchain.memory.ConversationKGMemory method)\nget_description() (langchain.tools.VectorStoreQATool static method)\n(langchain.tools.VectorStoreQAWithSourcesTool static method)\nget_format_instructions() (langchain.output_parsers.CommaSeparatedListOutputParser method)\n(langchain.output_parsers.GuardrailsOutputParser method)\n(langchain.output_parsers.OutputFixingParser method)\n(langchain.output_parsers.PydanticOutputParser method)\n(langchain.output_parsers.RetryOutputParser method)\n(langchain.output_parsers.RetryWithErrorOutputParser method)\n(langchain.output_parsers.StructuredOutputParser method)\nget_full_header() (langchain.experimental.GenerativeAgent method)\nget_full_inputs() (langchain.agents.Agent method)\nget_input (langchain.retrievers.document_compressors.LLMChainExtractor attribute)\n(langchain.retrievers.document_compressors.LLMChainFilter attribute)\nget_knowledge_triplets() (langchain.memory.ConversationKGMemory method)\nget_methods_for_path() (langchain.tools.OpenAPISpec method)\nget_next_task() (langchain.experimental.BabyAGI method)\nget_num_tokens() (langchain.chat_models.ChatAnthropic method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)", "source": "https://python.langchain.com/en/latest/genindex.html"}54{"id": "eddc1cc06a3b-40", "text": "(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)", "source": "https://python.langchain.com/en/latest/genindex.html"}55{"id": "eddc1cc06a3b-41", "text": "(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nget_num_tokens_from_messages() (langchain.chat_models.ChatOpenAI method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)", "source": "https://python.langchain.com/en/latest/genindex.html"}56{"id": "eddc1cc06a3b-42", "text": "(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nget_operation() (langchain.tools.OpenAPISpec method)\nget_parameters_for_operation() (langchain.tools.OpenAPISpec method)\nget_params() (langchain.serpapi.SerpAPIWrapper method)\n(langchain.utilities.SerpAPIWrapper method)\nget_principles() (langchain.chains.ConstitutionalChain class method)\nget_processed_pdf() (langchain.document_loaders.MathpixPDFLoader method)\nget_referenced_schema() (langchain.tools.OpenAPISpec method)\nget_relevant_documents() (langchain.retrievers.ArxivRetriever method)\n(langchain.retrievers.AzureCognitiveSearchRetriever method)\n(langchain.retrievers.ChatGPTPluginRetriever method)\n(langchain.retrievers.ContextualCompressionRetriever method)\n(langchain.retrievers.DataberryRetriever method)\n(langchain.retrievers.ElasticSearchBM25Retriever method)\n(langchain.retrievers.KNNRetriever method)\n(langchain.retrievers.MetalRetriever method)", "source": "https://python.langchain.com/en/latest/genindex.html"}57{"id": "eddc1cc06a3b-43", "text": "(langchain.retrievers.MetalRetriever method)\n(langchain.retrievers.PineconeHybridSearchRetriever method)\n(langchain.retrievers.RemoteLangChainRetriever method)\n(langchain.retrievers.SelfQueryRetriever method)\n(langchain.retrievers.SVMRetriever method)\n(langchain.retrievers.TFIDFRetriever method)\n(langchain.retrievers.TimeWeightedVectorStoreRetriever method)\n(langchain.retrievers.VespaRetriever method)\n(langchain.retrievers.WeaviateHybridSearchRetriever method)\n(langchain.retrievers.WikipediaRetriever method)\n(langchain.retrievers.ZepRetriever method)\nget_relevant_documents_with_filter() (langchain.retrievers.VespaRetriever method)\nget_request_body_for_operation() (langchain.tools.OpenAPISpec method)\nget_salient_docs() (langchain.retrievers.TimeWeightedVectorStoreRetriever method)\nget_schemas() (langchain.utilities.PowerBIDataset method)\nget_snippets() (langchain.utilities.DuckDuckGoSearchAPIWrapper method)\nget_stateful_documents() (in module langchain.document_transformers)\nget_sub_prompts() (langchain.llms.AzureOpenAI method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\nget_summary() (langchain.experimental.GenerativeAgent method)\nget_table_info() (langchain.utilities.PowerBIDataset method)\n(langchain.utilities.SparkSQL method)\nget_table_info_no_throw() (langchain.utilities.SparkSQL method)\nget_table_names() (langchain.utilities.PowerBIDataset method)\nget_text_length (langchain.prompts.example_selector.LengthBasedExampleSelector attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}58{"id": "eddc1cc06a3b-44", "text": "get_text_length (langchain.prompts.example_selector.LengthBasedExampleSelector attribute)\nget_token_ids() (langchain.chat_models.ChatOpenAI method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)", "source": "https://python.langchain.com/en/latest/genindex.html"}59{"id": "eddc1cc06a3b-45", "text": "(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nget_tools() (langchain.agents.agent_toolkits.AzureCognitiveServicesToolkit method)\n(langchain.agents.agent_toolkits.FileManagementToolkit method)\n(langchain.agents.agent_toolkits.GmailToolkit method)\n(langchain.agents.agent_toolkits.JiraToolkit method)\n(langchain.agents.agent_toolkits.JsonToolkit method)\n(langchain.agents.agent_toolkits.NLAToolkit method)\n(langchain.agents.agent_toolkits.OpenAPIToolkit method)\n(langchain.agents.agent_toolkits.PlayWrightBrowserToolkit method)\n(langchain.agents.agent_toolkits.PowerBIToolkit method)\n(langchain.agents.agent_toolkits.SparkSQLToolkit method)\n(langchain.agents.agent_toolkits.SQLDatabaseToolkit method)\n(langchain.agents.agent_toolkits.VectorStoreRouterToolkit method)\n(langchain.agents.agent_toolkits.VectorStoreToolkit method)\n(langchain.agents.agent_toolkits.ZapierToolkit method)\nget_usable_table_names() (langchain.utilities.SparkSQL method)\nGitbookLoader (class in langchain.document_loaders)\nGitLoader (class in langchain.document_loaders)\ngl (langchain.utilities.GoogleSerperAPIWrapper attribute)\nglobals (langchain.python.PythonREPL attribute)\n(langchain.utilities.PythonREPL attribute)\ngoogle_api_client (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\ngoogle_api_key (langchain.chat_models.ChatGooglePalm attribute)\n(langchain.utilities.GoogleSearchAPIWrapper attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}60{"id": "eddc1cc06a3b-46", "text": "(langchain.utilities.GoogleSearchAPIWrapper attribute)\ngoogle_cse_id (langchain.utilities.GoogleSearchAPIWrapper attribute)\nGoogleApiClient (class in langchain.document_loaders)\nGoogleApiYoutubeLoader (class in langchain.document_loaders)\ngplaces_api_key (langchain.utilities.GooglePlacesAPIWrapper attribute)\ngraph (langchain.chains.GraphCypherQAChain attribute)\n(langchain.chains.GraphQAChain attribute)\ngraphql_endpoint (langchain.utilities.GraphQLAPIWrapper attribute)\ngroup_id (langchain.utilities.PowerBIDataset attribute)\nguard (langchain.output_parsers.GuardrailsOutputParser attribute)\nGutenbergLoader (class in langchain.document_loaders)\nH\nhandle_parsing_errors (langchain.agents.AgentExecutor attribute)\nhardware (langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\nheaders (langchain.document_loaders.MathpixPDFLoader property)\n(langchain.retrievers.RemoteLangChainRetriever attribute)\n(langchain.utilities.PowerBIDataset property)\n(langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\n(langchain.utilities.TextRequestsWrapper attribute)\nheadless (langchain.document_loaders.PlaywrightURLLoader attribute)\n(langchain.document_loaders.SeleniumURLLoader attribute)\nhl (langchain.utilities.GoogleSerperAPIWrapper attribute)\nHNLoader (class in langchain.document_loaders)\nhost (langchain.llms.Databricks attribute)\n(langchain.vectorstores.MyScaleSettings attribute)\nhosting (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\nHuggingFaceDatasetLoader (class in langchain.document_loaders)\nhuman_prefix (langchain.memory.ConversationBufferMemory attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}61{"id": "eddc1cc06a3b-47", "text": "human_prefix (langchain.memory.ConversationBufferMemory attribute)\n(langchain.memory.ConversationBufferWindowMemory attribute)\n(langchain.memory.ConversationEntityMemory attribute)\n(langchain.memory.ConversationKGMemory attribute)\n(langchain.memory.ConversationStringBufferMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory attribute)\nI\nIFixitLoader (class in langchain.document_loaders)\nImageCaptionLoader (class in langchain.document_loaders)\nimpersonated_user_name (langchain.utilities.PowerBIDataset attribute)\nimportance_weight (langchain.experimental.GenerativeAgentMemory attribute)\nIMSDbLoader (class in langchain.document_loaders)\nindex (langchain.retrievers.KNNRetriever attribute)\n(langchain.retrievers.PineconeHybridSearchRetriever attribute)\n(langchain.retrievers.SVMRetriever attribute)\nindex_name (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\nindex_param (langchain.vectorstores.MyScaleSettings attribute)\nindex_type (langchain.vectorstores.MyScaleSettings attribute)\ninference_fn (langchain.embeddings.SelfHostedEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\ninference_kwargs (langchain.embeddings.SelfHostedEmbeddings attribute)\ninitialize_agent() (in module langchain.agents)\ninject_instruction_format (langchain.llms.MosaicML attribute)\nInMemoryDocstore (class in langchain.docstore)\nInMemoryEntityStore (class in langchain.memory)\ninput_func (langchain.tools.HumanInputRun attribute)\ninput_key (langchain.chains.QAGenerationChain attribute)\n(langchain.memory.ConversationStringBufferMemory attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}62{"id": "eddc1cc06a3b-48", "text": "(langchain.memory.ConversationStringBufferMemory attribute)\n(langchain.memory.VectorStoreRetrieverMemory attribute)\n(langchain.retrievers.RemoteLangChainRetriever attribute)\ninput_keys (langchain.chains.ConstitutionalChain property)\n(langchain.chains.ConversationChain property)\n(langchain.chains.FlareChain property)\n(langchain.chains.HypotheticalDocumentEmbedder property)\n(langchain.chains.QAGenerationChain property)\n(langchain.experimental.BabyAGI property)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector attribute)\ninput_variables (langchain.chains.SequentialChain attribute)\n(langchain.chains.TransformChain attribute)\n(langchain.prompts.BasePromptTemplate attribute)\n(langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\n(langchain.prompts.MessagesPlaceholder property)\n(langchain.prompts.PromptTemplate attribute)\nis_public_page() (langchain.document_loaders.ConfluenceLoader method)\nis_single_input (langchain.tools.BaseTool property)\nJ\nJoplinLoader (class in langchain.document_loaders)\njson() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)", "source": "https://python.langchain.com/en/latest/genindex.html"}63{"id": "eddc1cc06a3b-49", "text": "(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\njson_agent (langchain.agents.agent_toolkits.OpenAPIToolkit attribute)\nJSONLoader (class in langchain.document_loaders)\nK\nk (langchain.chains.QAGenerationChain attribute)\n(langchain.chains.VectorDBQA attribute)\n(langchain.chains.VectorDBQAWithSourcesChain attribute)\n(langchain.llms.Cohere attribute)\n(langchain.memory.ConversationBufferWindowMemory attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}64{"id": "eddc1cc06a3b-50", "text": "(langchain.llms.Cohere attribute)\n(langchain.memory.ConversationBufferWindowMemory attribute)\n(langchain.memory.ConversationEntityMemory attribute)\n(langchain.memory.ConversationKGMemory attribute)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector attribute)\n(langchain.retrievers.document_compressors.EmbeddingsFilter attribute)\n(langchain.retrievers.KNNRetriever attribute)\n(langchain.retrievers.SVMRetriever attribute)\n(langchain.retrievers.TFIDFRetriever attribute)\n(langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\n(langchain.utilities.BingSearchAPIWrapper attribute)\n(langchain.utilities.DuckDuckGoSearchAPIWrapper attribute)\n(langchain.utilities.GoogleSearchAPIWrapper attribute)\n(langchain.utilities.GoogleSerperAPIWrapper attribute)\n(langchain.utilities.MetaphorSearchAPIWrapper attribute)\n(langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nkey (langchain.memory.RedisChatMessageHistory property)\nkey_prefix (langchain.memory.RedisEntityStore attribute)\nkg (langchain.memory.ConversationKGMemory attribute)\nknowledge_extraction_prompt (langchain.memory.ConversationKGMemory attribute)\nL\nLanceDB (class in langchain.vectorstores)\nlang (langchain.utilities.WikipediaAPIWrapper attribute)\n    langchain.agents\n      \nmodule\n    langchain.agents.agent_toolkits\n      \nmodule\n    langchain.chains\n      \nmodule\n    langchain.chat_models\n      \nmodule\n    langchain.docstore\n      \nmodule\n    langchain.document_loaders\n      \nmodule\n    langchain.document_transformers\n      \nmodule\n    langchain.embeddings\n      \nmodule\n    langchain.llms\n      \nmodule\n    langchain.memory\n      \nmodule\n    langchain.output_parsers", "source": "https://python.langchain.com/en/latest/genindex.html"}65{"id": "eddc1cc06a3b-51", "text": "module\n    langchain.memory\n      \nmodule\n    langchain.output_parsers\n      \nmodule\n    langchain.prompts\n      \nmodule\n    langchain.prompts.example_selector\n      \nmodule\n    langchain.python\n      \nmodule\n    langchain.retrievers\n      \nmodule\n    langchain.retrievers.document_compressors\n      \nmodule\n    langchain.serpapi\n      \nmodule\n    langchain.text_splitter\n      \nmodule\n    langchain.tools\n      \nmodule\n    langchain.utilities\n      \nmodule\n    langchain.utilities.searx_search\n      \nmodule\n    langchain.vectorstores\n      \nmodule\nlast_n_tokens_size (langchain.llms.LlamaCpp attribute)\nlast_refreshed (langchain.experimental.GenerativeAgent attribute)\nLatexTextSplitter (class in langchain.text_splitter)\nlazy_load() (langchain.document_loaders.BibtexLoader method)\n(langchain.document_loaders.HuggingFaceDatasetLoader method)\n(langchain.document_loaders.JoplinLoader method)\n(langchain.document_loaders.PDFMinerLoader method)\n(langchain.document_loaders.PyPDFium2Loader method)\n(langchain.document_loaders.PyPDFLoader method)\n(langchain.document_loaders.ToMarkdownLoader method)\n(langchain.document_loaders.TomlLoader method)\n(langchain.document_loaders.WeatherDataLoader method)\nlength (langchain.llms.ForefrontAI attribute)\nlength_no_input (langchain.llms.NLPCloud attribute)\nlength_penalty (langchain.llms.NLPCloud attribute)\nlib (langchain.llms.CTransformers attribute)\nlist_assertions_prompt (langchain.chains.LLMCheckerChain attribute)\nllm (langchain.agents.agent_toolkits.PowerBIToolkit attribute)\n(langchain.agents.agent_toolkits.SparkSQLToolkit attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}66{"id": "eddc1cc06a3b-52", "text": "(langchain.agents.agent_toolkits.SparkSQLToolkit attribute)\n(langchain.agents.agent_toolkits.SQLDatabaseToolkit attribute)\n(langchain.agents.agent_toolkits.VectorStoreRouterToolkit attribute)\n(langchain.agents.agent_toolkits.VectorStoreToolkit attribute)\n(langchain.chains.LLMBashChain attribute)\n(langchain.chains.LLMChain attribute)\n(langchain.chains.LLMCheckerChain attribute)\n(langchain.chains.LLMMathChain attribute)\n(langchain.chains.LLMSummarizationCheckerChain attribute)\n(langchain.chains.PALChain attribute)\n(langchain.chains.SQLDatabaseChain attribute)\n(langchain.experimental.GenerativeAgent attribute)\n(langchain.experimental.GenerativeAgentMemory attribute)\n(langchain.memory.ConversationEntityMemory attribute)\n(langchain.memory.ConversationKGMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory attribute)\nllm_chain (langchain.agents.Agent attribute)\n(langchain.agents.LLMSingleActionAgent attribute)\n(langchain.chains.HypotheticalDocumentEmbedder attribute)\n(langchain.chains.LLMBashChain attribute)\n(langchain.chains.LLMMathChain attribute)\n(langchain.chains.LLMRequestsChain attribute)\n(langchain.chains.PALChain attribute)\n(langchain.chains.QAGenerationChain attribute)\n(langchain.chains.SQLDatabaseChain attribute)\n(langchain.retrievers.document_compressors.LLMChainExtractor attribute)\n(langchain.retrievers.document_compressors.LLMChainFilter attribute)\n(langchain.retrievers.SelfQueryRetriever attribute)\n(langchain.tools.QueryPowerBITool attribute)\nllm_prefix (langchain.agents.Agent property)\n(langchain.agents.ConversationalAgent property)\n(langchain.agents.ConversationalChatAgent property)\n(langchain.agents.StructuredChatAgent property)\n(langchain.agents.ZeroShotAgent property)", "source": "https://python.langchain.com/en/latest/genindex.html"}67{"id": "eddc1cc06a3b-53", "text": "(langchain.agents.StructuredChatAgent property)\n(langchain.agents.ZeroShotAgent property)\nload() (langchain.document_loaders.AirbyteJSONLoader method)\n(langchain.document_loaders.ApifyDatasetLoader method)\n(langchain.document_loaders.ArxivLoader method)\n(langchain.document_loaders.AZLyricsLoader method)\n(langchain.document_loaders.AzureBlobStorageContainerLoader method)\n(langchain.document_loaders.AzureBlobStorageFileLoader method)\n(langchain.document_loaders.BibtexLoader method)\n(langchain.document_loaders.BigQueryLoader method)\n(langchain.document_loaders.BiliBiliLoader method)\n(langchain.document_loaders.BlackboardLoader method)\n(langchain.document_loaders.BlockchainDocumentLoader method)\n(langchain.document_loaders.BSHTMLLoader method)\n(langchain.document_loaders.ChatGPTLoader method)\n(langchain.document_loaders.CollegeConfidentialLoader method)\n(langchain.document_loaders.ConfluenceLoader method)\n(langchain.document_loaders.CoNLLULoader method)\n(langchain.document_loaders.CSVLoader method)\n(langchain.document_loaders.DataFrameLoader method)\n(langchain.document_loaders.DiffbotLoader method)\n(langchain.document_loaders.DirectoryLoader method)\n(langchain.document_loaders.DiscordChatLoader method)\n(langchain.document_loaders.DocugamiLoader method)\n(langchain.document_loaders.Docx2txtLoader method)\n(langchain.document_loaders.DuckDBLoader method)\n(langchain.document_loaders.EverNoteLoader method)\n(langchain.document_loaders.FacebookChatLoader method)\n(langchain.document_loaders.GCSDirectoryLoader method)\n(langchain.document_loaders.GCSFileLoader method)\n(langchain.document_loaders.GitbookLoader method)\n(langchain.document_loaders.GitLoader method)\n(langchain.document_loaders.GoogleApiYoutubeLoader method)", "source": "https://python.langchain.com/en/latest/genindex.html"}68{"id": "eddc1cc06a3b-54", "text": "(langchain.document_loaders.GoogleApiYoutubeLoader method)\n(langchain.document_loaders.GoogleDriveLoader method)\n(langchain.document_loaders.GutenbergLoader method)\n(langchain.document_loaders.HNLoader method)\n(langchain.document_loaders.HuggingFaceDatasetLoader method)\n(langchain.document_loaders.IFixitLoader method)\n(langchain.document_loaders.ImageCaptionLoader method)\n(langchain.document_loaders.IMSDbLoader method)\n(langchain.document_loaders.JoplinLoader method)\n(langchain.document_loaders.JSONLoader method)\n(langchain.document_loaders.MastodonTootsLoader method)\n(langchain.document_loaders.MathpixPDFLoader method)\n(langchain.document_loaders.ModernTreasuryLoader method)\n(langchain.document_loaders.MWDumpLoader method)\n(langchain.document_loaders.NotebookLoader method)\n(langchain.document_loaders.NotionDBLoader method)\n(langchain.document_loaders.NotionDirectoryLoader method)\n(langchain.document_loaders.ObsidianLoader method)\n(langchain.document_loaders.OneDriveLoader method)\n(langchain.document_loaders.OnlinePDFLoader method)\n(langchain.document_loaders.OutlookMessageLoader method)\n(langchain.document_loaders.PDFMinerLoader method)\n(langchain.document_loaders.PDFMinerPDFasHTMLLoader method)\n(langchain.document_loaders.PDFPlumberLoader method)\n(langchain.document_loaders.PlaywrightURLLoader method)\n(langchain.document_loaders.PsychicLoader method)\n(langchain.document_loaders.PyMuPDFLoader method)\n(langchain.document_loaders.PyPDFDirectoryLoader method)\n(langchain.document_loaders.PyPDFium2Loader method)\n(langchain.document_loaders.PyPDFLoader method)\n(langchain.document_loaders.ReadTheDocsLoader method)\n(langchain.document_loaders.RedditPostsLoader method)", "source": "https://python.langchain.com/en/latest/genindex.html"}69{"id": "eddc1cc06a3b-55", "text": "(langchain.document_loaders.RedditPostsLoader method)\n(langchain.document_loaders.RoamLoader method)\n(langchain.document_loaders.S3DirectoryLoader method)\n(langchain.document_loaders.S3FileLoader method)\n(langchain.document_loaders.SeleniumURLLoader method)\n(langchain.document_loaders.SitemapLoader method)\n(langchain.document_loaders.SlackDirectoryLoader method)\n(langchain.document_loaders.SpreedlyLoader method)\n(langchain.document_loaders.SRTLoader method)\n(langchain.document_loaders.StripeLoader method)\n(langchain.document_loaders.TelegramChatApiLoader method)\n(langchain.document_loaders.TelegramChatFileLoader method)\n(langchain.document_loaders.TextLoader method)\n(langchain.document_loaders.ToMarkdownLoader method)\n(langchain.document_loaders.TomlLoader method)\n(langchain.document_loaders.TwitterTweetLoader method)\n(langchain.document_loaders.UnstructuredURLLoader method)\n(langchain.document_loaders.WeatherDataLoader method)\n(langchain.document_loaders.WebBaseLoader method)\n(langchain.document_loaders.WhatsAppChatLoader method)\n(langchain.document_loaders.WikipediaLoader method)\n(langchain.document_loaders.YoutubeLoader method)\n(langchain.utilities.ArxivAPIWrapper method)\n(langchain.utilities.WikipediaAPIWrapper method)\nload_agent() (in module langchain.agents)\nload_all_available_meta (langchain.utilities.ArxivAPIWrapper attribute)\n(langchain.utilities.WikipediaAPIWrapper attribute)\nload_all_recursively (langchain.document_loaders.BlackboardLoader attribute)\nload_chain() (in module langchain.chains)\nload_comments() (langchain.document_loaders.HNLoader method)\nload_device() (langchain.document_loaders.IFixitLoader method)\nload_file() (langchain.document_loaders.DirectoryLoader method)", "source": "https://python.langchain.com/en/latest/genindex.html"}70{"id": "eddc1cc06a3b-56", "text": "load_file() (langchain.document_loaders.DirectoryLoader method)\nload_fn_kwargs (langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\nload_guide() (langchain.document_loaders.IFixitLoader method)\nload_huggingface_tool() (in module langchain.agents)\nload_local() (langchain.vectorstores.Annoy class method)\n(langchain.vectorstores.FAISS class method)\nload_max_docs (langchain.utilities.ArxivAPIWrapper attribute)\nload_memory_variables() (langchain.experimental.GenerativeAgentMemory method)\n(langchain.memory.CombinedMemory method)\n(langchain.memory.ConversationBufferMemory method)\n(langchain.memory.ConversationBufferWindowMemory method)\n(langchain.memory.ConversationEntityMemory method)\n(langchain.memory.ConversationKGMemory method)\n(langchain.memory.ConversationStringBufferMemory method)\n(langchain.memory.ConversationSummaryBufferMemory method)\n(langchain.memory.ConversationSummaryMemory method)\n(langchain.memory.ConversationTokenBufferMemory method)\n(langchain.memory.ReadOnlySharedMemory method)\n(langchain.memory.SimpleMemory method)\n(langchain.memory.VectorStoreRetrieverMemory method)\nload_messages() (langchain.memory.CosmosDBChatMessageHistory method)\nload_page() (langchain.document_loaders.NotionDBLoader method)\nload_prompt() (in module langchain.prompts)\nload_questions_and_answers() (langchain.document_loaders.IFixitLoader method)\nload_results() (langchain.document_loaders.HNLoader method)\nload_suggestions() (langchain.document_loaders.IFixitLoader static method)\nload_tools() (in module langchain.agents)\nload_trashed_files (langchain.document_loaders.GoogleDriveLoader attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}71{"id": "eddc1cc06a3b-57", "text": "load_trashed_files (langchain.document_loaders.GoogleDriveLoader attribute)\nlocals (langchain.python.PythonREPL attribute)\n(langchain.utilities.PythonREPL attribute)\nlocation (langchain.llms.VertexAI attribute)\nlog_probs (langchain.llms.AlephAlpha attribute)\nlogit_bias (langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\nlogitBias (langchain.llms.AI21 attribute)\nlogits_all (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nlogprobs (langchain.llms.LlamaCpp attribute)\n(langchain.llms.Writer attribute)\nlookup_tool() (langchain.agents.AgentExecutor method)\nlora_base (langchain.llms.LlamaCpp attribute)\nlora_path (langchain.llms.LlamaCpp attribute)\nM\nMarkdownTextSplitter (class in langchain.text_splitter)\nMastodonTootsLoader (class in langchain.document_loaders)\nMathpixPDFLoader (class in langchain.document_loaders)\nmax_checks (langchain.chains.LLMSummarizationCheckerChain attribute)\nmax_execution_time (langchain.agents.AgentExecutor attribute)\nmax_iter (langchain.chains.FlareChain attribute)\nmax_iterations (langchain.agents.agent_toolkits.PowerBIToolkit attribute)\n(langchain.agents.AgentExecutor attribute)\n(langchain.tools.QueryPowerBITool attribute)\nmax_length (langchain.llms.NLPCloud attribute)\n(langchain.llms.Petals attribute)\n(langchain.prompts.example_selector.LengthBasedExampleSelector attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}72{"id": "eddc1cc06a3b-58", "text": "(langchain.prompts.example_selector.LengthBasedExampleSelector attribute)\nmax_marginal_relevance_search() (langchain.vectorstores.Annoy method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.Qdrant method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.VectorStore method)\n(langchain.vectorstores.Weaviate method)\nmax_marginal_relevance_search_by_vector() (langchain.vectorstores.Annoy method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.VectorStore method)\n(langchain.vectorstores.Weaviate method)\nmax_new_tokens (langchain.llms.Petals attribute)\nmax_output_tokens (langchain.llms.GooglePalm attribute)\n(langchain.llms.VertexAI attribute)\nmax_results (langchain.utilities.DuckDuckGoSearchAPIWrapper attribute)\nmax_retries (langchain.chat_models.ChatOpenAI attribute)\n(langchain.embeddings.OpenAIEmbeddings attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\nmax_token_limit (langchain.memory.ConversationSummaryBufferMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory attribute)\nmax_tokens (langchain.chat_models.ChatOpenAI attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Cohere attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}73{"id": "eddc1cc06a3b-59", "text": "(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.PredictionGuard attribute)\n(langchain.llms.Writer attribute)\nmax_tokens_for_prompt() (langchain.llms.AzureOpenAI method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\nmax_tokens_limit (langchain.chains.ConversationalRetrievalChain attribute)\n(langchain.chains.RetrievalQAWithSourcesChain attribute)\n(langchain.chains.VectorDBQAWithSourcesChain attribute)\nmax_tokens_per_generation (langchain.llms.RWKV attribute)\nmax_tokens_to_sample (langchain.llms.Anthropic attribute)\nmaximum_tokens (langchain.llms.AlephAlpha attribute)\nmaxTokens (langchain.llms.AI21 attribute)\nmemories (langchain.memory.CombinedMemory attribute)\n(langchain.memory.SimpleMemory attribute)\nmemory (langchain.chains.ConversationChain attribute)\n(langchain.experimental.GenerativeAgent attribute)\n(langchain.memory.ReadOnlySharedMemory attribute)\nmemory_key (langchain.memory.ConversationSummaryBufferMemory attribute)\n(langchain.memory.ConversationTokenBufferMemory attribute)\n(langchain.memory.VectorStoreRetrieverMemory attribute)\nmemory_retriever (langchain.experimental.GenerativeAgentMemory attribute)\nmemory_stream (langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\nmemory_variables (langchain.experimental.GenerativeAgentMemory property)\n(langchain.memory.CombinedMemory property)\n(langchain.memory.ConversationStringBufferMemory property)\n(langchain.memory.ReadOnlySharedMemory property)\n(langchain.memory.SimpleMemory property)", "source": "https://python.langchain.com/en/latest/genindex.html"}74{"id": "eddc1cc06a3b-60", "text": "(langchain.memory.ReadOnlySharedMemory property)\n(langchain.memory.SimpleMemory property)\n(langchain.memory.VectorStoreRetrieverMemory property)\nmerge_from() (langchain.vectorstores.FAISS method)\nmessages (langchain.memory.CassandraChatMessageHistory property)\n(langchain.memory.ChatMessageHistory attribute)\n(langchain.memory.DynamoDBChatMessageHistory property)\n(langchain.memory.FileChatMessageHistory property)\n(langchain.memory.MomentoChatMessageHistory property)\n(langchain.memory.MongoDBChatMessageHistory property)\n(langchain.memory.PostgresChatMessageHistory property)\n(langchain.memory.RedisChatMessageHistory property)\nmetadata_column (langchain.vectorstores.MyScale property)\nmetadata_key (langchain.retrievers.RemoteLangChainRetriever attribute)\nMETADATA_KEY (langchain.vectorstores.Qdrant attribute)\nMetalRetriever (class in langchain.retrievers)\nmetaphor_api_key (langchain.utilities.MetaphorSearchAPIWrapper attribute)\nmethod (langchain.tools.APIOperation attribute)\nmetric (langchain.vectorstores.MyScaleSettings attribute)\nMilvus (class in langchain.vectorstores)\nmin_chunk_size (langchain.document_loaders.DocugamiLoader attribute)\nmin_length (langchain.llms.NLPCloud attribute)\nmin_prob (langchain.chains.FlareChain attribute)\nmin_token_gap (langchain.chains.FlareChain attribute)\nmin_tokens (langchain.llms.GooseAI attribute)\n(langchain.llms.Writer attribute)\nminimax_api_key (langchain.embeddings.MiniMaxEmbeddings attribute)\nminimax_group_id (langchain.embeddings.MiniMaxEmbeddings attribute)\nminimum_tokens (langchain.llms.AlephAlpha attribute)\nminTokens (langchain.llms.AI21 attribute)\nmodel (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}75{"id": "eddc1cc06a3b-61", "text": "model (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\n(langchain.embeddings.CohereEmbeddings attribute)\n(langchain.embeddings.MiniMaxEmbeddings attribute)\n(langchain.llms.AI21 attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.CTransformers attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.RWKV attribute)\n(langchain.retrievers.document_compressors.CohereRerank attribute)\nmodel_file (langchain.llms.CTransformers attribute)\nmodel_id (langchain.embeddings.ModelScopeEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings attribute)\n(langchain.llms.HuggingFacePipeline attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.Writer attribute)\nmodel_key (langchain.llms.Banana attribute)\nmodel_kwargs (langchain.chat_models.ChatOpenAI attribute)\n(langchain.embeddings.HuggingFaceEmbeddings attribute)\n(langchain.embeddings.HuggingFaceHubEmbeddings attribute)\n(langchain.embeddings.HuggingFaceInstructEmbeddings attribute)\n(langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.Anyscale attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Banana attribute)\n(langchain.llms.Beam attribute)\n(langchain.llms.CerebriumAI attribute)\n(langchain.llms.Databricks attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.HuggingFaceEndpoint attribute)\n(langchain.llms.HuggingFaceHub attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}76{"id": "eddc1cc06a3b-62", "text": "(langchain.llms.HuggingFaceHub attribute)\n(langchain.llms.HuggingFacePipeline attribute)\n(langchain.llms.Modal attribute)\n(langchain.llms.MosaicML attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\n(langchain.llms.SagemakerEndpoint attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.StochasticAI attribute)\nmodel_load_fn (langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\nmodel_name (langchain.chains.OpenAIModerationChain attribute)\n(langchain.chat_models.ChatGooglePalm attribute)\n(langchain.chat_models.ChatOpenAI attribute)\n(langchain.chat_models.ChatVertexAI attribute)\n(langchain.embeddings.HuggingFaceEmbeddings attribute)\n(langchain.embeddings.HuggingFaceInstructEmbeddings attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\n(langchain.tools.SteamshipImageGenerationTool attribute)\nmodel_path (langchain.llms.LlamaCpp attribute)\nmodel_reqs (langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}77{"id": "eddc1cc06a3b-63", "text": "model_reqs (langchain.embeddings.SelfHostedHuggingFaceEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\nmodel_type (langchain.llms.CTransformers attribute)\nmodel_url (langchain.embeddings.TensorflowHubEmbeddings attribute)\nmodelname_to_contextsize() (langchain.llms.AzureOpenAI method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\nModernTreasuryLoader (class in langchain.document_loaders)\n    module\n      \nlangchain.agents\nlangchain.agents.agent_toolkits\nlangchain.chains\nlangchain.chat_models\nlangchain.docstore\nlangchain.document_loaders\nlangchain.document_transformers\nlangchain.embeddings\nlangchain.llms\nlangchain.memory\nlangchain.output_parsers\nlangchain.prompts\nlangchain.prompts.example_selector\nlangchain.python\nlangchain.retrievers\nlangchain.retrievers.document_compressors\nlangchain.serpapi\nlangchain.text_splitter\nlangchain.tools\nlangchain.utilities\nlangchain.utilities.searx_search\nlangchain.vectorstores\nMomentoChatMessageHistory (class in langchain.memory)\nMongoDBChatMessageHistory (class in langchain.memory)\nmoving_summary_buffer (langchain.memory.ConversationSummaryBufferMemory attribute)\nMWDumpLoader (class in langchain.document_loaders)\nMyScale (class in langchain.vectorstores)\nN\nn (langchain.chat_models.ChatGooglePalm attribute)\n(langchain.chat_models.ChatOpenAI attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}78{"id": "eddc1cc06a3b-64", "text": "(langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Writer attribute)\nn_batch (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nn_ctx (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nn_gpu_layers (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.LlamaCpp attribute)\nn_parts (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nn_predict (langchain.llms.GPT4All attribute)\nn_threads (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nname (langchain.agents.agent_toolkits.VectorStoreInfo attribute)\n(langchain.experimental.GenerativeAgent attribute)\n(langchain.llms.PredictionGuard attribute)\n(langchain.output_parsers.ResponseSchema attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.ClickTool attribute)\n(langchain.tools.CopyFileTool attribute)\n(langchain.tools.CurrentWebPageTool attribute)\n(langchain.tools.DeleteFileTool attribute)\n(langchain.tools.ExtractHyperlinksTool attribute)\n(langchain.tools.ExtractTextTool attribute)\n(langchain.tools.FileSearchTool attribute)\n(langchain.tools.GetElementsTool attribute)\n(langchain.tools.GmailCreateDraft attribute)\n(langchain.tools.GmailGetMessage attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}79{"id": "eddc1cc06a3b-65", "text": "(langchain.tools.GmailCreateDraft attribute)\n(langchain.tools.GmailGetMessage attribute)\n(langchain.tools.GmailGetThread attribute)\n(langchain.tools.GmailSearch attribute)\n(langchain.tools.GmailSendMessage attribute)\n(langchain.tools.ListDirectoryTool attribute)\n(langchain.tools.MoveFileTool attribute)\n(langchain.tools.NavigateBackTool attribute)\n(langchain.tools.NavigateTool attribute)\n(langchain.tools.ReadFileTool attribute)\n(langchain.tools.ShellTool attribute)\n(langchain.tools.Tool attribute)\n(langchain.tools.WriteFileTool attribute)\nnla_tools (langchain.agents.agent_toolkits.NLAToolkit attribute)\nNLTKTextSplitter (class in langchain.text_splitter)\nno_update_value (langchain.output_parsers.RegexDictParser attribute)\nnormalize (langchain.embeddings.AlephAlphaAsymmetricSemanticEmbedding attribute)\nNotebookLoader (class in langchain.document_loaders)\nNotionDBLoader (class in langchain.document_loaders)\nNotionDirectoryLoader (class in langchain.document_loaders)\nnum_beams (langchain.llms.NLPCloud attribute)\nnum_pad_tokens (langchain.chains.FlareChain attribute)\nnum_results (langchain.tools.BingSearchResults attribute)\n(langchain.tools.DuckDuckGoSearchResults attribute)\n(langchain.tools.GoogleSearchResults attribute)\nnum_return_sequences (langchain.llms.NLPCloud attribute)\nnumResults (langchain.llms.AI21 attribute)\nO\nobject_ids (langchain.document_loaders.OneDriveLoader attribute)\nobservation_prefix (langchain.agents.Agent property)\n(langchain.agents.ConversationalAgent property)\n(langchain.agents.ConversationalChatAgent property)\n(langchain.agents.StructuredChatAgent property)\n(langchain.agents.ZeroShotAgent property)\nObsidianLoader (class in langchain.document_loaders)", "source": "https://python.langchain.com/en/latest/genindex.html"}80{"id": "eddc1cc06a3b-66", "text": "ObsidianLoader (class in langchain.document_loaders)\nOnlinePDFLoader (class in langchain.document_loaders)\nopenai_api_base (langchain.chat_models.AzureChatOpenAI attribute)\n(langchain.chat_models.ChatOpenAI attribute)\nopenai_api_key (langchain.chains.OpenAIModerationChain attribute)\n(langchain.chat_models.AzureChatOpenAI attribute)\n(langchain.chat_models.ChatOpenAI attribute)\nopenai_api_type (langchain.chat_models.AzureChatOpenAI attribute)\nopenai_api_version (langchain.chat_models.AzureChatOpenAI attribute)\nopenai_organization (langchain.chains.OpenAIModerationChain attribute)\n(langchain.chat_models.AzureChatOpenAI attribute)\n(langchain.chat_models.ChatOpenAI attribute)\nopenai_proxy (langchain.chat_models.AzureChatOpenAI attribute)\n(langchain.chat_models.ChatOpenAI attribute)\nOpenSearchVectorSearch (class in langchain.vectorstores)\nopenweathermap_api_key (langchain.utilities.OpenWeatherMapAPIWrapper attribute)\noperation_id (langchain.tools.APIOperation attribute)\nother_score_keys (langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\nOutlookMessageLoader (class in langchain.document_loaders)\noutput_key (langchain.chains.QAGenerationChain attribute)\n(langchain.memory.ConversationStringBufferMemory attribute)\noutput_key_to_format (langchain.output_parsers.RegexDictParser attribute)\noutput_keys (langchain.chains.ConstitutionalChain property)\n(langchain.chains.FlareChain property)\n(langchain.chains.HypotheticalDocumentEmbedder property)\n(langchain.chains.QAGenerationChain property)\n(langchain.experimental.BabyAGI property)\n(langchain.output_parsers.RegexParser attribute)\noutput_parser (langchain.agents.Agent attribute)\n(langchain.agents.ConversationalAgent attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}81{"id": "eddc1cc06a3b-67", "text": "(langchain.agents.ConversationalAgent attribute)\n(langchain.agents.ConversationalChatAgent attribute)\n(langchain.agents.LLMSingleActionAgent attribute)\n(langchain.agents.StructuredChatAgent attribute)\n(langchain.agents.ZeroShotAgent attribute)\n(langchain.chains.FlareChain attribute)\n(langchain.prompts.BasePromptTemplate attribute)\noutput_variables (langchain.chains.TransformChain attribute)\nowm (langchain.utilities.OpenWeatherMapAPIWrapper attribute)\nP\np (langchain.llms.Cohere attribute)\npage_content_key (langchain.retrievers.RemoteLangChainRetriever attribute)\nPagedPDFSplitter (in module langchain.document_loaders)\npaginate_request() (langchain.document_loaders.ConfluenceLoader method)\nparam_mapping (langchain.chains.OpenAPIEndpointChain attribute)\nparams (langchain.serpapi.SerpAPIWrapper attribute)\n(langchain.tools.ZapierNLARunAction attribute)\n(langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\n(langchain.utilities.SerpAPIWrapper attribute)\nparams_schema (langchain.tools.ZapierNLARunAction attribute)\nparse() (langchain.agents.AgentOutputParser method)\n(langchain.output_parsers.CommaSeparatedListOutputParser method)\n(langchain.output_parsers.GuardrailsOutputParser method)\n(langchain.output_parsers.ListOutputParser method)\n(langchain.output_parsers.OutputFixingParser method)\n(langchain.output_parsers.PydanticOutputParser method)\n(langchain.output_parsers.RegexDictParser method)\n(langchain.output_parsers.RegexParser method)\n(langchain.output_parsers.RetryOutputParser method)\n(langchain.output_parsers.RetryWithErrorOutputParser method)\n(langchain.output_parsers.StructuredOutputParser method)", "source": "https://python.langchain.com/en/latest/genindex.html"}82{"id": "eddc1cc06a3b-68", "text": "(langchain.output_parsers.StructuredOutputParser method)\nparse_filename() (langchain.document_loaders.BlackboardLoader method)\nparse_obj() (langchain.tools.OpenAPISpec class method)\nparse_sitemap() (langchain.document_loaders.SitemapLoader method)\nparse_with_prompt() (langchain.output_parsers.RetryOutputParser method)\n(langchain.output_parsers.RetryWithErrorOutputParser method)\nparser (langchain.output_parsers.OutputFixingParser attribute)\n(langchain.output_parsers.RetryOutputParser attribute)\n(langchain.output_parsers.RetryWithErrorOutputParser attribute)\npartial() (langchain.prompts.BasePromptTemplate method)\n(langchain.prompts.ChatPromptTemplate method)\npassword (langchain.vectorstores.MyScaleSettings attribute)\npatch() (langchain.utilities.TextRequestsWrapper method)\npath (langchain.tools.APIOperation attribute)\npath_params (langchain.tools.APIOperation property)\npause_to_reflect() (langchain.experimental.GenerativeAgentMemory method)\nPDFMinerLoader (class in langchain.document_loaders)\nPDFMinerPDFasHTMLLoader (class in langchain.document_loaders)\nPDFPlumberLoader (class in langchain.document_loaders)\npenalty_alpha_frequency (langchain.llms.RWKV attribute)\npenalty_alpha_presence (langchain.llms.RWKV attribute)\npenalty_bias (langchain.llms.AlephAlpha attribute)\npenalty_exceptions (langchain.llms.AlephAlpha attribute)\npenalty_exceptions_include_stop_sequences (langchain.llms.AlephAlpha attribute)\npersist() (langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.SKLearnVectorStore method)\nPinecone (class in langchain.vectorstores)\npipeline_key (langchain.llms.PipelineAI attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}83{"id": "eddc1cc06a3b-69", "text": "pipeline_key (langchain.llms.PipelineAI attribute)\npipeline_kwargs (langchain.llms.HuggingFacePipeline attribute)\n(langchain.llms.PipelineAI attribute)\npl_tags (langchain.chat_models.PromptLayerChatOpenAI attribute)\nplan() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\n(langchain.agents.LLMSingleActionAgent method)\nplaywright_strict (langchain.tools.ClickTool attribute)\nplaywright_timeout (langchain.tools.ClickTool attribute)\nPlaywrightURLLoader (class in langchain.document_loaders)\nplugin (langchain.tools.AIPluginTool attribute)\nport (langchain.vectorstores.MyScaleSettings attribute)\npost() (langchain.utilities.TextRequestsWrapper method)\nPostgresChatMessageHistory (class in langchain.memory)\npowerbi (langchain.agents.agent_toolkits.PowerBIToolkit attribute)\n(langchain.tools.InfoPowerBITool attribute)\n(langchain.tools.ListPowerBITool attribute)\n(langchain.tools.QueryPowerBITool attribute)\npredict() (langchain.chains.LLMChain method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)", "source": "https://python.langchain.com/en/latest/genindex.html"}84{"id": "eddc1cc06a3b-70", "text": "(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\npredict_and_parse() (langchain.chains.LLMChain method)\npredict_messages() (langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)", "source": "https://python.langchain.com/en/latest/genindex.html"}85{"id": "eddc1cc06a3b-71", "text": "(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)", "source": "https://python.langchain.com/en/latest/genindex.html"}86{"id": "eddc1cc06a3b-72", "text": "(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\nprefix (langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\nprefix_messages (langchain.llms.OpenAIChat attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)\nprep_prompts() (langchain.chains.LLMChain method)\nprep_streaming_params() (langchain.llms.AzureOpenAI method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\nprepare_cosmos() (langchain.memory.CosmosDBChatMessageHistory method)\npresence_penalty (langchain.llms.AlephAlpha attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Writer attribute)\npresencePenalty (langchain.llms.AI21 attribute)\nprioritize_tasks() (langchain.experimental.BabyAGI method)\nprocess (langchain.tools.ShellTool attribute)\nprocess_attachment() (langchain.document_loaders.ConfluenceLoader method)\nprocess_doc() (langchain.document_loaders.ConfluenceLoader method)\nprocess_image() (langchain.document_loaders.ConfluenceLoader method)\nprocess_index_results() (langchain.vectorstores.Annoy method)\nprocess_output() (langchain.utilities.BashProcess method)\nprocess_page() (langchain.document_loaders.ConfluenceLoader method)\nprocess_pages() (langchain.document_loaders.ConfluenceLoader method)\nprocess_pdf() (langchain.document_loaders.ConfluenceLoader method)\nprocess_svg() (langchain.document_loaders.ConfluenceLoader method)", "source": "https://python.langchain.com/en/latest/genindex.html"}87{"id": "eddc1cc06a3b-73", "text": "process_svg() (langchain.document_loaders.ConfluenceLoader method)\nprocess_xls() (langchain.document_loaders.ConfluenceLoader method)\nproject (langchain.llms.VertexAI attribute)\nPrompt (in module langchain.prompts)\nprompt (langchain.chains.ConversationChain attribute)\n(langchain.chains.LLMBashChain attribute)\n(langchain.chains.LLMChain attribute)\n(langchain.chains.LLMMathChain attribute)\n(langchain.chains.PALChain attribute)\n(langchain.chains.SQLDatabaseChain attribute)\nprompt_func (langchain.tools.HumanInputRun attribute)\nproperties (langchain.tools.APIOperation attribute)\nprune() (langchain.memory.ConversationSummaryBufferMemory method)\nPsychicLoader (class in langchain.document_loaders)\nput() (langchain.utilities.TextRequestsWrapper method)\npydantic_object (langchain.output_parsers.PydanticOutputParser attribute)\nPyMuPDFLoader (class in langchain.document_loaders)\nPyPDFDirectoryLoader (class in langchain.document_loaders)\nPyPDFium2Loader (class in langchain.document_loaders)\nPyPDFLoader (class in langchain.document_loaders)\npython_globals (langchain.chains.PALChain attribute)\npython_locals (langchain.chains.PALChain attribute)\nPythonCodeTextSplitter (class in langchain.text_splitter)\nPythonLoader (class in langchain.document_loaders)\nQ\nqa_chain (langchain.chains.GraphCypherQAChain attribute)\n(langchain.chains.GraphQAChain attribute)\nQdrant (class in langchain.vectorstores)\nquery_checker_prompt (langchain.chains.SQLDatabaseChain attribute)\nquery_instruction (langchain.embeddings.HuggingFaceInstructEmbeddings attribute)\n(langchain.embeddings.MosaicMLInstructorEmbeddings attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}88{"id": "eddc1cc06a3b-74", "text": "(langchain.embeddings.MosaicMLInstructorEmbeddings attribute)\n(langchain.embeddings.SelfHostedHuggingFaceInstructEmbeddings attribute)\nquery_name (langchain.vectorstores.SupabaseVectorStore attribute)\nquery_params (langchain.tools.APIOperation property)\nquery_suffix (langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nquestion_generator_chain (langchain.chains.FlareChain attribute)\nquestion_to_checked_assertions_chain (langchain.chains.LLMCheckerChain attribute)\nR\nraw_completion (langchain.llms.AlephAlpha attribute)\nREACT_DOCSTORE (langchain.agents.AgentType attribute)\nReadTheDocsLoader (class in langchain.document_loaders)\nrecall_ttl (langchain.memory.RedisEntityStore attribute)\nrecursive (langchain.document_loaders.GoogleDriveLoader attribute)\nRecursiveCharacterTextSplitter (class in langchain.text_splitter)\nRedditPostsLoader (class in langchain.document_loaders)\nRedis (class in langchain.vectorstores)\nredis_client (langchain.memory.RedisEntityStore attribute)\nRedisChatMessageHistory (class in langchain.memory)\nRedisEntityStore (class in langchain.memory)\nreduce_k_below_max_tokens (langchain.chains.RetrievalQAWithSourcesChain attribute)\n(langchain.chains.VectorDBQAWithSourcesChain attribute)\nreflection_threshold (langchain.experimental.GenerativeAgentMemory attribute)\nregex (langchain.output_parsers.RegexParser attribute)\nregex_pattern (langchain.output_parsers.RegexDictParser attribute)\nregion (langchain.utilities.DuckDuckGoSearchAPIWrapper attribute)\nregion_name (langchain.embeddings.SagemakerEndpointEmbeddings attribute)\n(langchain.llms.SagemakerEndpoint attribute)\nrelevancy_threshold (langchain.retrievers.KNNRetriever attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}89{"id": "eddc1cc06a3b-75", "text": "relevancy_threshold (langchain.retrievers.KNNRetriever attribute)\n(langchain.retrievers.SVMRetriever attribute)\nremove_end_sequence (langchain.llms.NLPCloud attribute)\nremove_input (langchain.llms.NLPCloud attribute)\nrepeat_last_n (langchain.llms.GPT4All attribute)\nrepeat_penalty (langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nrepetition_penalties_include_completion (langchain.llms.AlephAlpha attribute)\nrepetition_penalties_include_prompt (langchain.llms.AlephAlpha attribute)\nrepetition_penalty (langchain.llms.ForefrontAI attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.Writer attribute)\nrepo_id (langchain.embeddings.HuggingFaceHubEmbeddings attribute)\n(langchain.llms.HuggingFaceHub attribute)\nrequest_body (langchain.tools.APIOperation attribute)\nrequest_timeout (langchain.chat_models.ChatOpenAI attribute)\n(langchain.embeddings.OpenAIEmbeddings attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\nrequest_url (langchain.utilities.PowerBIDataset property)\nrequests (langchain.chains.OpenAPIEndpointChain attribute)\n(langchain.utilities.TextRequestsWrapper property)\nrequests_per_second (langchain.document_loaders.WebBaseLoader attribute)\nrequests_wrapper (langchain.agents.agent_toolkits.OpenAPIToolkit attribute)\n(langchain.chains.APIChain attribute)\n(langchain.chains.LLMRequestsChain attribute)\nresponse_chain (langchain.chains.FlareChain attribute)\nresponse_key (langchain.retrievers.RemoteLangChainRetriever attribute)\nresponse_schemas (langchain.output_parsers.StructuredOutputParser attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}90{"id": "eddc1cc06a3b-76", "text": "response_schemas (langchain.output_parsers.StructuredOutputParser attribute)\nresults() (langchain.serpapi.SerpAPIWrapper method)\n(langchain.utilities.BingSearchAPIWrapper method)\n(langchain.utilities.DuckDuckGoSearchAPIWrapper method)\n(langchain.utilities.GoogleSearchAPIWrapper method)\n(langchain.utilities.GoogleSerperAPIWrapper method)\n(langchain.utilities.MetaphorSearchAPIWrapper method)\n(langchain.utilities.searx_search.SearxSearchWrapper method)\n(langchain.utilities.SearxSearchWrapper method)\n(langchain.utilities.SerpAPIWrapper method)\nresults_async() (langchain.utilities.MetaphorSearchAPIWrapper method)\nretriever (langchain.chains.ConversationalRetrievalChain attribute)\n(langchain.chains.FlareChain attribute)\n(langchain.chains.RetrievalQA attribute)\n(langchain.chains.RetrievalQAWithSourcesChain attribute)\n(langchain.memory.VectorStoreRetrieverMemory attribute)\nretry_chain (langchain.output_parsers.OutputFixingParser attribute)\n(langchain.output_parsers.RetryOutputParser attribute)\n(langchain.output_parsers.RetryWithErrorOutputParser attribute)\nretry_sleep (langchain.embeddings.MosaicMLInstructorEmbeddings attribute)\n(langchain.llms.MosaicML attribute)\nreturn_all (langchain.chains.SequentialChain attribute)\nreturn_direct (langchain.chains.SQLDatabaseChain attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.Tool attribute)\nreturn_docs (langchain.memory.VectorStoreRetrieverMemory attribute)\nreturn_intermediate_steps (langchain.agents.AgentExecutor attribute)\n(langchain.chains.ConstitutionalChain attribute)\n(langchain.chains.OpenAPIEndpointChain attribute)\n(langchain.chains.PALChain attribute)\n(langchain.chains.SQLDatabaseChain attribute)\n(langchain.chains.SQLDatabaseSequentialChain attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}91{"id": "eddc1cc06a3b-77", "text": "(langchain.chains.SQLDatabaseChain attribute)\n(langchain.chains.SQLDatabaseSequentialChain attribute)\nreturn_pl_id (langchain.chat_models.PromptLayerChatOpenAI attribute)\nreturn_stopped_response() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\nreturn_urls (langchain.tools.SteamshipImageGenerationTool attribute)\nreturn_values (langchain.agents.Agent property)\n(langchain.agents.BaseMultiActionAgent property)\n(langchain.agents.BaseSingleActionAgent property)\nrevised_answer_prompt (langchain.chains.LLMCheckerChain attribute)\nrevised_summary_prompt (langchain.chains.LLMSummarizationCheckerChain attribute)\nrevision_chain (langchain.chains.ConstitutionalChain attribute)\nRoamLoader (class in langchain.document_loaders)\nroot_dir (langchain.agents.agent_toolkits.FileManagementToolkit attribute)\nrun() (langchain.python.PythonREPL method)\n(langchain.serpapi.SerpAPIWrapper method)\n(langchain.tools.BaseTool method)\n(langchain.utilities.ArxivAPIWrapper method)\n(langchain.utilities.BashProcess method)\n(langchain.utilities.BingSearchAPIWrapper method)\n(langchain.utilities.DuckDuckGoSearchAPIWrapper method)\n(langchain.utilities.GooglePlacesAPIWrapper method)\n(langchain.utilities.GoogleSearchAPIWrapper method)\n(langchain.utilities.GoogleSerperAPIWrapper method)\n(langchain.utilities.GraphQLAPIWrapper method)\n(langchain.utilities.LambdaWrapper method)\n(langchain.utilities.OpenWeatherMapAPIWrapper method)\n(langchain.utilities.PowerBIDataset method)\n(langchain.utilities.PythonREPL method)\n(langchain.utilities.searx_search.SearxSearchWrapper method)\n(langchain.utilities.SearxSearchWrapper method)\n(langchain.utilities.SerpAPIWrapper method)", "source": "https://python.langchain.com/en/latest/genindex.html"}92{"id": "eddc1cc06a3b-78", "text": "(langchain.utilities.SearxSearchWrapper method)\n(langchain.utilities.SerpAPIWrapper method)\n(langchain.utilities.SparkSQL method)\n(langchain.utilities.TwilioAPIWrapper method)\n(langchain.utilities.WikipediaAPIWrapper method)\n(langchain.utilities.WolframAlphaAPIWrapper method)\nrun_creation() (langchain.llms.Beam method)\nrun_no_throw() (langchain.utilities.SparkSQL method)\nrwkv_verbose (langchain.llms.RWKV attribute)\nS\nS3DirectoryLoader (class in langchain.document_loaders)\nS3FileLoader (class in langchain.document_loaders)\nsafesearch (langchain.utilities.DuckDuckGoSearchAPIWrapper attribute)\nsample_rows_in_table_info (langchain.utilities.PowerBIDataset attribute)\nsave() (langchain.agents.AgentExecutor method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\n(langchain.llms.AI21 method)\n(langchain.llms.AlephAlpha method)\n(langchain.llms.Anthropic method)\n(langchain.llms.Anyscale method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.Banana method)\n(langchain.llms.Beam method)\n(langchain.llms.CerebriumAI method)\n(langchain.llms.Cohere method)\n(langchain.llms.CTransformers method)\n(langchain.llms.Databricks method)\n(langchain.llms.DeepInfra method)\n(langchain.llms.FakeListLLM method)\n(langchain.llms.ForefrontAI method)\n(langchain.llms.GooglePalm method)\n(langchain.llms.GooseAI method)\n(langchain.llms.GPT4All method)\n(langchain.llms.HuggingFaceEndpoint method)\n(langchain.llms.HuggingFaceHub method)", "source": "https://python.langchain.com/en/latest/genindex.html"}93{"id": "eddc1cc06a3b-79", "text": "(langchain.llms.HuggingFaceHub method)\n(langchain.llms.HuggingFacePipeline method)\n(langchain.llms.HuggingFaceTextGenInference method)\n(langchain.llms.HumanInputLLM method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.Modal method)\n(langchain.llms.MosaicML method)\n(langchain.llms.NLPCloud method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenAIChat method)\n(langchain.llms.OpenLM method)\n(langchain.llms.Petals method)\n(langchain.llms.PipelineAI method)\n(langchain.llms.PredictionGuard method)\n(langchain.llms.PromptLayerOpenAI method)\n(langchain.llms.PromptLayerOpenAIChat method)\n(langchain.llms.Replicate method)\n(langchain.llms.RWKV method)\n(langchain.llms.SagemakerEndpoint method)\n(langchain.llms.SelfHostedHuggingFaceLLM method)\n(langchain.llms.SelfHostedPipeline method)\n(langchain.llms.StochasticAI method)\n(langchain.llms.VertexAI method)\n(langchain.llms.Writer method)\n(langchain.prompts.BasePromptTemplate method)\n(langchain.prompts.ChatPromptTemplate method)\nsave_agent() (langchain.agents.AgentExecutor method)\nsave_context() (langchain.experimental.GenerativeAgentMemory method)\n(langchain.memory.CombinedMemory method)\n(langchain.memory.ConversationEntityMemory method)\n(langchain.memory.ConversationKGMemory method)\n(langchain.memory.ConversationStringBufferMemory method)\n(langchain.memory.ConversationSummaryBufferMemory method)\n(langchain.memory.ConversationSummaryMemory method)\n(langchain.memory.ConversationTokenBufferMemory method)\n(langchain.memory.ReadOnlySharedMemory method)\n(langchain.memory.SimpleMemory method)", "source": "https://python.langchain.com/en/latest/genindex.html"}94{"id": "eddc1cc06a3b-80", "text": "(langchain.memory.ReadOnlySharedMemory method)\n(langchain.memory.SimpleMemory method)\n(langchain.memory.VectorStoreRetrieverMemory method)\nsave_local() (langchain.vectorstores.Annoy method)\n(langchain.vectorstores.FAISS method)\nschemas (langchain.utilities.PowerBIDataset attribute)\nscrape() (langchain.document_loaders.WebBaseLoader method)\nscrape_all() (langchain.document_loaders.WebBaseLoader method)\nscrape_page() (langchain.tools.ExtractHyperlinksTool static method)\nsearch() (langchain.docstore.InMemoryDocstore method)\n(langchain.docstore.Wikipedia method)\n(langchain.vectorstores.VectorStore method)\nsearch_kwargs (langchain.chains.ChatVectorDBChain attribute)\n(langchain.chains.VectorDBQA attribute)\n(langchain.chains.VectorDBQAWithSourcesChain attribute)\n(langchain.retrievers.SelfQueryRetriever attribute)\n(langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\nsearch_type (langchain.chains.VectorDBQA attribute)\n(langchain.retrievers.SelfQueryRetriever attribute)\nsearx_host (langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nSearxResults (class in langchain.utilities.searx_search)\nseed (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nselect_examples() (langchain.prompts.example_selector.LengthBasedExampleSelector method)\n(langchain.prompts.example_selector.MaxMarginalRelevanceExampleSelector method)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector method)\nselected_tools (langchain.agents.agent_toolkits.FileManagementToolkit attribute)\nSeleniumURLLoader (class in langchain.document_loaders)", "source": "https://python.langchain.com/en/latest/genindex.html"}95{"id": "eddc1cc06a3b-81", "text": "SeleniumURLLoader (class in langchain.document_loaders)\nSELF_ASK_WITH_SEARCH (langchain.agents.AgentType attribute)\nsend_pdf() (langchain.document_loaders.MathpixPDFLoader method)\nSentenceTransformerEmbeddings (in module langchain.embeddings)\nsequential_chain (langchain.chains.LLMSummarizationCheckerChain attribute)\nserpapi_api_key (langchain.serpapi.SerpAPIWrapper attribute)\n(langchain.utilities.SerpAPIWrapper attribute)\nserper_api_key (langchain.utilities.GoogleSerperAPIWrapper attribute)\nservice_account_key (langchain.document_loaders.GoogleDriveLoader attribute)\nservice_account_path (langchain.document_loaders.GoogleApiClient attribute)\nservice_name (langchain.retrievers.AzureCognitiveSearchRetriever attribute)\nsession_cache (langchain.tools.QueryPowerBITool attribute)\nsession_id (langchain.memory.RedisEntityStore attribute)\nset() (langchain.memory.InMemoryEntityStore method)\n(langchain.memory.RedisEntityStore method)\nsettings (langchain.document_loaders.OneDriveLoader attribute)\nsimilarity_fn (langchain.document_transformers.EmbeddingsRedundantFilter attribute)\n(langchain.retrievers.document_compressors.EmbeddingsFilter attribute)\nsimilarity_search() (langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Annoy method)\n(langchain.vectorstores.AtlasDB method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.ElasticVectorSearch method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.LanceDB method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.MyScale method)\n(langchain.vectorstores.OpenSearchVectorSearch method)\n(langchain.vectorstores.Pinecone method)\n(langchain.vectorstores.Qdrant method)", "source": "https://python.langchain.com/en/latest/genindex.html"}96{"id": "eddc1cc06a3b-82", "text": "(langchain.vectorstores.Pinecone method)\n(langchain.vectorstores.Qdrant method)\n(langchain.vectorstores.Redis method)\n(langchain.vectorstores.SKLearnVectorStore method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.Tair method)\n(langchain.vectorstores.Typesense method)\n(langchain.vectorstores.Vectara method)\n(langchain.vectorstores.VectorStore method)\n(langchain.vectorstores.Weaviate method)\nsimilarity_search_by_index() (langchain.vectorstores.Annoy method)\nsimilarity_search_by_text() (langchain.vectorstores.Weaviate method)\nsimilarity_search_by_vector() (langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Annoy method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.MyScale method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.VectorStore method)\n(langchain.vectorstores.Weaviate method)\nsimilarity_search_by_vector_returning_embeddings() (langchain.vectorstores.SupabaseVectorStore method)\nsimilarity_search_by_vector_with_relevance_scores() (langchain.vectorstores.SupabaseVectorStore method)\nsimilarity_search_limit_score() (langchain.vectorstores.Redis method)\nsimilarity_search_with_relevance_scores() (langchain.vectorstores.MyScale method)\n(langchain.vectorstores.SupabaseVectorStore method)\n(langchain.vectorstores.VectorStore method)\nsimilarity_search_with_score() (langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Annoy method)\n(langchain.vectorstores.Chroma method)\n(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.ElasticVectorSearch method)", "source": "https://python.langchain.com/en/latest/genindex.html"}97{"id": "eddc1cc06a3b-83", "text": "(langchain.vectorstores.DeepLake method)\n(langchain.vectorstores.ElasticVectorSearch method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.Milvus method)\n(langchain.vectorstores.OpenSearchVectorSearch method)\n(langchain.vectorstores.Pinecone method)\n(langchain.vectorstores.Qdrant method)\n(langchain.vectorstores.Redis method)\n(langchain.vectorstores.SKLearnVectorStore method)\n(langchain.vectorstores.Typesense method)\n(langchain.vectorstores.Vectara method)\n(langchain.vectorstores.Weaviate method)\nsimilarity_search_with_score_by_index() (langchain.vectorstores.Annoy method)\nsimilarity_search_with_score_by_vector() (langchain.vectorstores.AnalyticDB method)\n(langchain.vectorstores.Annoy method)\n(langchain.vectorstores.FAISS method)\n(langchain.vectorstores.Milvus method)\nsimilarity_threshold (langchain.document_transformers.EmbeddingsRedundantFilter attribute)\n(langchain.retrievers.document_compressors.EmbeddingsFilter attribute)\nSitemapLoader (class in langchain.document_loaders)\nsiterestrict (langchain.utilities.GoogleSearchAPIWrapper attribute)\nsize (langchain.tools.SteamshipImageGenerationTool attribute)\nSKLearnVectorStore (class in langchain.vectorstores)\nSlackDirectoryLoader (class in langchain.document_loaders)\nSpacyTextSplitter (class in langchain.text_splitter)\nSparkSQL (class in langchain.utilities)\nsparse_encoder (langchain.retrievers.PineconeHybridSearchRetriever attribute)\nspec (langchain.agents.agent_toolkits.JsonToolkit attribute)\nsplit_documents() (langchain.text_splitter.TextSplitter method)\nsplit_text() (langchain.text_splitter.CharacterTextSplitter method)\n(langchain.text_splitter.NLTKTextSplitter method)", "source": "https://python.langchain.com/en/latest/genindex.html"}98{"id": "eddc1cc06a3b-84", "text": "(langchain.text_splitter.NLTKTextSplitter method)\n(langchain.text_splitter.RecursiveCharacterTextSplitter method)\n(langchain.text_splitter.SpacyTextSplitter method)\n(langchain.text_splitter.TextSplitter method)\n(langchain.text_splitter.TokenTextSplitter method)\nSpreedlyLoader (class in langchain.document_loaders)\nsql_chain (langchain.chains.SQLDatabaseSequentialChain attribute)\nSRTLoader (class in langchain.document_loaders)\nstart_with_retrieval (langchain.chains.FlareChain attribute)\nstatus (langchain.experimental.GenerativeAgent attribute)\nsteamship (langchain.tools.SteamshipImageGenerationTool attribute)\nstop (langchain.agents.LLMSingleActionAgent attribute)\n(langchain.chains.PALChain attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.Writer attribute)\nstop_sequences (langchain.llms.AlephAlpha attribute)\nstore (langchain.memory.InMemoryEntityStore attribute)\nstrategy (langchain.llms.RWKV attribute)\nstream() (langchain.llms.Anthropic method)\n(langchain.llms.AzureOpenAI method)\n(langchain.llms.LlamaCpp method)\n(langchain.llms.OpenAI method)\n(langchain.llms.OpenLM method)\n(langchain.llms.PromptLayerOpenAI method)\nstreaming (langchain.chat_models.ChatOpenAI attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.PromptLayerOpenAIChat attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}99{"id": "eddc1cc06a3b-85", "text": "(langchain.llms.PromptLayerOpenAIChat attribute)\nstrip_outputs (langchain.chains.SimpleSequentialChain attribute)\nStripeLoader (class in langchain.document_loaders)\nSTRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION (langchain.agents.AgentType attribute)\nstructured_query_translator (langchain.retrievers.SelfQueryRetriever attribute)\nsuffix (langchain.llms.LlamaCpp attribute)\n(langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\nsummarize_related_memories() (langchain.experimental.GenerativeAgent method)\nsummary (langchain.experimental.GenerativeAgent attribute)\nsummary_message_cls (langchain.memory.ConversationKGMemory attribute)\nsummary_refresh_seconds (langchain.experimental.GenerativeAgent attribute)\nSupabaseVectorStore (class in langchain.vectorstores)\nsync_browser (langchain.agents.agent_toolkits.PlayWrightBrowserToolkit attribute)\nT\ntable (langchain.vectorstores.MyScaleSettings attribute)\ntable_info (langchain.utilities.PowerBIDataset property)\ntable_name (langchain.vectorstores.SupabaseVectorStore attribute)\ntable_names (langchain.utilities.PowerBIDataset attribute)\nTair (class in langchain.vectorstores)\ntask (langchain.embeddings.HuggingFaceHubEmbeddings attribute)\n(langchain.llms.HuggingFaceEndpoint attribute)\n(langchain.llms.HuggingFaceHub attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\ntbs (langchain.utilities.GoogleSerperAPIWrapper attribute)\nTelegramChatApiLoader (class in langchain.document_loaders)\nTelegramChatFileLoader (class in langchain.document_loaders)\nTelegramChatLoader (in module langchain.document_loaders)\ntemp (langchain.llms.GPT4All attribute)\ntemperature (langchain.chat_models.ChatGooglePalm attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}100{"id": "eddc1cc06a3b-86", "text": "temperature (langchain.chat_models.ChatGooglePalm attribute)\n(langchain.chat_models.ChatOpenAI attribute)\n(langchain.llms.AI21 attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.PredictionGuard attribute)\n(langchain.llms.RWKV attribute)\n(langchain.llms.VertexAI attribute)\n(langchain.llms.Writer attribute)\ntemplate (langchain.prompts.PromptTemplate attribute)\n(langchain.tools.QueryPowerBITool attribute)\ntemplate_format (langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\n(langchain.prompts.PromptTemplate attribute)\ntemplate_tool_response (langchain.agents.ConversationalChatAgent attribute)\ntext_length (langchain.chains.LLMRequestsChain attribute)\ntext_splitter (langchain.chains.AnalyzeDocumentChain attribute)\n(langchain.chains.MapReduceChain attribute)\n(langchain.chains.QAGenerationChain attribute)\nTextLoader (class in langchain.document_loaders)\ntexts (langchain.retrievers.KNNRetriever attribute)\n(langchain.retrievers.SVMRetriever attribute)\nTextSplitter (class in langchain.text_splitter)\ntfidf_array (langchain.retrievers.TFIDFRetriever attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}101{"id": "eddc1cc06a3b-87", "text": "tfidf_array (langchain.retrievers.TFIDFRetriever attribute)\ntime (langchain.utilities.DuckDuckGoSearchAPIWrapper attribute)\nto_typescript() (langchain.tools.APIOperation method)\ntoken (langchain.utilities.PowerBIDataset attribute)\ntoken_path (langchain.document_loaders.GoogleApiClient attribute)\n(langchain.document_loaders.GoogleDriveLoader attribute)\ntokenizer (langchain.llms.Petals attribute)\ntokens (langchain.llms.AlephAlpha attribute)\ntokens_path (langchain.llms.RWKV attribute)\nTokenTextSplitter (class in langchain.text_splitter)\nToMarkdownLoader (class in langchain.document_loaders)\nTomlLoader (class in langchain.document_loaders)\ntool() (in module langchain.agents)\n(in module langchain.tools)\ntool_run_logging_kwargs() (langchain.agents.Agent method)\n(langchain.agents.BaseMultiActionAgent method)\n(langchain.agents.BaseSingleActionAgent method)\n(langchain.agents.LLMSingleActionAgent method)\ntools (langchain.agents.agent_toolkits.JiraToolkit attribute)\n(langchain.agents.agent_toolkits.ZapierToolkit attribute)\n(langchain.agents.AgentExecutor attribute)\ntop_k (langchain.chains.SQLDatabaseChain attribute)\n(langchain.chat_models.ChatGooglePalm attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.VertexAI attribute)\n(langchain.retrievers.ChatGPTPluginRetriever attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}102{"id": "eddc1cc06a3b-88", "text": "(langchain.retrievers.ChatGPTPluginRetriever attribute)\n(langchain.retrievers.DataberryRetriever attribute)\n(langchain.retrievers.PineconeHybridSearchRetriever attribute)\ntop_k_docs_for_context (langchain.chains.ChatVectorDBChain attribute)\ntop_k_results (langchain.utilities.ArxivAPIWrapper attribute)\n(langchain.utilities.GooglePlacesAPIWrapper attribute)\n(langchain.utilities.WikipediaAPIWrapper attribute)\ntop_n (langchain.retrievers.document_compressors.CohereRerank attribute)\ntop_p (langchain.chat_models.ChatGooglePalm attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.RWKV attribute)\n(langchain.llms.VertexAI attribute)\n(langchain.llms.Writer attribute)\ntopP (langchain.llms.AI21 attribute)\ntraits (langchain.experimental.GenerativeAgent attribute)\ntransform (langchain.chains.TransformChain attribute)\ntransform_documents() (langchain.document_transformers.EmbeddingsRedundantFilter method)\n(langchain.text_splitter.TextSplitter method)\ntransform_input_fn (langchain.llms.Databricks attribute)\ntransform_output_fn (langchain.llms.Databricks attribute)\ntransformers (langchain.retrievers.document_compressors.DocumentCompressorPipeline attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}103{"id": "eddc1cc06a3b-89", "text": "transformers (langchain.retrievers.document_compressors.DocumentCompressorPipeline attribute)\ntruncate (langchain.embeddings.CohereEmbeddings attribute)\n(langchain.llms.Cohere attribute)\nts_type_from_python() (langchain.tools.APIOperation static method)\nttl (langchain.memory.RedisEntityStore attribute)\ntuned_model_name (langchain.llms.VertexAI attribute)\nTwitterTweetLoader (class in langchain.document_loaders)\ntype (langchain.utilities.GoogleSerperAPIWrapper attribute)\nTypesense (class in langchain.vectorstores)\nU\nunsecure (langchain.utilities.searx_search.SearxSearchWrapper attribute)\n(langchain.utilities.SearxSearchWrapper attribute)\nUnstructuredAPIFileIOLoader (class in langchain.document_loaders)\nUnstructuredAPIFileLoader (class in langchain.document_loaders)\nUnstructuredEmailLoader (class in langchain.document_loaders)\nUnstructuredEPubLoader (class in langchain.document_loaders)\nUnstructuredFileIOLoader (class in langchain.document_loaders)\nUnstructuredFileLoader (class in langchain.document_loaders)\nUnstructuredHTMLLoader (class in langchain.document_loaders)\nUnstructuredImageLoader (class in langchain.document_loaders)\nUnstructuredMarkdownLoader (class in langchain.document_loaders)\nUnstructuredODTLoader (class in langchain.document_loaders)\nUnstructuredPDFLoader (class in langchain.document_loaders)\nUnstructuredPowerPointLoader (class in langchain.document_loaders)\nUnstructuredRTFLoader (class in langchain.document_loaders)\nUnstructuredURLLoader (class in langchain.document_loaders)\nUnstructuredWordDocumentLoader (class in langchain.document_loaders)\nupdate_document() (langchain.vectorstores.Chroma method)\nupdate_forward_refs() (langchain.llms.AI21 class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}104{"id": "eddc1cc06a3b-90", "text": "update_forward_refs() (langchain.llms.AI21 class method)\n(langchain.llms.AlephAlpha class method)\n(langchain.llms.Anthropic class method)\n(langchain.llms.Anyscale class method)\n(langchain.llms.AzureOpenAI class method)\n(langchain.llms.Banana class method)\n(langchain.llms.Beam class method)\n(langchain.llms.CerebriumAI class method)\n(langchain.llms.Cohere class method)\n(langchain.llms.CTransformers class method)\n(langchain.llms.Databricks class method)\n(langchain.llms.DeepInfra class method)\n(langchain.llms.FakeListLLM class method)\n(langchain.llms.ForefrontAI class method)\n(langchain.llms.GooglePalm class method)\n(langchain.llms.GooseAI class method)\n(langchain.llms.GPT4All class method)\n(langchain.llms.HuggingFaceEndpoint class method)\n(langchain.llms.HuggingFaceHub class method)\n(langchain.llms.HuggingFacePipeline class method)\n(langchain.llms.HuggingFaceTextGenInference class method)\n(langchain.llms.HumanInputLLM class method)\n(langchain.llms.LlamaCpp class method)\n(langchain.llms.Modal class method)\n(langchain.llms.MosaicML class method)\n(langchain.llms.NLPCloud class method)\n(langchain.llms.OpenAI class method)\n(langchain.llms.OpenAIChat class method)\n(langchain.llms.OpenLM class method)\n(langchain.llms.Petals class method)\n(langchain.llms.PipelineAI class method)\n(langchain.llms.PredictionGuard class method)\n(langchain.llms.PromptLayerOpenAI class method)\n(langchain.llms.PromptLayerOpenAIChat class method)", "source": "https://python.langchain.com/en/latest/genindex.html"}105{"id": "eddc1cc06a3b-91", "text": "(langchain.llms.PromptLayerOpenAIChat class method)\n(langchain.llms.Replicate class method)\n(langchain.llms.RWKV class method)\n(langchain.llms.SagemakerEndpoint class method)\n(langchain.llms.SelfHostedHuggingFaceLLM class method)\n(langchain.llms.SelfHostedPipeline class method)\n(langchain.llms.StochasticAI class method)\n(langchain.llms.VertexAI class method)\n(langchain.llms.Writer class method)\nupsert_messages() (langchain.memory.CosmosDBChatMessageHistory method)\nurl (langchain.document_loaders.MathpixPDFLoader property)\n(langchain.llms.Beam attribute)\n(langchain.retrievers.ChatGPTPluginRetriever attribute)\n(langchain.retrievers.RemoteLangChainRetriever attribute)\n(langchain.tools.IFTTTWebhook attribute)\nurls (langchain.document_loaders.PlaywrightURLLoader attribute)\n(langchain.document_loaders.SeleniumURLLoader attribute)\nuse_mlock (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nuse_mmap (langchain.llms.LlamaCpp attribute)\nuse_multiplicative_presence_penalty (langchain.llms.AlephAlpha attribute)\nuse_query_checker (langchain.chains.SQLDatabaseChain attribute)\nusername (langchain.vectorstores.MyScaleSettings attribute)\nV\nvalidate_channel_or_videoIds_is_set() (langchain.document_loaders.GoogleApiClient class method)\n(langchain.document_loaders.GoogleApiYoutubeLoader class method)\nvalidate_init_args() (langchain.document_loaders.ConfluenceLoader static method)\nvalidate_template (langchain.prompts.FewShotPromptTemplate attribute)\n(langchain.prompts.FewShotPromptWithTemplates attribute)\n(langchain.prompts.PromptTemplate attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}106{"id": "eddc1cc06a3b-92", "text": "(langchain.prompts.PromptTemplate attribute)\nVectara (class in langchain.vectorstores)\nvectorizer (langchain.retrievers.TFIDFRetriever attribute)\nVectorStore (class in langchain.vectorstores)\nvectorstore (langchain.agents.agent_toolkits.VectorStoreInfo attribute)\n(langchain.chains.ChatVectorDBChain attribute)\n(langchain.chains.VectorDBQA attribute)\n(langchain.chains.VectorDBQAWithSourcesChain attribute)\n(langchain.prompts.example_selector.SemanticSimilarityExampleSelector attribute)\n(langchain.retrievers.SelfQueryRetriever attribute)\n(langchain.retrievers.TimeWeightedVectorStoreRetriever attribute)\nvectorstore_info (langchain.agents.agent_toolkits.VectorStoreToolkit attribute)\nvectorstores (langchain.agents.agent_toolkits.VectorStoreRouterToolkit attribute)\nverbose (langchain.llms.AI21 attribute)\n(langchain.llms.AlephAlpha attribute)\n(langchain.llms.Anthropic attribute)\n(langchain.llms.Anyscale attribute)\n(langchain.llms.AzureOpenAI attribute)\n(langchain.llms.Banana attribute)\n(langchain.llms.Beam attribute)\n(langchain.llms.CerebriumAI attribute)\n(langchain.llms.Cohere attribute)\n(langchain.llms.CTransformers attribute)\n(langchain.llms.Databricks attribute)\n(langchain.llms.DeepInfra attribute)\n(langchain.llms.FakeListLLM attribute)\n(langchain.llms.ForefrontAI attribute)\n(langchain.llms.GooglePalm attribute)\n(langchain.llms.GooseAI attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.HuggingFaceEndpoint attribute)\n(langchain.llms.HuggingFaceHub attribute)\n(langchain.llms.HuggingFacePipeline attribute)\n(langchain.llms.HuggingFaceTextGenInference attribute)", "source": "https://python.langchain.com/en/latest/genindex.html"}107{"id": "eddc1cc06a3b-93", "text": "(langchain.llms.HuggingFaceTextGenInference attribute)\n(langchain.llms.HumanInputLLM attribute)\n(langchain.llms.LlamaCpp attribute)\n(langchain.llms.Modal attribute)\n(langchain.llms.MosaicML attribute)\n(langchain.llms.NLPCloud attribute)\n(langchain.llms.OpenAI attribute)\n(langchain.llms.OpenAIChat attribute)\n(langchain.llms.OpenLM attribute)\n(langchain.llms.Petals attribute)\n(langchain.llms.PipelineAI attribute)\n(langchain.llms.PredictionGuard attribute)\n(langchain.llms.Replicate attribute)\n(langchain.llms.RWKV attribute)\n(langchain.llms.SagemakerEndpoint attribute)\n(langchain.llms.SelfHostedHuggingFaceLLM attribute)\n(langchain.llms.SelfHostedPipeline attribute)\n(langchain.llms.StochasticAI attribute)\n(langchain.llms.VertexAI attribute)\n(langchain.llms.Writer attribute)\n(langchain.retrievers.SelfQueryRetriever attribute)\n(langchain.tools.BaseTool attribute)\n(langchain.tools.Tool attribute)\nVespaRetriever (class in langchain.retrievers)\nvideo_ids (langchain.document_loaders.GoogleApiYoutubeLoader attribute)\nvisible_only (langchain.tools.ClickTool attribute)\nvocab_only (langchain.embeddings.LlamaCppEmbeddings attribute)\n(langchain.llms.GPT4All attribute)\n(langchain.llms.LlamaCpp attribute)\nW\nwait_for_processing() (langchain.document_loaders.MathpixPDFLoader method)\nWeatherDataLoader (class in langchain.document_loaders)\nWeaviate (class in langchain.vectorstores)\nWeaviateHybridSearchRetriever (class in langchain.retrievers)\nWeaviateHybridSearchRetriever.Config (class in langchain.retrievers)", "source": "https://python.langchain.com/en/latest/genindex.html"}108{"id": "eddc1cc06a3b-94", "text": "WeaviateHybridSearchRetriever.Config (class in langchain.retrievers)\nweb_path (langchain.document_loaders.WebBaseLoader property)\nweb_paths (langchain.document_loaders.WebBaseLoader attribute)\nWebBaseLoader (class in langchain.document_loaders)\nWhatsAppChatLoader (class in langchain.document_loaders)\nWikipedia (class in langchain.docstore)\nWikipediaLoader (class in langchain.document_loaders)\nwolfram_alpha_appid (langchain.utilities.WolframAlphaAPIWrapper attribute)\nwriter_api_key (langchain.llms.Writer attribute)\nwriter_org_id (langchain.llms.Writer attribute)\nY\nYoutubeLoader (class in langchain.document_loaders)\nZ\nzapier_description (langchain.tools.ZapierNLARunAction attribute)\nZepRetriever (class in langchain.retrievers)\nZERO_SHOT_REACT_DESCRIPTION (langchain.agents.AgentType attribute)\nZilliz (class in langchain.vectorstores)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/genindex.html"}109{"id": "ca3915152821-0", "text": "Search\nError\nPlease activate JavaScript to enable the search functionality.\nCtrl+K\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/search.html"}110{"id": "3afce59ef5a1-0", "text": ".md\n.pdf\nDeployments\n Contents \nStreamlit\nGradio (on Hugging Face)\nChainlit\nBeam\nVercel\nFastAPI + Vercel\nKinsta\nFly.io\nDigitalocean App Platform\nGoogle Cloud Run\nSteamShip\nLangchain-serve\nBentoML\nDatabutton\nDeployments#\nSo, you\u2019ve created a really cool chain - now what? How do you deploy it and make it easily shareable with the world?\nThis section covers several options for that. Note that these options are meant for quick deployment of prototypes and demos, not for production systems. If you need help with the deployment of a production system, please contact us directly.\nWhat follows is a list of template GitHub repositories designed to be easily forked and modified to use your chain. This list is far from exhaustive, and we are EXTREMELY open to contributions here.\nStreamlit#\nThis repo serves as a template for how to deploy a LangChain with Streamlit.\nIt implements a chatbot interface.\nIt also contains instructions for how to deploy this app on the Streamlit platform.\nGradio (on Hugging Face)#\nThis repo serves as a template for how deploy a LangChain with Gradio.\nIt implements a chatbot interface, with a \u201cBring-Your-Own-Token\u201d approach (nice for not wracking up big bills).\nIt also contains instructions for how to deploy this app on the Hugging Face platform.\nThis is heavily influenced by James Weaver\u2019s excellent examples.\nChainlit#\nThis repo is a cookbook explaining how to visualize and deploy LangChain agents with Chainlit.\nYou create ChatGPT-like UIs with Chainlit. Some of the key features include intermediary steps visualisation, element management & display (images, text, carousel, etc.) as well as cloud deployment.\nChainlit doc on the integration with LangChain\nBeam#", "source": "https://python.langchain.com/en/latest/ecosystem/deployments.html"}111{"id": "3afce59ef5a1-1", "text": "Chainlit doc on the integration with LangChain\nBeam#\nThis repo serves as a template for how deploy a LangChain with Beam.\nIt implements a Question Answering app and contains instructions for deploying the app as a serverless REST API.\nVercel#\nA minimal example on how to run LangChain on Vercel using Flask.\nFastAPI + Vercel#\nA minimal example on how to run LangChain on Vercel using FastAPI and LangCorn/Uvicorn.\nKinsta#\nA minimal example on how to deploy LangChain to Kinsta using Flask.\nFly.io#\nA minimal example of how to deploy LangChain to Fly.io using Flask.\nDigitalocean App Platform#\nA minimal example on how to deploy LangChain to DigitalOcean App Platform.\nGoogle Cloud Run#\nA minimal example on how to deploy LangChain to Google Cloud Run.\nSteamShip#\nThis repository contains LangChain adapters for Steamship, enabling LangChain developers to rapidly deploy their apps on Steamship. This includes: production-ready endpoints, horizontal scaling across dependencies, persistent storage of app state, multi-tenancy support, etc.\nLangchain-serve#\nThis repository allows users to serve local chains and agents as RESTful, gRPC, or WebSocket APIs, thanks to Jina. Deploy your chains & agents with ease and enjoy independent scaling, serverless and autoscaling APIs, as well as a Streamlit playground on Jina AI Cloud.\nBentoML#\nThis repository provides an example of how to deploy a LangChain application with BentoML. BentoML is a framework that enables the containerization of machine learning applications as standard OCI images. BentoML also allows for the automatic generation of OpenAPI and gRPC endpoints. With BentoML, you can integrate models from all popular ML frameworks and deploy them as microservices running on the most optimal hardware and scaling independently.", "source": "https://python.langchain.com/en/latest/ecosystem/deployments.html"}112{"id": "3afce59ef5a1-2", "text": "Databutton#\nThese templates serve as examples of how to build, deploy, and share LangChain applications using Databutton. You can create user interfaces with Streamlit, automate tasks by scheduling Python code, and store files and data in the built-in store. Examples include a Chatbot interface with conversational memory, a Personal search engine, and a starter template for LangChain apps. Deploying and sharing is just one click away.\nprevious\nDependents\nnext\nTracing\n Contents\n  \nStreamlit\nGradio (on Hugging Face)\nChainlit\nBeam\nVercel\nFastAPI + Vercel\nKinsta\nFly.io\nDigitalocean App Platform\nGoogle Cloud Run\nSteamShip\nLangchain-serve\nBentoML\nDatabutton\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/ecosystem/deployments.html"}113{"id": "049a17eb27e7-0", "text": ".md\n.pdf\nLocally Hosted Setup\n Contents \nInstallation\nEnvironment Setup\nLocally Hosted Setup#\nThis page contains instructions for installing and then setting up the environment to use the locally hosted version of tracing.\nInstallation#\nEnsure you have Docker installed (see Get Docker) and that it\u2019s running.\nInstall the latest version of langchain: pip install langchain or pip install langchain -U to upgrade your\nexisting version.\nRun langchain-server. This command was installed automatically when you ran the above command (pip install langchain).\nThis will spin up the server in the terminal, hosted on port 4137 by default.\nOnce you see the terminal\noutput langchain-langchain-frontend-1 | \u279c Local: [http://localhost:4173/](http://localhost:4173/), navigate\nto http://localhost:4173/\nYou should see a page with your tracing sessions. See the overview page for a walkthrough of the UI.\nCurrently, trace data is not guaranteed to be persisted between runs of langchain-server. If you want to\npersist your data, you can mount a volume to the Docker container. See the Docker docs for more info.\nTo stop the server, press Ctrl+C in the terminal where you ran langchain-server.\nEnvironment Setup#\nAfter installation, you must now set up your environment to use tracing.\nThis can be done by setting an environment variable in your terminal by running export LANGCHAIN_HANDLER=langchain.\nYou can also do this by adding the below snippet to the top of every script. IMPORTANT: this must go at the VERY TOP of your script, before you import anything from langchain.\nimport os\nos.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\"\n Contents\n  \nInstallation\nEnvironment Setup\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/tracing/local_installation.html"}114{"id": "049a17eb27e7-1", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/tracing/local_installation.html"}115{"id": "a66c6babef30-0", "text": ".md\n.pdf\nCloud Hosted Setup\n Contents \nInstallation\nEnvironment Setup\nCloud Hosted Setup#\nWe offer a hosted version of tracing at langchainplus.vercel.app. You can use this to view traces from your run without having to run the server locally.\nNote: we are currently only offering this to a limited number of users. The hosted platform is VERY alpha, in active development, and data might be dropped at any time. Don\u2019t depend on data being persisted in the system long term and don\u2019t log traces that may contain sensitive information. If you\u2019re interested in using the hosted platform, please fill out the form here.\nInstallation#\nLogin to the system and click \u201cAPI Key\u201d in the top right corner. Generate a new key and keep it safe. You will need it to authenticate with the system.\nEnvironment Setup#\nAfter installation, you must now set up your environment to use tracing.\nThis can be done by setting an environment variable in your terminal by running export LANGCHAIN_HANDLER=langchain.\nYou can also do this by adding the below snippet to the top of every script. IMPORTANT: this must go at the VERY TOP of your script, before you import anything from langchain.\nimport os\nos.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\"\nYou will also need to set an environment variable to specify the endpoint and your API key. This can be done with the following environment variables:\nLANGCHAIN_ENDPOINT = \u201chttps://langchain-api-gateway-57eoxz8z.uc.gateway.dev\u201d\nLANGCHAIN_API_KEY - set this to the API key you generated during installation.\nAn example of adding all relevant environment variables is below:\nimport os\nos.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\"\nos.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://langchain-api-gateway-57eoxz8z.uc.gateway.dev\"", "source": "https://python.langchain.com/en/latest/tracing/hosted_installation.html"}116{"id": "a66c6babef30-1", "text": "os.environ[\"LANGCHAIN_API_KEY\"] = \"my_api_key\"  # Don't commit this to your repo! Better to set it in your terminal.\n Contents\n  \nInstallation\nEnvironment Setup\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/tracing/hosted_installation.html"}117{"id": "8fbd356c7ab5-0", "text": ".ipynb\n.pdf\nTracing Walkthrough\n Contents \n[Beta] Tracing V2\nTracing Walkthrough#\nThere are two recommended ways to trace your LangChains:\nSetting the LANGCHAIN_TRACING environment variable to \u201ctrue\u201d.\nUsing a context manager with tracing_enabled() to trace a particular block of code.\nNote if the environment variable is set, all code will be traced, regardless of whether or not it\u2019s within the context manager.\nimport os\nos.environ[\"LANGCHAIN_TRACING\"] = \"true\"\n## Uncomment below if using hosted setup.\n# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://langchain-api-gateway-57eoxz8z.uc.gateway.dev\" \n## Uncomment below if you want traces to be recorded to \"my_session\" instead of \"default\".\n# os.environ[\"LANGCHAIN_SESSION\"] = \"my_session\"  \n## Better to set this environment variable in the terminal\n## Uncomment below if using hosted version. Replace \"my_api_key\" with your actual API Key.\n# os.environ[\"LANGCHAIN_API_KEY\"] = \"my_api_key\"  \nimport langchain\nfrom langchain.agents import Tool, initialize_agent, load_tools\nfrom langchain.agents import AgentType\nfrom langchain.callbacks import tracing_enabled\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.llms import OpenAI\n# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.\nllm = OpenAI(temperature=0)\ntools = load_tools([\"llm-math\"], llm=llm)\nagent = initialize_agent(\n    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n)\nagent.run(\"What is 2 raised to .123243 power?\")\n> Entering new AgentExecutor chain...", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}118{"id": "8fbd356c7ab5-1", "text": "> Entering new AgentExecutor chain...\n I need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2^.123243\nObservation: Answer: 1.0891804557407723\nThought: I now know the final answer.\nFinal Answer: 1.0891804557407723\n> Finished chain.\n'1.0891804557407723'\n# Agent run with tracing using a chat model\nagent = initialize_agent(\n    tools, ChatOpenAI(temperature=0), agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n)\nagent.run(\"What is 2 raised to .123243 power?\")\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2 ^ .123243\nObservation: Answer: 1.0891804557407723\nThought:I now know the answer to the question. \nFinal Answer: 1.0891804557407723\n> Finished chain.\n'1.0891804557407723'\n# Both of the agent runs will be traced because the environment variable is set\nagent.run(\"What is 2 raised to .123243 power?\")\nwith tracing_enabled() as session:\n    agent.run(\"What is 5 raised to .123243 power?\")\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2 ^ .123243\nObservation: Answer: 1.0891804557407723\nThought:I now know the answer to the question. \nFinal Answer: 1.0891804557407723\n> Finished chain.\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}119{"id": "8fbd356c7ab5-2", "text": "I need to use a calculator to solve this.\nAction: Calculator\nAction Input: 5 ^ .123243\nObservation: Answer: 1.2193914912400514\nThought:I now know the answer to the question. \nFinal Answer: 1.2193914912400514\n> Finished chain.\n# Now, we unset the environment variable and use a context manager.\nif \"LANGCHAIN_TRACING\" in os.environ:\n    del os.environ[\"LANGCHAIN_TRACING\"]\n# here, we are writing traces to \"my_test_session\"\nwith tracing_enabled(\"my_session\") as session:\n    assert session\n    agent.run(\"What is 5 raised to .123243 power?\")  # this should be traced\nagent.run(\"What is 2 raised to .123243 power?\")  # this should not be traced\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator\nAction Input: 5 ^ .123243\nObservation: Answer: 1.2193914912400514\nThought:I now know the answer to the question. \nFinal Answer: 1.2193914912400514\n> Finished chain.\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2 ^ .123243\nObservation: Answer: 1.0891804557407723\nThought:I now know the answer to the question. \nFinal Answer: 1.0891804557407723\n> Finished chain.\n'1.0891804557407723'\n# The context manager is concurrency safe:\nimport asyncio \nif \"LANGCHAIN_TRACING\" in os.environ:\n    del os.environ[\"LANGCHAIN_TRACING\"]", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}120{"id": "8fbd356c7ab5-3", "text": "del os.environ[\"LANGCHAIN_TRACING\"]\n    \nquestions = [f\"What is {i} raised to .123 power?\" for i in range(1,4)]\n# start a background task\ntask = asyncio.create_task(agent.arun(questions[0]))  # this should not be traced\nwith tracing_enabled() as session:\n    assert session\n    tasks = [agent.arun(q) for q in questions[1:3]]  # these should be traced\n    await asyncio.gather(*tasks)\nawait task\n> Entering new AgentExecutor chain...\n> Entering new AgentExecutor chain...\n> Entering new AgentExecutor chain...\nI need to use a calculator to solve this.\nAction: Calculator\nAction Input: 3^0.123I need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2^0.123Any number raised to the power of 0 is 1, but I'm not sure about a decimal power.\nAction: Calculator\nAction Input: 1^.123\nObservation: Answer: 1.1446847956963533\nThought:\nObservation: Answer: 1.0889970153361064\nThought:\nObservation: Answer: 1.0\nThought:\n> Finished chain.\n> Finished chain.\n> Finished chain.\n'1.0'\n[Beta] Tracing V2#\nWe are rolling out a newer version of our tracing service with more features coming soon. Here are the instructions on how to use it to trace your runs.\nTo use, you can use the tracing_v2_enabled context manager or set LANGCHAIN_TRACING_V2 = 'true'\nOption 1 (Local):\nRun the local LangChainPlus Server\npip install --upgrade langchain\nlangchain plus start\nOption 2 (Hosted):", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}121{"id": "8fbd356c7ab5-4", "text": "pip install --upgrade langchain\nlangchain plus start\nOption 2 (Hosted):\nAfter making an account an grabbing a LangChainPlus API Key, set the LANGCHAIN_ENDPOINT and LANGCHAIN_API_KEY environment variables\nimport os\nos.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://langchainpro-api-gateway-12bfv6cf.uc.gateway.dev\"  # Uncomment this line if you want to use the hosted version\n# os.environ[\"LANGCHAIN_API_KEY\"] = \"<YOUR-LANGCHAINPLUS-API-KEY>\"  # Uncomment this line if you want to use the hosted version.\nimport langchain\nfrom langchain.agents import Tool, initialize_agent, load_tools\nfrom langchain.agents import AgentType\nfrom langchain.callbacks import tracing_enabled\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.llms import OpenAI\n# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.\nllm = OpenAI(temperature=0)\ntools = load_tools([\"llm-math\"], llm=llm)\nagent = initialize_agent(\n    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n)\nagent.run(\"What is 2 raised to .123243 power?\")\n> Entering new AgentExecutor chain...\n I need to use a calculator to solve this.\nAction: Calculator\nAction Input: 2^.123243\nObservation: Answer: 1.0891804557407723\nThought: I now know the final answer.\nFinal Answer: 1.0891804557407723\n> Finished chain.\n'1.0891804557407723'\n Contents\n  \n[Beta] Tracing V2\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}122{"id": "8fbd356c7ab5-5", "text": "Contents\n  \n[Beta] Tracing V2\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/tracing/agent_with_tracing.html"}123{"id": "9f13a6ebc64c-0", "text": "Source code for langchain.text_splitter\n\"\"\"Functionality for splitting text.\"\"\"\nfrom __future__ import annotations\nimport copy\nimport logging\nfrom abc import ABC, abstractmethod\nfrom typing import (\n    AbstractSet,\n    Any,\n    Callable,\n    Collection,\n    Iterable,\n    List,\n    Literal,\n    Optional,\n    Sequence,\n    Type,\n    TypeVar,\n    Union,\n)\nfrom langchain.docstore.document import Document\nfrom langchain.schema import BaseDocumentTransformer\nlogger = logging.getLogger(__name__)\nTS = TypeVar(\"TS\", bound=\"TextSplitter\")\n[docs]class TextSplitter(BaseDocumentTransformer, ABC):\n    \"\"\"Interface for splitting text into chunks.\"\"\"\n    def __init__(\n        self,\n        chunk_size: int = 4000,\n        chunk_overlap: int = 200,\n        length_function: Callable[[str], int] = len,\n    ):\n        \"\"\"Create a new TextSplitter.\"\"\"\n        if chunk_overlap > chunk_size:\n            raise ValueError(\n                f\"Got a larger chunk overlap ({chunk_overlap}) than chunk size \"\n                f\"({chunk_size}), should be smaller.\"\n            )\n        self._chunk_size = chunk_size\n        self._chunk_overlap = chunk_overlap\n        self._length_function = length_function\n[docs]    @abstractmethod\n    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split text into multiple components.\"\"\"\n[docs]    def create_documents(\n        self, texts: List[str], metadatas: Optional[List[dict]] = None\n    ) -> List[Document]:\n        \"\"\"Create documents from a list of texts.\"\"\"\n        _metadatas = metadatas or [{}] * len(texts)\n        documents = []", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}124{"id": "9f13a6ebc64c-1", "text": "documents = []\n        for i, text in enumerate(texts):\n            for chunk in self.split_text(text):\n                new_doc = Document(\n                    page_content=chunk, metadata=copy.deepcopy(_metadatas[i])\n                )\n                documents.append(new_doc)\n        return documents\n[docs]    def split_documents(self, documents: Iterable[Document]) -> List[Document]:\n        \"\"\"Split documents.\"\"\"\n        texts, metadatas = [], []\n        for doc in documents:\n            texts.append(doc.page_content)\n            metadatas.append(doc.metadata)\n        return self.create_documents(texts, metadatas=metadatas)\n    def _join_docs(self, docs: List[str], separator: str) -> Optional[str]:\n        text = separator.join(docs)\n        text = text.strip()\n        if text == \"\":\n            return None\n        else:\n            return text\n    def _merge_splits(self, splits: Iterable[str], separator: str) -> List[str]:\n        # We now want to combine these smaller pieces into medium size\n        # chunks to send to the LLM.\n        separator_len = self._length_function(separator)\n        docs = []\n        current_doc: List[str] = []\n        total = 0\n        for d in splits:\n            _len = self._length_function(d)\n            if (\n                total + _len + (separator_len if len(current_doc) > 0 else 0)\n                > self._chunk_size\n            ):\n                if total > self._chunk_size:\n                    logger.warning(\n                        f\"Created a chunk of size {total}, \"\n                        f\"which is longer than the specified {self._chunk_size}\"\n                    )\n                if len(current_doc) > 0:\n                    doc = self._join_docs(current_doc, separator)", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}125{"id": "9f13a6ebc64c-2", "text": "doc = self._join_docs(current_doc, separator)\n                    if doc is not None:\n                        docs.append(doc)\n                    # Keep on popping if:\n                    # - we have a larger chunk than in the chunk overlap\n                    # - or if we still have any chunks and the length is long\n                    while total > self._chunk_overlap or (\n                        total + _len + (separator_len if len(current_doc) > 0 else 0)\n                        > self._chunk_size\n                        and total > 0\n                    ):\n                        total -= self._length_function(current_doc[0]) + (\n                            separator_len if len(current_doc) > 1 else 0\n                        )\n                        current_doc = current_doc[1:]\n            current_doc.append(d)\n            total += _len + (separator_len if len(current_doc) > 1 else 0)\n        doc = self._join_docs(current_doc, separator)\n        if doc is not None:\n            docs.append(doc)\n        return docs\n[docs]    @classmethod\n    def from_huggingface_tokenizer(cls, tokenizer: Any, **kwargs: Any) -> TextSplitter:\n        \"\"\"Text splitter that uses HuggingFace tokenizer to count length.\"\"\"\n        try:\n            from transformers import PreTrainedTokenizerBase\n            if not isinstance(tokenizer, PreTrainedTokenizerBase):\n                raise ValueError(\n                    \"Tokenizer received was not an instance of PreTrainedTokenizerBase\"\n                )\n            def _huggingface_tokenizer_length(text: str) -> int:\n                return len(tokenizer.encode(text))\n        except ImportError:\n            raise ValueError(\n                \"Could not import transformers python package. \"\n                \"Please install it with `pip install transformers`.\"\n            )\n        return cls(length_function=_huggingface_tokenizer_length, **kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}126{"id": "9f13a6ebc64c-3", "text": ")\n        return cls(length_function=_huggingface_tokenizer_length, **kwargs)\n[docs]    @classmethod\n    def from_tiktoken_encoder(\n        cls: Type[TS],\n        encoding_name: str = \"gpt2\",\n        model_name: Optional[str] = None,\n        allowed_special: Union[Literal[\"all\"], AbstractSet[str]] = set(),\n        disallowed_special: Union[Literal[\"all\"], Collection[str]] = \"all\",\n        **kwargs: Any,\n    ) -> TS:\n        \"\"\"Text splitter that uses tiktoken encoder to count length.\"\"\"\n        try:\n            import tiktoken\n        except ImportError:\n            raise ImportError(\n                \"Could not import tiktoken python package. \"\n                \"This is needed in order to calculate max_tokens_for_prompt. \"\n                \"Please install it with `pip install tiktoken`.\"\n            )\n        if model_name is not None:\n            enc = tiktoken.encoding_for_model(model_name)\n        else:\n            enc = tiktoken.get_encoding(encoding_name)\n        def _tiktoken_encoder(text: str) -> int:\n            return len(\n                enc.encode(\n                    text,\n                    allowed_special=allowed_special,\n                    disallowed_special=disallowed_special,\n                )\n            )\n        if issubclass(cls, TokenTextSplitter):\n            extra_kwargs = {\n                \"encoding_name\": encoding_name,\n                \"model_name\": model_name,\n                \"allowed_special\": allowed_special,\n                \"disallowed_special\": disallowed_special,\n            }\n            kwargs = {**kwargs, **extra_kwargs}\n        return cls(length_function=_tiktoken_encoder, **kwargs)\n[docs]    def transform_documents(\n        self, documents: Sequence[Document], **kwargs: Any\n    ) -> Sequence[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}127{"id": "9f13a6ebc64c-4", "text": ") -> Sequence[Document]:\n        \"\"\"Transform sequence of documents by splitting them.\"\"\"\n        return self.split_documents(list(documents))\n[docs]    async def atransform_documents(\n        self, documents: Sequence[Document], **kwargs: Any\n    ) -> Sequence[Document]:\n        \"\"\"Asynchronously transform a sequence of documents by splitting them.\"\"\"\n        raise NotImplementedError\n[docs]class CharacterTextSplitter(TextSplitter):\n    \"\"\"Implementation of splitting text that looks at characters.\"\"\"\n    def __init__(self, separator: str = \"\\n\\n\", **kwargs: Any):\n        \"\"\"Create a new TextSplitter.\"\"\"\n        super().__init__(**kwargs)\n        self._separator = separator\n[docs]    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split incoming text and return chunks.\"\"\"\n        # First we naively split the large input into a bunch of smaller ones.\n        if self._separator:\n            splits = text.split(self._separator)\n        else:\n            splits = list(text)\n        return self._merge_splits(splits, self._separator)\n[docs]class TokenTextSplitter(TextSplitter):\n    \"\"\"Implementation of splitting text that looks at tokens.\"\"\"\n    def __init__(\n        self,\n        encoding_name: str = \"gpt2\",\n        model_name: Optional[str] = None,\n        allowed_special: Union[Literal[\"all\"], AbstractSet[str]] = set(),\n        disallowed_special: Union[Literal[\"all\"], Collection[str]] = \"all\",\n        **kwargs: Any,\n    ):\n        \"\"\"Create a new TextSplitter.\"\"\"\n        super().__init__(**kwargs)\n        try:\n            import tiktoken\n        except ImportError:\n            raise ImportError(\n                \"Could not import tiktoken python package. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}128{"id": "9f13a6ebc64c-5", "text": "raise ImportError(\n                \"Could not import tiktoken python package. \"\n                \"This is needed in order to for TokenTextSplitter. \"\n                \"Please install it with `pip install tiktoken`.\"\n            )\n        if model_name is not None:\n            enc = tiktoken.encoding_for_model(model_name)\n        else:\n            enc = tiktoken.get_encoding(encoding_name)\n        self._tokenizer = enc\n        self._allowed_special = allowed_special\n        self._disallowed_special = disallowed_special\n[docs]    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split incoming text and return chunks.\"\"\"\n        splits = []\n        input_ids = self._tokenizer.encode(\n            text,\n            allowed_special=self._allowed_special,\n            disallowed_special=self._disallowed_special,\n        )\n        start_idx = 0\n        cur_idx = min(start_idx + self._chunk_size, len(input_ids))\n        chunk_ids = input_ids[start_idx:cur_idx]\n        while start_idx < len(input_ids):\n            splits.append(self._tokenizer.decode(chunk_ids))\n            start_idx += self._chunk_size - self._chunk_overlap\n            cur_idx = min(start_idx + self._chunk_size, len(input_ids))\n            chunk_ids = input_ids[start_idx:cur_idx]\n        return splits\n[docs]class RecursiveCharacterTextSplitter(TextSplitter):\n    \"\"\"Implementation of splitting text that looks at characters.\n    Recursively tries to split by different characters to find one\n    that works.\n    \"\"\"\n    def __init__(self, separators: Optional[List[str]] = None, **kwargs: Any):\n        \"\"\"Create a new TextSplitter.\"\"\"\n        super().__init__(**kwargs)\n        self._separators = separators or [\"\\n\\n\", \"\\n\", \" \", \"\"]", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}129{"id": "9f13a6ebc64c-6", "text": "[docs]    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split incoming text and return chunks.\"\"\"\n        final_chunks = []\n        # Get appropriate separator to use\n        separator = self._separators[-1]\n        for _s in self._separators:\n            if _s == \"\":\n                separator = _s\n                break\n            if _s in text:\n                separator = _s\n                break\n        # Now that we have the separator, split the text\n        if separator:\n            splits = text.split(separator)\n        else:\n            splits = list(text)\n        # Now go merging things, recursively splitting longer texts.\n        _good_splits = []\n        for s in splits:\n            if self._length_function(s) < self._chunk_size:\n                _good_splits.append(s)\n            else:\n                if _good_splits:\n                    merged_text = self._merge_splits(_good_splits, separator)\n                    final_chunks.extend(merged_text)\n                    _good_splits = []\n                other_info = self.split_text(s)\n                final_chunks.extend(other_info)\n        if _good_splits:\n            merged_text = self._merge_splits(_good_splits, separator)\n            final_chunks.extend(merged_text)\n        return final_chunks\n[docs]class NLTKTextSplitter(TextSplitter):\n    \"\"\"Implementation of splitting text that looks at sentences using NLTK.\"\"\"\n    def __init__(self, separator: str = \"\\n\\n\", **kwargs: Any):\n        \"\"\"Initialize the NLTK splitter.\"\"\"\n        super().__init__(**kwargs)\n        try:\n            from nltk.tokenize import sent_tokenize\n            self._tokenizer = sent_tokenize\n        except ImportError:\n            raise ImportError(\n                \"NLTK is not installed, please install it with `pip install nltk`.\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}130{"id": "9f13a6ebc64c-7", "text": "\"NLTK is not installed, please install it with `pip install nltk`.\"\n            )\n        self._separator = separator\n[docs]    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split incoming text and return chunks.\"\"\"\n        # First we naively split the large input into a bunch of smaller ones.\n        splits = self._tokenizer(text)\n        return self._merge_splits(splits, self._separator)\n[docs]class SpacyTextSplitter(TextSplitter):\n    \"\"\"Implementation of splitting text that looks at sentences using Spacy.\"\"\"\n    def __init__(\n        self, separator: str = \"\\n\\n\", pipeline: str = \"en_core_web_sm\", **kwargs: Any\n    ):\n        \"\"\"Initialize the spacy text splitter.\"\"\"\n        super().__init__(**kwargs)\n        try:\n            import spacy\n        except ImportError:\n            raise ImportError(\n                \"Spacy is not installed, please install it with `pip install spacy`.\"\n            )\n        self._tokenizer = spacy.load(pipeline)\n        self._separator = separator\n[docs]    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split incoming text and return chunks.\"\"\"\n        splits = (str(s) for s in self._tokenizer(text).sents)\n        return self._merge_splits(splits, self._separator)\n[docs]class MarkdownTextSplitter(RecursiveCharacterTextSplitter):\n    \"\"\"Attempts to split the text along Markdown-formatted headings.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize a MarkdownTextSplitter.\"\"\"\n        separators = [\n            # First, try to split along Markdown headings (starting with level 2)\n            \"\\n## \",\n            \"\\n### \",\n            \"\\n#### \",", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}131{"id": "9f13a6ebc64c-8", "text": "\"\\n## \",\n            \"\\n### \",\n            \"\\n#### \",\n            \"\\n##### \",\n            \"\\n###### \",\n            # Note the alternative syntax for headings (below) is not handled here\n            # Heading level 2\n            # ---------------\n            # End of code block\n            \"```\\n\\n\",\n            # Horizontal lines\n            \"\\n\\n***\\n\\n\",\n            \"\\n\\n---\\n\\n\",\n            \"\\n\\n___\\n\\n\",\n            # Note that this splitter doesn't handle horizontal lines defined\n            # by *three or more* of ***, ---, or ___, but this is not handled\n            \"\\n\\n\",\n            \"\\n\",\n            \" \",\n            \"\",\n        ]\n        super().__init__(separators=separators, **kwargs)\n[docs]class LatexTextSplitter(RecursiveCharacterTextSplitter):\n    \"\"\"Attempts to split the text along Latex-formatted layout elements.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize a LatexTextSplitter.\"\"\"\n        separators = [\n            # First, try to split along Latex sections\n            \"\\n\\\\chapter{\",\n            \"\\n\\\\section{\",\n            \"\\n\\\\subsection{\",\n            \"\\n\\\\subsubsection{\",\n            # Now split by environments\n            \"\\n\\\\begin{enumerate}\",\n            \"\\n\\\\begin{itemize}\",\n            \"\\n\\\\begin{description}\",\n            \"\\n\\\\begin{list}\",\n            \"\\n\\\\begin{quote}\",\n            \"\\n\\\\begin{quotation}\",\n            \"\\n\\\\begin{verse}\",\n            \"\\n\\\\begin{verbatim}\",\n            ## Now split by math environments\n            \"\\n\\\\begin{align}\",\n            \"$$\",\n            \"$\",", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}132{"id": "9f13a6ebc64c-9", "text": "\"\\n\\\\begin{align}\",\n            \"$$\",\n            \"$\",\n            # Now split by the normal type of lines\n            \" \",\n            \"\",\n        ]\n        super().__init__(separators=separators, **kwargs)\n[docs]class PythonCodeTextSplitter(RecursiveCharacterTextSplitter):\n    \"\"\"Attempts to split the text along Python syntax.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize a PythonCodeTextSplitter.\"\"\"\n        separators = [\n            # First, try to split along class definitions\n            \"\\nclass \",\n            \"\\ndef \",\n            \"\\n\\tdef \",\n            # Now split by the normal type of lines\n            \"\\n\\n\",\n            \"\\n\",\n            \" \",\n            \"\",\n        ]\n        super().__init__(separators=separators, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/text_splitter.html"}133{"id": "1a2b191e7bd9-0", "text": "Source code for langchain.requests\n\"\"\"Lightweight wrapper around requests library, with async support.\"\"\"\nfrom contextlib import asynccontextmanager\nfrom typing import Any, AsyncGenerator, Dict, Optional\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel, Extra\nclass Requests(BaseModel):\n    \"\"\"Wrapper around requests to handle auth and async.\n    The main purpose of this wrapper is to handle authentication (by saving\n    headers) and enable easy async methods on the same base object.\n    \"\"\"\n    headers: Optional[Dict[str, str]] = None\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    def get(self, url: str, **kwargs: Any) -> requests.Response:\n        \"\"\"GET the URL and return the text.\"\"\"\n        return requests.get(url, headers=self.headers, **kwargs)\n    def post(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:\n        \"\"\"POST to the URL and return the text.\"\"\"\n        return requests.post(url, json=data, headers=self.headers, **kwargs)\n    def patch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:\n        \"\"\"PATCH the URL and return the text.\"\"\"\n        return requests.patch(url, json=data, headers=self.headers, **kwargs)\n    def put(self, url: str, data: Dict[str, Any], **kwargs: Any) -> requests.Response:\n        \"\"\"PUT the URL and return the text.\"\"\"\n        return requests.put(url, json=data, headers=self.headers, **kwargs)\n    def delete(self, url: str, **kwargs: Any) -> requests.Response:", "source": "https://python.langchain.com/en/latest/_modules/langchain/requests.html"}134{"id": "1a2b191e7bd9-1", "text": "def delete(self, url: str, **kwargs: Any) -> requests.Response:\n        \"\"\"DELETE the URL and return the text.\"\"\"\n        return requests.delete(url, headers=self.headers, **kwargs)\n    @asynccontextmanager\n    async def _arequest(\n        self, method: str, url: str, **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"Make an async request.\"\"\"\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.request(\n                    method, url, headers=self.headers, **kwargs\n                ) as response:\n                    yield response\n        else:\n            async with self.aiosession.request(\n                method, url, headers=self.headers, **kwargs\n            ) as response:\n                yield response\n    @asynccontextmanager\n    async def aget(\n        self, url: str, **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"GET the URL and return the text asynchronously.\"\"\"\n        async with self._arequest(\"GET\", url, **kwargs) as response:\n            yield response\n    @asynccontextmanager\n    async def apost(\n        self, url: str, data: Dict[str, Any], **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"POST to the URL and return the text asynchronously.\"\"\"\n        async with self._arequest(\"POST\", url, **kwargs) as response:\n            yield response\n    @asynccontextmanager\n    async def apatch(\n        self, url: str, data: Dict[str, Any], **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"PATCH the URL and return the text asynchronously.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/requests.html"}135{"id": "1a2b191e7bd9-2", "text": "\"\"\"PATCH the URL and return the text asynchronously.\"\"\"\n        async with self._arequest(\"PATCH\", url, **kwargs) as response:\n            yield response\n    @asynccontextmanager\n    async def aput(\n        self, url: str, data: Dict[str, Any], **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"PUT the URL and return the text asynchronously.\"\"\"\n        async with self._arequest(\"PUT\", url, **kwargs) as response:\n            yield response\n    @asynccontextmanager\n    async def adelete(\n        self, url: str, **kwargs: Any\n    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:\n        \"\"\"DELETE the URL and return the text asynchronously.\"\"\"\n        async with self._arequest(\"DELETE\", url, **kwargs) as response:\n            yield response\n[docs]class TextRequestsWrapper(BaseModel):\n    \"\"\"Lightweight wrapper around requests library.\n    The main purpose of this wrapper is to always return a text output.\n    \"\"\"\n    headers: Optional[Dict[str, str]] = None\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @property\n    def requests(self) -> Requests:\n        return Requests(headers=self.headers, aiosession=self.aiosession)\n[docs]    def get(self, url: str, **kwargs: Any) -> str:\n        \"\"\"GET the URL and return the text.\"\"\"\n        return self.requests.get(url, **kwargs).text\n[docs]    def post(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:\n        \"\"\"POST to the URL and return the text.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/requests.html"}136{"id": "1a2b191e7bd9-3", "text": "\"\"\"POST to the URL and return the text.\"\"\"\n        return self.requests.post(url, data, **kwargs).text\n[docs]    def patch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:\n        \"\"\"PATCH the URL and return the text.\"\"\"\n        return self.requests.patch(url, data, **kwargs).text\n[docs]    def put(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:\n        \"\"\"PUT the URL and return the text.\"\"\"\n        return self.requests.put(url, data, **kwargs).text\n[docs]    def delete(self, url: str, **kwargs: Any) -> str:\n        \"\"\"DELETE the URL and return the text.\"\"\"\n        return self.requests.delete(url, **kwargs).text\n[docs]    async def aget(self, url: str, **kwargs: Any) -> str:\n        \"\"\"GET the URL and return the text asynchronously.\"\"\"\n        async with self.requests.aget(url, **kwargs) as response:\n            return await response.text()\n[docs]    async def apost(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:\n        \"\"\"POST to the URL and return the text asynchronously.\"\"\"\n        async with self.requests.apost(url, **kwargs) as response:\n            return await response.text()\n[docs]    async def apatch(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:\n        \"\"\"PATCH the URL and return the text asynchronously.\"\"\"\n        async with self.requests.apatch(url, **kwargs) as response:\n            return await response.text()\n[docs]    async def aput(self, url: str, data: Dict[str, Any], **kwargs: Any) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/requests.html"}137{"id": "1a2b191e7bd9-4", "text": "\"\"\"PUT the URL and return the text asynchronously.\"\"\"\n        async with self.requests.aput(url, **kwargs) as response:\n            return await response.text()\n[docs]    async def adelete(self, url: str, **kwargs: Any) -> str:\n        \"\"\"DELETE the URL and return the text asynchronously.\"\"\"\n        async with self.requests.adelete(url, **kwargs) as response:\n            return await response.text()\n# For backwards compatibility\nRequestsWrapper = TextRequestsWrapper\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/requests.html"}138{"id": "afb575afd7f4-0", "text": "Source code for langchain.document_transformers\n\"\"\"Transform documents\"\"\"\nfrom typing import Any, Callable, List, Sequence\nimport numpy as np\nfrom pydantic import BaseModel, Field\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.math_utils import cosine_similarity\nfrom langchain.schema import BaseDocumentTransformer, Document\nclass _DocumentWithState(Document):\n    \"\"\"Wrapper for a document that includes arbitrary state.\"\"\"\n    state: dict = Field(default_factory=dict)\n    \"\"\"State associated with the document.\"\"\"\n    def to_document(self) -> Document:\n        \"\"\"Convert the DocumentWithState to a Document.\"\"\"\n        return Document(page_content=self.page_content, metadata=self.metadata)\n    @classmethod\n    def from_document(cls, doc: Document) -> \"_DocumentWithState\":\n        \"\"\"Create a DocumentWithState from a Document.\"\"\"\n        if isinstance(doc, cls):\n            return doc\n        return cls(page_content=doc.page_content, metadata=doc.metadata)\n[docs]def get_stateful_documents(\n    documents: Sequence[Document],\n) -> Sequence[_DocumentWithState]:\n    return [_DocumentWithState.from_document(doc) for doc in documents]\ndef _filter_similar_embeddings(\n    embedded_documents: List[List[float]], similarity_fn: Callable, threshold: float\n) -> List[int]:\n    \"\"\"Filter redundant documents based on the similarity of their embeddings.\"\"\"\n    similarity = np.tril(similarity_fn(embedded_documents, embedded_documents), k=-1)\n    redundant = np.where(similarity > threshold)\n    redundant_stacked = np.column_stack(redundant)\n    redundant_sorted = np.argsort(similarity[redundant])[::-1]\n    included_idxs = set(range(len(embedded_documents)))\n    for first_idx, second_idx in redundant_stacked[redundant_sorted]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_transformers.html"}139{"id": "afb575afd7f4-1", "text": "for first_idx, second_idx in redundant_stacked[redundant_sorted]:\n        if first_idx in included_idxs and second_idx in included_idxs:\n            # Default to dropping the second document of any highly similar pair.\n            included_idxs.remove(second_idx)\n    return list(sorted(included_idxs))\ndef _get_embeddings_from_stateful_docs(\n    embeddings: Embeddings, documents: Sequence[_DocumentWithState]\n) -> List[List[float]]:\n    if len(documents) and \"embedded_doc\" in documents[0].state:\n        embedded_documents = [doc.state[\"embedded_doc\"] for doc in documents]\n    else:\n        embedded_documents = embeddings.embed_documents(\n            [d.page_content for d in documents]\n        )\n        for doc, embedding in zip(documents, embedded_documents):\n            doc.state[\"embedded_doc\"] = embedding\n    return embedded_documents\n[docs]class EmbeddingsRedundantFilter(BaseDocumentTransformer, BaseModel):\n    \"\"\"Filter that drops redundant documents by comparing their embeddings.\"\"\"\n    embeddings: Embeddings\n    \"\"\"Embeddings to use for embedding document contents.\"\"\"\n    similarity_fn: Callable = cosine_similarity\n    \"\"\"Similarity function for comparing documents. Function expected to take as input\n    two matrices (List[List[float]]) and return a matrix of scores where higher values\n    indicate greater similarity.\"\"\"\n    similarity_threshold: float = 0.95\n    \"\"\"Threshold for determining when two documents are similar enough\n    to be considered redundant.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def transform_documents(\n        self, documents: Sequence[Document], **kwargs: Any\n    ) -> Sequence[Document]:\n        \"\"\"Filter down documents.\"\"\"\n        stateful_documents = get_stateful_documents(documents)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_transformers.html"}140{"id": "afb575afd7f4-2", "text": "\"\"\"Filter down documents.\"\"\"\n        stateful_documents = get_stateful_documents(documents)\n        embedded_documents = _get_embeddings_from_stateful_docs(\n            self.embeddings, stateful_documents\n        )\n        included_idxs = _filter_similar_embeddings(\n            embedded_documents, self.similarity_fn, self.similarity_threshold\n        )\n        return [stateful_documents[i] for i in sorted(included_idxs)]\n[docs]    async def atransform_documents(\n        self, documents: Sequence[Document], **kwargs: Any\n    ) -> Sequence[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_transformers.html"}141{"id": "53569bb9289f-0", "text": "Source code for langchain.experimental.autonomous_agents.baby_agi.baby_agi\n\"\"\"BabyAGI agent.\"\"\"\nfrom collections import deque\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Field\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.manager import CallbackManagerForChainRun\nfrom langchain.chains.base import Chain\nfrom langchain.experimental.autonomous_agents.baby_agi.task_creation import (\n    TaskCreationChain,\n)\nfrom langchain.experimental.autonomous_agents.baby_agi.task_execution import (\n    TaskExecutionChain,\n)\nfrom langchain.experimental.autonomous_agents.baby_agi.task_prioritization import (\n    TaskPrioritizationChain,\n)\nfrom langchain.vectorstores.base import VectorStore\n[docs]class BabyAGI(Chain, BaseModel):\n    \"\"\"Controller model for the BabyAGI agent.\"\"\"\n    task_list: deque = Field(default_factory=deque)\n    task_creation_chain: Chain = Field(...)\n    task_prioritization_chain: Chain = Field(...)\n    execution_chain: Chain = Field(...)\n    task_id_counter: int = Field(1)\n    vectorstore: VectorStore = Field(init=False)\n    max_iterations: Optional[int] = None\n[docs]    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    def add_task(self, task: Dict) -> None:\n        self.task_list.append(task)\n    def print_task_list(self) -> None:\n        print(\"\\033[95m\\033[1m\" + \"\\n*****TASK LIST*****\\n\" + \"\\033[0m\\033[0m\")\n        for t in self.task_list:\n            print(str(t[\"task_id\"]) + \": \" + t[\"task_name\"])", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/baby_agi/baby_agi.html"}142{"id": "53569bb9289f-1", "text": "print(str(t[\"task_id\"]) + \": \" + t[\"task_name\"])\n    def print_next_task(self, task: Dict) -> None:\n        print(\"\\033[92m\\033[1m\" + \"\\n*****NEXT TASK*****\\n\" + \"\\033[0m\\033[0m\")\n        print(str(task[\"task_id\"]) + \": \" + task[\"task_name\"])\n    def print_task_result(self, result: str) -> None:\n        print(\"\\033[93m\\033[1m\" + \"\\n*****TASK RESULT*****\\n\" + \"\\033[0m\\033[0m\")\n        print(result)\n    @property\n    def input_keys(self) -> List[str]:\n        return [\"objective\"]\n    @property\n    def output_keys(self) -> List[str]:\n        return []\n[docs]    def get_next_task(\n        self, result: str, task_description: str, objective: str\n    ) -> List[Dict]:\n        \"\"\"Get the next task.\"\"\"\n        task_names = [t[\"task_name\"] for t in self.task_list]\n        incomplete_tasks = \", \".join(task_names)\n        response = self.task_creation_chain.run(\n            result=result,\n            task_description=task_description,\n            incomplete_tasks=incomplete_tasks,\n            objective=objective,\n        )\n        new_tasks = response.split(\"\\n\")\n        return [\n            {\"task_name\": task_name} for task_name in new_tasks if task_name.strip()\n        ]\n[docs]    def prioritize_tasks(self, this_task_id: int, objective: str) -> List[Dict]:\n        \"\"\"Prioritize tasks.\"\"\"\n        task_names = [t[\"task_name\"] for t in list(self.task_list)]\n        next_task_id = int(this_task_id) + 1", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/baby_agi/baby_agi.html"}143{"id": "53569bb9289f-2", "text": "next_task_id = int(this_task_id) + 1\n        response = self.task_prioritization_chain.run(\n            task_names=\", \".join(task_names),\n            next_task_id=str(next_task_id),\n            objective=objective,\n        )\n        new_tasks = response.split(\"\\n\")\n        prioritized_task_list = []\n        for task_string in new_tasks:\n            if not task_string.strip():\n                continue\n            task_parts = task_string.strip().split(\".\", 1)\n            if len(task_parts) == 2:\n                task_id = task_parts[0].strip()\n                task_name = task_parts[1].strip()\n                prioritized_task_list.append(\n                    {\"task_id\": task_id, \"task_name\": task_name}\n                )\n        return prioritized_task_list\n    def _get_top_tasks(self, query: str, k: int) -> List[str]:\n        \"\"\"Get the top k tasks based on the query.\"\"\"\n        results = self.vectorstore.similarity_search(query, k=k)\n        if not results:\n            return []\n        return [str(item.metadata[\"task\"]) for item in results]\n[docs]    def execute_task(self, objective: str, task: str, k: int = 5) -> str:\n        \"\"\"Execute a task.\"\"\"\n        context = self._get_top_tasks(query=objective, k=k)\n        return self.execution_chain.run(\n            objective=objective, context=\"\\n\".join(context), task=task\n        )\n    def _call(\n        self,\n        inputs: Dict[str, Any],\n        run_manager: Optional[CallbackManagerForChainRun] = None,\n    ) -> Dict[str, Any]:\n        \"\"\"Run the agent.\"\"\"\n        objective = inputs[\"objective\"]", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/baby_agi/baby_agi.html"}144{"id": "53569bb9289f-3", "text": "\"\"\"Run the agent.\"\"\"\n        objective = inputs[\"objective\"]\n        first_task = inputs.get(\"first_task\", \"Make a todo list\")\n        self.add_task({\"task_id\": 1, \"task_name\": first_task})\n        num_iters = 0\n        while True:\n            if self.task_list:\n                self.print_task_list()\n                # Step 1: Pull the first task\n                task = self.task_list.popleft()\n                self.print_next_task(task)\n                # Step 2: Execute the task\n                result = self.execute_task(objective, task[\"task_name\"])\n                this_task_id = int(task[\"task_id\"])\n                self.print_task_result(result)\n                # Step 3: Store the result in Pinecone\n                result_id = f\"result_{task['task_id']}\"\n                self.vectorstore.add_texts(\n                    texts=[result],\n                    metadatas=[{\"task\": task[\"task_name\"]}],\n                    ids=[result_id],\n                )\n                # Step 4: Create new tasks and reprioritize task list\n                new_tasks = self.get_next_task(result, task[\"task_name\"], objective)\n                for new_task in new_tasks:\n                    self.task_id_counter += 1\n                    new_task.update({\"task_id\": self.task_id_counter})\n                    self.add_task(new_task)\n                self.task_list = deque(self.prioritize_tasks(this_task_id, objective))\n            num_iters += 1\n            if self.max_iterations is not None and num_iters == self.max_iterations:\n                print(\n                    \"\\033[91m\\033[1m\" + \"\\n*****TASK ENDING*****\\n\" + \"\\033[0m\\033[0m\"\n                )\n                break\n        return {}\n[docs]    @classmethod\n    def from_llm(", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/baby_agi/baby_agi.html"}145{"id": "53569bb9289f-4", "text": "break\n        return {}\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        vectorstore: VectorStore,\n        verbose: bool = False,\n        task_execution_chain: Optional[Chain] = None,\n        **kwargs: Dict[str, Any],\n    ) -> \"BabyAGI\":\n        \"\"\"Initialize the BabyAGI Controller.\"\"\"\n        task_creation_chain = TaskCreationChain.from_llm(llm, verbose=verbose)\n        task_prioritization_chain = TaskPrioritizationChain.from_llm(\n            llm, verbose=verbose\n        )\n        if task_execution_chain is None:\n            execution_chain: Chain = TaskExecutionChain.from_llm(llm, verbose=verbose)\n        else:\n            execution_chain = task_execution_chain\n        return cls(\n            task_creation_chain=task_creation_chain,\n            task_prioritization_chain=task_prioritization_chain,\n            execution_chain=execution_chain,\n            vectorstore=vectorstore,\n            **kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/baby_agi/baby_agi.html"}146{"id": "60dfaa8b036a-0", "text": "Source code for langchain.experimental.autonomous_agents.autogpt.agent\nfrom __future__ import annotations\nfrom typing import List, Optional\nfrom pydantic import ValidationError\nfrom langchain.chains.llm import LLMChain\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.experimental.autonomous_agents.autogpt.output_parser import (\n    AutoGPTOutputParser,\n    BaseAutoGPTOutputParser,\n)\nfrom langchain.experimental.autonomous_agents.autogpt.prompt import AutoGPTPrompt\nfrom langchain.experimental.autonomous_agents.autogpt.prompt_generator import (\n    FINISH_NAME,\n)\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    Document,\n    HumanMessage,\n    SystemMessage,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.human.tool import HumanInputRun\nfrom langchain.vectorstores.base import VectorStoreRetriever\n[docs]class AutoGPT:\n    \"\"\"Agent class for interacting with Auto-GPT.\"\"\"\n    def __init__(\n        self,\n        ai_name: str,\n        memory: VectorStoreRetriever,\n        chain: LLMChain,\n        output_parser: BaseAutoGPTOutputParser,\n        tools: List[BaseTool],\n        feedback_tool: Optional[HumanInputRun] = None,\n    ):\n        self.ai_name = ai_name\n        self.memory = memory\n        self.full_message_history: List[BaseMessage] = []\n        self.next_action_count = 0\n        self.chain = chain\n        self.output_parser = output_parser\n        self.tools = tools\n        self.feedback_tool = feedback_tool\n    @classmethod\n    def from_llm_and_tools(\n        cls,\n        ai_name: str,\n        ai_role: str,\n        memory: VectorStoreRetriever,", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/autogpt/agent.html"}147{"id": "60dfaa8b036a-1", "text": "ai_role: str,\n        memory: VectorStoreRetriever,\n        tools: List[BaseTool],\n        llm: BaseChatModel,\n        human_in_the_loop: bool = False,\n        output_parser: Optional[BaseAutoGPTOutputParser] = None,\n    ) -> AutoGPT:\n        prompt = AutoGPTPrompt(\n            ai_name=ai_name,\n            ai_role=ai_role,\n            tools=tools,\n            input_variables=[\"memory\", \"messages\", \"goals\", \"user_input\"],\n            token_counter=llm.get_num_tokens,\n        )\n        human_feedback_tool = HumanInputRun() if human_in_the_loop else None\n        chain = LLMChain(llm=llm, prompt=prompt)\n        return cls(\n            ai_name,\n            memory,\n            chain,\n            output_parser or AutoGPTOutputParser(),\n            tools,\n            feedback_tool=human_feedback_tool,\n        )\n    def run(self, goals: List[str]) -> str:\n        user_input = (\n            \"Determine which next command to use, \"\n            \"and respond using the format specified above:\"\n        )\n        # Interaction Loop\n        loop_count = 0\n        while True:\n            # Discontinue if continuous limit is reached\n            loop_count += 1\n            # Send message to AI, get response\n            assistant_reply = self.chain.run(\n                goals=goals,\n                messages=self.full_message_history,\n                memory=self.memory,\n                user_input=user_input,\n            )\n            # Print Assistant thoughts\n            print(assistant_reply)\n            self.full_message_history.append(HumanMessage(content=user_input))\n            self.full_message_history.append(AIMessage(content=assistant_reply))\n            # Get command name and arguments", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/autogpt/agent.html"}148{"id": "60dfaa8b036a-2", "text": "# Get command name and arguments\n            action = self.output_parser.parse(assistant_reply)\n            tools = {t.name: t for t in self.tools}\n            if action.name == FINISH_NAME:\n                return action.args[\"response\"]\n            if action.name in tools:\n                tool = tools[action.name]\n                try:\n                    observation = tool.run(action.args)\n                except ValidationError as e:\n                    observation = (\n                        f\"Validation Error in args: {str(e)}, args: {action.args}\"\n                    )\n                except Exception as e:\n                    observation = (\n                        f\"Error: {str(e)}, {type(e).__name__}, args: {action.args}\"\n                    )\n                result = f\"Command {tool.name} returned: {observation}\"\n            elif action.name == \"ERROR\":\n                result = f\"Error: {action.args}. \"\n            else:\n                result = (\n                    f\"Unknown command '{action.name}'. \"\n                    f\"Please refer to the 'COMMANDS' list for available \"\n                    f\"commands and only respond in the specified JSON format.\"\n                )\n            memory_to_add = (\n                f\"Assistant Reply: {assistant_reply} \" f\"\\nResult: {result} \"\n            )\n            if self.feedback_tool is not None:\n                feedback = f\"\\n{self.feedback_tool.run('Input: ')}\"\n                if feedback in {\"q\", \"stop\"}:\n                    print(\"EXITING\")\n                    return \"EXITING\"\n                memory_to_add += feedback\n            self.memory.add_documents([Document(page_content=memory_to_add)])\n            self.full_message_history.append(SystemMessage(content=result))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/autonomous_agents/autogpt/agent.html"}149{"id": "2ff250d6c931-0", "text": "Source code for langchain.experimental.generative_agents.generative_agent\nimport re\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom pydantic import BaseModel, Field\nfrom langchain import LLMChain\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.experimental.generative_agents.memory import GenerativeAgentMemory\nfrom langchain.prompts import PromptTemplate\n[docs]class GenerativeAgent(BaseModel):\n    \"\"\"A character with memory and innate characteristics.\"\"\"\n    name: str\n    \"\"\"The character's name.\"\"\"\n    age: Optional[int] = None\n    \"\"\"The optional age of the character.\"\"\"\n    traits: str = \"N/A\"\n    \"\"\"Permanent traits to ascribe to the character.\"\"\"\n    status: str\n    \"\"\"The traits of the character you wish not to change.\"\"\"\n    memory: GenerativeAgentMemory\n    \"\"\"The memory object that combines relevance, recency, and 'importance'.\"\"\"\n    llm: BaseLanguageModel\n    \"\"\"The underlying language model.\"\"\"\n    verbose: bool = False\n    summary: str = \"\"  #: :meta private:\n    \"\"\"Stateful self-summary generated via reflection on the character's memory.\"\"\"\n    summary_refresh_seconds: int = 3600  #: :meta private:\n    \"\"\"How frequently to re-generate the summary.\"\"\"\n    last_refreshed: datetime = Field(default_factory=datetime.now)  # : :meta private:\n    \"\"\"The last time the character's summary was regenerated.\"\"\"\n    daily_summaries: List[str] = Field(default_factory=list)  # : :meta private:\n    \"\"\"Summary of the events in the plan that the agent took.\"\"\"\n[docs]    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    # LLM-related methods\n    @staticmethod", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}150{"id": "2ff250d6c931-1", "text": "arbitrary_types_allowed = True\n    # LLM-related methods\n    @staticmethod\n    def _parse_list(text: str) -> List[str]:\n        \"\"\"Parse a newline-separated string into a list of strings.\"\"\"\n        lines = re.split(r\"\\n\", text.strip())\n        return [re.sub(r\"^\\s*\\d+\\.\\s*\", \"\", line).strip() for line in lines]\n    def chain(self, prompt: PromptTemplate) -> LLMChain:\n        return LLMChain(\n            llm=self.llm, prompt=prompt, verbose=self.verbose, memory=self.memory\n        )\n    def _get_entity_from_observation(self, observation: str) -> str:\n        prompt = PromptTemplate.from_template(\n            \"What is the observed entity in the following observation? {observation}\"\n            + \"\\nEntity=\"\n        )\n        return self.chain(prompt).run(observation=observation).strip()\n    def _get_entity_action(self, observation: str, entity_name: str) -> str:\n        prompt = PromptTemplate.from_template(\n            \"What is the {entity} doing in the following observation? {observation}\"\n            + \"\\nThe {entity} is\"\n        )\n        return (\n            self.chain(prompt).run(entity=entity_name, observation=observation).strip()\n        )\n[docs]    def summarize_related_memories(self, observation: str) -> str:\n        \"\"\"Summarize memories that are most relevant to an observation.\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"\"\"\n{q1}?\nContext from memory:\n{relevant_memories}\nRelevant context: \n\"\"\"\n        )\n        entity_name = self._get_entity_from_observation(observation)\n        entity_action = self._get_entity_action(observation, entity_name)", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}151{"id": "2ff250d6c931-2", "text": "entity_action = self._get_entity_action(observation, entity_name)\n        q1 = f\"What is the relationship between {self.name} and {entity_name}\"\n        q2 = f\"{entity_name} is {entity_action}\"\n        return self.chain(prompt=prompt).run(q1=q1, queries=[q1, q2]).strip()\n    def _generate_reaction(\n        self, observation: str, suffix: str, now: Optional[datetime] = None\n    ) -> str:\n        \"\"\"React to a given observation or dialogue act.\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"{agent_summary_description}\"\n            + \"\\nIt is {current_time}.\"\n            + \"\\n{agent_name}'s status: {agent_status}\"\n            + \"\\nSummary of relevant context from {agent_name}'s memory:\"\n            + \"\\n{relevant_memories}\"\n            + \"\\nMost recent observations: {most_recent_memories}\"\n            + \"\\nObservation: {observation}\"\n            + \"\\n\\n\"\n            + suffix\n        )\n        agent_summary_description = self.get_summary(now=now)\n        relevant_memories_str = self.summarize_related_memories(observation)\n        current_time_str = (\n            datetime.now().strftime(\"%B %d, %Y, %I:%M %p\")\n            if now is None\n            else now.strftime(\"%B %d, %Y, %I:%M %p\")\n        )\n        kwargs: Dict[str, Any] = dict(\n            agent_summary_description=agent_summary_description,\n            current_time=current_time_str,\n            relevant_memories=relevant_memories_str,\n            agent_name=self.name,\n            observation=observation,\n            agent_status=self.status,\n        )\n        consumed_tokens = self.llm.get_num_tokens(", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}152{"id": "2ff250d6c931-3", "text": ")\n        consumed_tokens = self.llm.get_num_tokens(\n            prompt.format(most_recent_memories=\"\", **kwargs)\n        )\n        kwargs[self.memory.most_recent_memories_token_key] = consumed_tokens\n        return self.chain(prompt=prompt).run(**kwargs).strip()\n    def _clean_response(self, text: str) -> str:\n        return re.sub(f\"^{self.name} \", \"\", text.strip()).strip()\n[docs]    def generate_reaction(\n        self, observation: str, now: Optional[datetime] = None\n    ) -> Tuple[bool, str]:\n        \"\"\"React to a given observation.\"\"\"\n        call_to_action_template = (\n            \"Should {agent_name} react to the observation, and if so,\"\n            + \" what would be an appropriate reaction? Respond in one line.\"\n            + ' If the action is to engage in dialogue, write:\\nSAY: \"what to say\"'\n            + \"\\notherwise, write:\\nREACT: {agent_name}'s reaction (if anything).\"\n            + \"\\nEither do nothing, react, or say something but not both.\\n\\n\"\n        )\n        full_result = self._generate_reaction(\n            observation, call_to_action_template, now=now\n        )\n        result = full_result.strip().split(\"\\n\")[0]\n        # AAA\n        self.memory.save_context(\n            {},\n            {\n                self.memory.add_memory_key: f\"{self.name} observed \"\n                f\"{observation} and reacted by {result}\",\n                self.memory.now_key: now,\n            },\n        )\n        if \"REACT:\" in result:\n            reaction = self._clean_response(result.split(\"REACT:\")[-1])\n            return False, f\"{self.name} {reaction}\"\n        if \"SAY:\" in result:", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}153{"id": "2ff250d6c931-4", "text": "if \"SAY:\" in result:\n            said_value = self._clean_response(result.split(\"SAY:\")[-1])\n            return True, f\"{self.name} said {said_value}\"\n        else:\n            return False, result\n[docs]    def generate_dialogue_response(\n        self, observation: str, now: Optional[datetime] = None\n    ) -> Tuple[bool, str]:\n        \"\"\"React to a given observation.\"\"\"\n        call_to_action_template = (\n            \"What would {agent_name} say? To end the conversation, write:\"\n            ' GOODBYE: \"what to say\". Otherwise to continue the conversation,'\n            ' write: SAY: \"what to say next\"\\n\\n'\n        )\n        full_result = self._generate_reaction(\n            observation, call_to_action_template, now=now\n        )\n        result = full_result.strip().split(\"\\n\")[0]\n        if \"GOODBYE:\" in result:\n            farewell = self._clean_response(result.split(\"GOODBYE:\")[-1])\n            self.memory.save_context(\n                {},\n                {\n                    self.memory.add_memory_key: f\"{self.name} observed \"\n                    f\"{observation} and said {farewell}\",\n                    self.memory.now_key: now,\n                },\n            )\n            return False, f\"{self.name} said {farewell}\"\n        if \"SAY:\" in result:\n            response_text = self._clean_response(result.split(\"SAY:\")[-1])\n            self.memory.save_context(\n                {},\n                {\n                    self.memory.add_memory_key: f\"{self.name} observed \"\n                    f\"{observation} and said {response_text}\",\n                    self.memory.now_key: now,\n                },\n            )\n            return True, f\"{self.name} said {response_text}\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}154{"id": "2ff250d6c931-5", "text": "},\n            )\n            return True, f\"{self.name} said {response_text}\"\n        else:\n            return False, result\n    ######################################################\n    # Agent stateful' summary methods.                   #\n    # Each dialog or response prompt includes a header   #\n    # summarizing the agent's self-description. This is  #\n    # updated periodically through probing its memories  #\n    ######################################################\n    def _compute_agent_summary(self) -> str:\n        \"\"\"\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"How would you summarize {name}'s core characteristics given the\"\n            + \" following statements:\\n\"\n            + \"{relevant_memories}\"\n            + \"Do not embellish.\"\n            + \"\\n\\nSummary: \"\n        )\n        # The agent seeks to think about their core characteristics.\n        return (\n            self.chain(prompt)\n            .run(name=self.name, queries=[f\"{self.name}'s core characteristics\"])\n            .strip()\n        )\n[docs]    def get_summary(\n        self, force_refresh: bool = False, now: Optional[datetime] = None\n    ) -> str:\n        \"\"\"Return a descriptive summary of the agent.\"\"\"\n        current_time = datetime.now() if now is None else now\n        since_refresh = (current_time - self.last_refreshed).seconds\n        if (\n            not self.summary\n            or since_refresh >= self.summary_refresh_seconds\n            or force_refresh\n        ):\n            self.summary = self._compute_agent_summary()\n            self.last_refreshed = current_time\n        age = self.age if self.age is not None else \"N/A\"\n        return (\n            f\"Name: {self.name} (age: {age})\"\n            + f\"\\nInnate traits: {self.traits}\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}155{"id": "2ff250d6c931-6", "text": "+ f\"\\nInnate traits: {self.traits}\"\n            + f\"\\n{self.summary}\"\n        )\n[docs]    def get_full_header(\n        self, force_refresh: bool = False, now: Optional[datetime] = None\n    ) -> str:\n        \"\"\"Return a full header of the agent's status, summary, and current time.\"\"\"\n        now = datetime.now() if now is None else now\n        summary = self.get_summary(force_refresh=force_refresh, now=now)\n        current_time_str = now.strftime(\"%B %d, %Y, %I:%M %p\")\n        return (\n            f\"{summary}\\nIt is {current_time_str}.\\n{self.name}'s status: {self.status}\"\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/generative_agent.html"}156{"id": "8c6af54652e4-0", "text": "Source code for langchain.experimental.generative_agents.memory\nimport logging\nimport re\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional\nfrom langchain import LLMChain\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.prompts import PromptTemplate\nfrom langchain.retrievers import TimeWeightedVectorStoreRetriever\nfrom langchain.schema import BaseMemory, Document\nfrom langchain.utils import mock_now\nlogger = logging.getLogger(__name__)\n[docs]class GenerativeAgentMemory(BaseMemory):\n    llm: BaseLanguageModel\n    \"\"\"The core language model.\"\"\"\n    memory_retriever: TimeWeightedVectorStoreRetriever\n    \"\"\"The retriever to fetch related memories.\"\"\"\n    verbose: bool = False\n    reflection_threshold: Optional[float] = None\n    \"\"\"When aggregate_importance exceeds reflection_threshold, stop to reflect.\"\"\"\n    current_plan: List[str] = []\n    \"\"\"The current plan of the agent.\"\"\"\n    # A weight of 0.15 makes this less important than it\n    # would be otherwise, relative to salience and time\n    importance_weight: float = 0.15\n    \"\"\"How much weight to assign the memory importance.\"\"\"\n    aggregate_importance: float = 0.0  # : :meta private:\n    \"\"\"Track the sum of the 'importance' of recent memories.\n    Triggers reflection when it reaches reflection_threshold.\"\"\"\n    max_tokens_limit: int = 1200  # : :meta private:\n    # input keys\n    queries_key: str = \"queries\"\n    most_recent_memories_token_key: str = \"recent_memories_token\"\n    add_memory_key: str = \"add_memory\"\n    # output keys\n    relevant_memories_key: str = \"relevant_memories\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}157{"id": "8c6af54652e4-1", "text": "# output keys\n    relevant_memories_key: str = \"relevant_memories\"\n    relevant_memories_simple_key: str = \"relevant_memories_simple\"\n    most_recent_memories_key: str = \"most_recent_memories\"\n    now_key: str = \"now\"\n    reflecting: bool = False\n    def chain(self, prompt: PromptTemplate) -> LLMChain:\n        return LLMChain(llm=self.llm, prompt=prompt, verbose=self.verbose)\n    @staticmethod\n    def _parse_list(text: str) -> List[str]:\n        \"\"\"Parse a newline-separated string into a list of strings.\"\"\"\n        lines = re.split(r\"\\n\", text.strip())\n        lines = [line for line in lines if line.strip()]  # remove empty lines\n        return [re.sub(r\"^\\s*\\d+\\.\\s*\", \"\", line).strip() for line in lines]\n    def _get_topics_of_reflection(self, last_k: int = 50) -> List[str]:\n        \"\"\"Return the 3 most salient high-level questions about recent observations.\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"{observations}\\n\\n\"\n            + \"Given only the information above, what are the 3 most salient\"\n            + \" high-level questions we can answer about the subjects in\"\n            + \" the statements? Provide each question on a new line.\\n\\n\"\n        )\n        observations = self.memory_retriever.memory_stream[-last_k:]\n        observation_str = \"\\n\".join([o.page_content for o in observations])\n        result = self.chain(prompt).run(observations=observation_str)\n        return self._parse_list(result)\n    def _get_insights_on_topic(\n        self, topic: str, now: Optional[datetime] = None\n    ) -> List[str]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}158{"id": "8c6af54652e4-2", "text": ") -> List[str]:\n        \"\"\"Generate 'insights' on a topic of reflection, based on pertinent memories.\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"Statements about {topic}\\n\"\n            + \"{related_statements}\\n\\n\"\n            + \"What 5 high-level insights can you infer from the above statements?\"\n            + \" (example format: insight (because of 1, 5, 3))\"\n        )\n        related_memories = self.fetch_memories(topic, now=now)\n        related_statements = \"\\n\".join(\n            [\n                f\"{i+1}. {memory.page_content}\"\n                for i, memory in enumerate(related_memories)\n            ]\n        )\n        result = self.chain(prompt).run(\n            topic=topic, related_statements=related_statements\n        )\n        # TODO: Parse the connections between memories and insights\n        return self._parse_list(result)\n[docs]    def pause_to_reflect(self, now: Optional[datetime] = None) -> List[str]:\n        \"\"\"Reflect on recent observations and generate 'insights'.\"\"\"\n        if self.verbose:\n            logger.info(\"Character is reflecting\")\n        new_insights = []\n        topics = self._get_topics_of_reflection()\n        for topic in topics:\n            insights = self._get_insights_on_topic(topic, now=now)\n            for insight in insights:\n                self.add_memory(insight, now=now)\n            new_insights.extend(insights)\n        return new_insights\n    def _score_memory_importance(self, memory_content: str) -> float:\n        \"\"\"Score the absolute importance of the given memory.\"\"\"\n        prompt = PromptTemplate.from_template(\n            \"On the scale of 1 to 10, where 1 is purely mundane\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}159{"id": "8c6af54652e4-3", "text": "\"On the scale of 1 to 10, where 1 is purely mundane\"\n            + \" (e.g., brushing teeth, making bed) and 10 is\"\n            + \" extremely poignant (e.g., a break up, college\"\n            + \" acceptance), rate the likely poignancy of the\"\n            + \" following piece of memory. Respond with a single integer.\"\n            + \"\\nMemory: {memory_content}\"\n            + \"\\nRating: \"\n        )\n        score = self.chain(prompt).run(memory_content=memory_content).strip()\n        if self.verbose:\n            logger.info(f\"Importance score: {score}\")\n        match = re.search(r\"^\\D*(\\d+)\", score)\n        if match:\n            return (float(match.group(1)) / 10) * self.importance_weight\n        else:\n            return 0.0\n[docs]    def add_memory(\n        self, memory_content: str, now: Optional[datetime] = None\n    ) -> List[str]:\n        \"\"\"Add an observation or memory to the agent's memory.\"\"\"\n        importance_score = self._score_memory_importance(memory_content)\n        self.aggregate_importance += importance_score\n        document = Document(\n            page_content=memory_content, metadata={\"importance\": importance_score}\n        )\n        result = self.memory_retriever.add_documents([document], current_time=now)\n        # After an agent has processed a certain amount of memories (as measured by\n        # aggregate importance), it is time to reflect on recent events to add\n        # more synthesized memories to the agent's memory stream.\n        if (\n            self.reflection_threshold is not None\n            and self.aggregate_importance > self.reflection_threshold\n            and not self.reflecting\n        ):\n            self.reflecting = True", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}160{"id": "8c6af54652e4-4", "text": "and not self.reflecting\n        ):\n            self.reflecting = True\n            self.pause_to_reflect(now=now)\n            # Hack to clear the importance from reflection\n            self.aggregate_importance = 0.0\n            self.reflecting = False\n        return result\n[docs]    def fetch_memories(\n        self, observation: str, now: Optional[datetime] = None\n    ) -> List[Document]:\n        \"\"\"Fetch related memories.\"\"\"\n        if now is not None:\n            with mock_now(now):\n                return self.memory_retriever.get_relevant_documents(observation)\n        else:\n            return self.memory_retriever.get_relevant_documents(observation)\n    def format_memories_detail(self, relevant_memories: List[Document]) -> str:\n        content_strs = set()\n        content = []\n        for mem in relevant_memories:\n            if mem.page_content in content_strs:\n                continue\n            content_strs.add(mem.page_content)\n            created_time = mem.metadata[\"created_at\"].strftime(\"%B %d, %Y, %I:%M %p\")\n            content.append(f\"- {created_time}: {mem.page_content.strip()}\")\n        return \"\\n\".join([f\"{mem}\" for mem in content])\n    def format_memories_simple(self, relevant_memories: List[Document]) -> str:\n        return \"; \".join([f\"{mem.page_content}\" for mem in relevant_memories])\n    def _get_memories_until_limit(self, consumed_tokens: int) -> str:\n        \"\"\"Reduce the number of tokens in the documents.\"\"\"\n        result = []\n        for doc in self.memory_retriever.memory_stream[::-1]:\n            if consumed_tokens >= self.max_tokens_limit:\n                break\n            consumed_tokens += self.llm.get_num_tokens(doc.page_content)", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}161{"id": "8c6af54652e4-5", "text": "break\n            consumed_tokens += self.llm.get_num_tokens(doc.page_content)\n            if consumed_tokens < self.max_tokens_limit:\n                result.append(doc)\n        return self.format_memories_simple(result)\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Input keys this memory class will load dynamically.\"\"\"\n        return []\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        \"\"\"Return key-value pairs given the text input to the chain.\"\"\"\n        queries = inputs.get(self.queries_key)\n        now = inputs.get(self.now_key)\n        if queries is not None:\n            relevant_memories = [\n                mem for query in queries for mem in self.fetch_memories(query, now=now)\n            ]\n            return {\n                self.relevant_memories_key: self.format_memories_detail(\n                    relevant_memories\n                ),\n                self.relevant_memories_simple_key: self.format_memories_simple(\n                    relevant_memories\n                ),\n            }\n        most_recent_memories_token = inputs.get(self.most_recent_memories_token_key)\n        if most_recent_memories_token is not None:\n            return {\n                self.most_recent_memories_key: self._get_memories_until_limit(\n                    most_recent_memories_token\n                )\n            }\n        return {}\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, Any]) -> None:\n        \"\"\"Save the context of this model run to memory.\"\"\"\n        # TODO: fix the save memory key\n        mem = outputs.get(self.add_memory_key)\n        now = outputs.get(self.now_key)\n        if mem:\n            self.add_memory(mem, now=now)\n[docs]    def clear(self) -> None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}162{"id": "8c6af54652e4-6", "text": "[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        # TODO\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/experimental/generative_agents/memory.html"}163{"id": "f436e6e350ad-0", "text": "Source code for langchain.retrievers.time_weighted_retriever\n\"\"\"Retriever that combines embedding similarity with recency in retrieving values.\"\"\"\nimport datetime\nfrom copy import deepcopy\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom pydantic import BaseModel, Field\nfrom langchain.schema import BaseRetriever, Document\nfrom langchain.vectorstores.base import VectorStore\ndef _get_hours_passed(time: datetime.datetime, ref_time: datetime.datetime) -> float:\n    \"\"\"Get the hours passed between two datetime objects.\"\"\"\n    return (time - ref_time).total_seconds() / 3600\n[docs]class TimeWeightedVectorStoreRetriever(BaseRetriever, BaseModel):\n    \"\"\"Retriever combining embedding similarity with recency.\"\"\"\n    vectorstore: VectorStore\n    \"\"\"The vectorstore to store documents and determine salience.\"\"\"\n    search_kwargs: dict = Field(default_factory=lambda: dict(k=100))\n    \"\"\"Keyword arguments to pass to the vectorstore similarity search.\"\"\"\n    # TODO: abstract as a queue\n    memory_stream: List[Document] = Field(default_factory=list)\n    \"\"\"The memory_stream of documents to search through.\"\"\"\n    decay_rate: float = Field(default=0.01)\n    \"\"\"The exponential decay factor used as (1.0-decay_rate)**(hrs_passed).\"\"\"\n    k: int = 4\n    \"\"\"The maximum number of documents to retrieve in a given call.\"\"\"\n    other_score_keys: List[str] = []\n    \"\"\"Other keys in the metadata to factor into the score, e.g. 'importance'.\"\"\"\n    default_salience: Optional[float] = None\n    \"\"\"The salience to assign memories not retrieved from the vector store.\n    None assigns no salience to documents not fetched from the vector store.\n    \"\"\"\n    class Config:", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/time_weighted_retriever.html"}164{"id": "f436e6e350ad-1", "text": "\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    def _get_combined_score(\n        self,\n        document: Document,\n        vector_relevance: Optional[float],\n        current_time: datetime.datetime,\n    ) -> float:\n        \"\"\"Return the combined score for a document.\"\"\"\n        hours_passed = _get_hours_passed(\n            current_time,\n            document.metadata[\"last_accessed_at\"],\n        )\n        score = (1.0 - self.decay_rate) ** hours_passed\n        for key in self.other_score_keys:\n            if key in document.metadata:\n                score += document.metadata[key]\n        if vector_relevance is not None:\n            score += vector_relevance\n        return score\n[docs]    def get_salient_docs(self, query: str) -> Dict[int, Tuple[Document, float]]:\n        \"\"\"Return documents that are salient to the query.\"\"\"\n        docs_and_scores: List[Tuple[Document, float]]\n        docs_and_scores = self.vectorstore.similarity_search_with_relevance_scores(\n            query, **self.search_kwargs\n        )\n        results = {}\n        for fetched_doc, relevance in docs_and_scores:\n            if \"buffer_idx\" in fetched_doc.metadata:\n                buffer_idx = fetched_doc.metadata[\"buffer_idx\"]\n                doc = self.memory_stream[buffer_idx]\n                results[buffer_idx] = (doc, relevance)\n        return results\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        \"\"\"Return documents that are relevant to the query.\"\"\"\n        current_time = datetime.datetime.now()\n        docs_and_scores = {\n            doc.metadata[\"buffer_idx\"]: (doc, self.default_salience)\n            for doc in self.memory_stream[-self.k :]\n        }", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/time_weighted_retriever.html"}165{"id": "f436e6e350ad-2", "text": "for doc in self.memory_stream[-self.k :]\n        }\n        # If a doc is considered salient, update the salience score\n        docs_and_scores.update(self.get_salient_docs(query))\n        rescored_docs = [\n            (doc, self._get_combined_score(doc, relevance, current_time))\n            for doc, relevance in docs_and_scores.values()\n        ]\n        rescored_docs.sort(key=lambda x: x[1], reverse=True)\n        result = []\n        # Ensure frequently accessed memories aren't forgotten\n        for doc, _ in rescored_docs[: self.k]:\n            # TODO: Update vector store doc once `update` method is exposed.\n            buffered_doc = self.memory_stream[doc.metadata[\"buffer_idx\"]]\n            buffered_doc.metadata[\"last_accessed_at\"] = current_time\n            result.append(buffered_doc)\n        return result\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        \"\"\"Return documents that are relevant to the query.\"\"\"\n        raise NotImplementedError\n[docs]    def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        current_time = kwargs.get(\"current_time\")\n        if current_time is None:\n            current_time = datetime.datetime.now()\n        # Avoid mutating input documents\n        dup_docs = [deepcopy(d) for d in documents]\n        for i, doc in enumerate(dup_docs):\n            if \"last_accessed_at\" not in doc.metadata:\n                doc.metadata[\"last_accessed_at\"] = current_time\n            if \"created_at\" not in doc.metadata:\n                doc.metadata[\"created_at\"] = current_time\n            doc.metadata[\"buffer_idx\"] = len(self.memory_stream) + i\n        self.memory_stream.extend(dup_docs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/time_weighted_retriever.html"}166{"id": "f436e6e350ad-3", "text": "self.memory_stream.extend(dup_docs)\n        return self.vectorstore.add_documents(dup_docs, **kwargs)\n[docs]    async def aadd_documents(\n        self, documents: List[Document], **kwargs: Any\n    ) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        current_time = kwargs.get(\"current_time\")\n        if current_time is None:\n            current_time = datetime.datetime.now()\n        # Avoid mutating input documents\n        dup_docs = [deepcopy(d) for d in documents]\n        for i, doc in enumerate(dup_docs):\n            if \"last_accessed_at\" not in doc.metadata:\n                doc.metadata[\"last_accessed_at\"] = current_time\n            if \"created_at\" not in doc.metadata:\n                doc.metadata[\"created_at\"] = current_time\n            doc.metadata[\"buffer_idx\"] = len(self.memory_stream) + i\n        self.memory_stream.extend(dup_docs)\n        return await self.vectorstore.aadd_documents(dup_docs, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/time_weighted_retriever.html"}167{"id": "71e783f568f8-0", "text": "Source code for langchain.retrievers.pinecone_hybrid_search\n\"\"\"Taken from: https://docs.pinecone.io/docs/hybrid-search\"\"\"\nimport hashlib\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.schema import BaseRetriever, Document\ndef hash_text(text: str) -> str:\n    return str(hashlib.sha256(text.encode(\"utf-8\")).hexdigest())\ndef create_index(\n    contexts: List[str],\n    index: Any,\n    embeddings: Embeddings,\n    sparse_encoder: Any,\n    ids: Optional[List[str]] = None,\n    metadatas: Optional[List[dict]] = None,\n) -> None:\n    batch_size = 32\n    _iterator = range(0, len(contexts), batch_size)\n    try:\n        from tqdm.auto import tqdm\n        _iterator = tqdm(_iterator)\n    except ImportError:\n        pass\n    if ids is None:\n        # create unique ids using hash of the text\n        ids = [hash_text(context) for context in contexts]\n    for i in _iterator:\n        # find end of batch\n        i_end = min(i + batch_size, len(contexts))\n        # extract batch\n        context_batch = contexts[i:i_end]\n        batch_ids = ids[i:i_end]\n        metadata_batch = (\n            metadatas[i:i_end] if metadatas else [{} for _ in context_batch]\n        )\n        # add context passages as metadata\n        meta = [\n            {\"context\": context, **metadata}\n            for context, metadata in zip(context_batch, metadata_batch)\n        ]\n        # create dense vectors\n        dense_embeds = embeddings.embed_documents(context_batch)", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/pinecone_hybrid_search.html"}168{"id": "71e783f568f8-1", "text": "]\n        # create dense vectors\n        dense_embeds = embeddings.embed_documents(context_batch)\n        # create sparse vectors\n        sparse_embeds = sparse_encoder.encode_documents(context_batch)\n        for s in sparse_embeds:\n            s[\"values\"] = [float(s1) for s1 in s[\"values\"]]\n        vectors = []\n        # loop through the data and create dictionaries for upserts\n        for doc_id, sparse, dense, metadata in zip(\n            batch_ids, sparse_embeds, dense_embeds, meta\n        ):\n            vectors.append(\n                {\n                    \"id\": doc_id,\n                    \"sparse_values\": sparse,\n                    \"values\": dense,\n                    \"metadata\": metadata,\n                }\n            )\n        # upload the documents to the new hybrid index\n        index.upsert(vectors)\n[docs]class PineconeHybridSearchRetriever(BaseRetriever, BaseModel):\n    embeddings: Embeddings\n    sparse_encoder: Any\n    index: Any\n    top_k: int = 4\n    alpha: float = 0.5\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n[docs]    def add_texts(\n        self,\n        texts: List[str],\n        ids: Optional[List[str]] = None,\n        metadatas: Optional[List[dict]] = None,\n    ) -> None:\n        create_index(\n            texts,\n            self.index,\n            self.embeddings,\n            self.sparse_encoder,\n            ids=ids,\n            metadatas=metadatas,\n        )\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/pinecone_hybrid_search.html"}169{"id": "71e783f568f8-2", "text": "\"\"\"Validate that api key and python package exists in environment.\"\"\"\n        try:\n            from pinecone_text.hybrid import hybrid_convex_scale  # noqa:F401\n            from pinecone_text.sparse.base_sparse_encoder import (\n                BaseSparseEncoder,  # noqa:F401\n            )\n        except ImportError:\n            raise ValueError(\n                \"Could not import pinecone_text python package. \"\n                \"Please install it with `pip install pinecone_text`.\"\n            )\n        return values\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        from pinecone_text.hybrid import hybrid_convex_scale\n        sparse_vec = self.sparse_encoder.encode_queries(query)\n        # convert the question into a dense vector\n        dense_vec = self.embeddings.embed_query(query)\n        # scale alpha with hybrid_scale\n        dense_vec, sparse_vec = hybrid_convex_scale(dense_vec, sparse_vec, self.alpha)\n        sparse_vec[\"values\"] = [float(s1) for s1 in sparse_vec[\"values\"]]\n        # query pinecone with the query parameters\n        result = self.index.query(\n            vector=dense_vec,\n            sparse_vector=sparse_vec,\n            top_k=self.top_k,\n            include_metadata=True,\n        )\n        final_result = []\n        for res in result[\"matches\"]:\n            context = res[\"metadata\"].pop(\"context\")\n            final_result.append(\n                Document(page_content=context, metadata=res[\"metadata\"])\n            )\n        # return search results as json\n        return final_result\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/pinecone_hybrid_search.html"}170{"id": "fb2adbac36ba-0", "text": "Source code for langchain.retrievers.vespa_retriever\n\"\"\"Wrapper for retrieving documents from Vespa.\"\"\"\nfrom __future__ import annotations\nimport json\nfrom typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Sequence, Union\nfrom langchain.schema import BaseRetriever, Document\nif TYPE_CHECKING:\n    from vespa.application import Vespa\n[docs]class VespaRetriever(BaseRetriever):\n    def __init__(\n        self,\n        app: Vespa,\n        body: Dict,\n        content_field: str,\n        metadata_fields: Optional[Sequence[str]] = None,\n    ):\n        self._application = app\n        self._query_body = body\n        self._content_field = content_field\n        self._metadata_fields = metadata_fields or ()\n    def _query(self, body: Dict) -> List[Document]:\n        response = self._application.query(body)\n        if not str(response.status_code).startswith(\"2\"):\n            raise RuntimeError(\n                \"Could not retrieve data from Vespa. Error code: {}\".format(\n                    response.status_code\n                )\n            )\n        root = response.json[\"root\"]\n        if \"errors\" in root:\n            raise RuntimeError(json.dumps(root[\"errors\"]))\n        docs = []\n        for child in response.hits:\n            page_content = child[\"fields\"].pop(self._content_field, \"\")\n            if self._metadata_fields == \"*\":\n                metadata = child[\"fields\"]\n            else:\n                metadata = {mf: child[\"fields\"].get(mf) for mf in self._metadata_fields}\n            metadata[\"id\"] = child[\"id\"]\n            docs.append(Document(page_content=page_content, metadata=metadata))\n        return docs", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/vespa_retriever.html"}171{"id": "fb2adbac36ba-1", "text": "docs.append(Document(page_content=page_content, metadata=metadata))\n        return docs\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        body = self._query_body.copy()\n        body[\"query\"] = query\n        return self._query(body)\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\n[docs]    def get_relevant_documents_with_filter(\n        self, query: str, *, _filter: Optional[str] = None\n    ) -> List[Document]:\n        body = self._query_body.copy()\n        _filter = f\" and {_filter}\" if _filter else \"\"\n        body[\"yql\"] = body[\"yql\"] + _filter\n        body[\"query\"] = query\n        return self._query(body)\n[docs]    @classmethod\n    def from_params(\n        cls,\n        url: str,\n        content_field: str,\n        *,\n        k: Optional[int] = None,\n        metadata_fields: Union[Sequence[str], Literal[\"*\"]] = (),\n        sources: Union[Sequence[str], Literal[\"*\"], None] = None,\n        _filter: Optional[str] = None,\n        yql: Optional[str] = None,\n        **kwargs: Any,\n    ) -> VespaRetriever:\n        \"\"\"Instantiate retriever from params.\n        Args:\n            url (str): Vespa app URL.\n            content_field (str): Field in results to return as Document page_content.\n            k (Optional[int]): Number of Documents to return. Defaults to None.\n            metadata_fields(Sequence[str] or \"*\"): Fields in results to include in\n                document metadata. Defaults to empty tuple ().", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/vespa_retriever.html"}172{"id": "fb2adbac36ba-2", "text": "document metadata. Defaults to empty tuple ().\n            sources (Sequence[str] or \"*\" or None): Sources to retrieve\n                from. Defaults to None.\n            _filter (Optional[str]): Document filter condition expressed in YQL.\n                Defaults to None.\n            yql (Optional[str]): Full YQL query to be used. Should not be specified\n                if _filter or sources are specified. Defaults to None.\n            kwargs (Any): Keyword arguments added to query body.\n        \"\"\"\n        try:\n            from vespa.application import Vespa\n        except ImportError:\n            raise ImportError(\n                \"pyvespa is not installed, please install with `pip install pyvespa`\"\n            )\n        app = Vespa(url)\n        body = kwargs.copy()\n        if yql and (sources or _filter):\n            raise ValueError(\n                \"yql should only be specified if both sources and _filter are not \"\n                \"specified.\"\n            )\n        else:\n            if metadata_fields == \"*\":\n                _fields = \"*\"\n                body[\"summary\"] = \"short\"\n            else:\n                _fields = \", \".join([content_field] + list(metadata_fields or []))\n            _sources = \", \".join(sources) if isinstance(sources, Sequence) else \"*\"\n            _filter = f\" and {_filter}\" if _filter else \"\"\n            yql = f\"select {_fields} from sources {_sources} where userQuery(){_filter}\"\n        body[\"yql\"] = yql\n        if k:\n            body[\"hits\"] = k\n        return cls(app, body, content_field, metadata_fields=metadata_fields)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/vespa_retriever.html"}173{"id": "1a11929144fb-0", "text": "Source code for langchain.retrievers.tfidf\n\"\"\"TF-IDF Retriever.\nLargely based on\nhttps://github.com/asvskartheek/Text-Retrieval/blob/master/TF-IDF%20Search%20Engine%20(SKLEARN).ipynb\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, Dict, Iterable, List, Optional\nfrom pydantic import BaseModel\nfrom langchain.schema import BaseRetriever, Document\n[docs]class TFIDFRetriever(BaseRetriever, BaseModel):\n    vectorizer: Any\n    docs: List[Document]\n    tfidf_array: Any\n    k: int = 4\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: Iterable[str],\n        metadatas: Optional[Iterable[dict]] = None,\n        tfidf_params: Optional[Dict[str, Any]] = None,\n        **kwargs: Any,\n    ) -> TFIDFRetriever:\n        try:\n            from sklearn.feature_extraction.text import TfidfVectorizer\n        except ImportError:\n            raise ImportError(\n                \"Could not import scikit-learn, please install with `pip install \"\n                \"scikit-learn`.\"\n            )\n        tfidf_params = tfidf_params or {}\n        vectorizer = TfidfVectorizer(**tfidf_params)\n        tfidf_array = vectorizer.fit_transform(texts)\n        metadatas = metadatas or ({} for _ in texts)\n        docs = [Document(page_content=t, metadata=m) for t, m in zip(texts, metadatas)]", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/tfidf.html"}174{"id": "1a11929144fb-1", "text": "return cls(vectorizer=vectorizer, docs=docs, tfidf_array=tfidf_array, **kwargs)\n[docs]    @classmethod\n    def from_documents(\n        cls,\n        documents: Iterable[Document],\n        *,\n        tfidf_params: Optional[Dict[str, Any]] = None,\n        **kwargs: Any,\n    ) -> TFIDFRetriever:\n        texts, metadatas = zip(*((d.page_content, d.metadata) for d in documents))\n        return cls.from_texts(\n            texts=texts, tfidf_params=tfidf_params, metadatas=metadatas, **kwargs\n        )\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        from sklearn.metrics.pairwise import cosine_similarity\n        query_vec = self.vectorizer.transform(\n            [query]\n        )  # Ip -- (n_docs,x), Op -- (n_docs,n_Feats)\n        results = cosine_similarity(self.tfidf_array, query_vec).reshape(\n            (-1,)\n        )  # Op -- (n_docs,1) -- Cosine Sim with each doc\n        return_docs = []\n        for i in results.argsort()[-self.k :][::-1]:\n            return_docs.append(self.docs[i])\n        return return_docs\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/tfidf.html"}175{"id": "c6a51b88e88c-0", "text": "Source code for langchain.retrievers.svm\n\"\"\"SMV Retriever.\nLargely based on\nhttps://github.com/karpathy/randomfun/blob/master/knn_vs_svm.ipynb\"\"\"\nfrom __future__ import annotations\nimport concurrent.futures\nfrom typing import Any, List, Optional\nimport numpy as np\nfrom pydantic import BaseModel\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.schema import BaseRetriever, Document\ndef create_index(contexts: List[str], embeddings: Embeddings) -> np.ndarray:\n    with concurrent.futures.ThreadPoolExecutor() as executor:\n        return np.array(list(executor.map(embeddings.embed_query, contexts)))\n[docs]class SVMRetriever(BaseRetriever, BaseModel):\n    embeddings: Embeddings\n    index: Any\n    texts: List[str]\n    k: int = 4\n    relevancy_threshold: Optional[float] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    @classmethod\n    def from_texts(\n        cls, texts: List[str], embeddings: Embeddings, **kwargs: Any\n    ) -> SVMRetriever:\n        index = create_index(texts, embeddings)\n        return cls(embeddings=embeddings, index=index, texts=texts, **kwargs)\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        from sklearn import svm\n        query_embeds = np.array(self.embeddings.embed_query(query))\n        x = np.concatenate([query_embeds[None, ...], self.index])\n        y = np.zeros(x.shape[0])\n        y[0] = 1\n        clf = svm.LinearSVC(", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/svm.html"}176{"id": "c6a51b88e88c-1", "text": "y[0] = 1\n        clf = svm.LinearSVC(\n            class_weight=\"balanced\", verbose=False, max_iter=10000, tol=1e-6, C=0.1\n        )\n        clf.fit(x, y)\n        similarities = clf.decision_function(x)\n        sorted_ix = np.argsort(-similarities)\n        # svm.LinearSVC in scikit-learn is non-deterministic.\n        # if a text is the same as a query, there is no guarantee\n        # the query will be in the first index.\n        # this performs a simple swap, this works because anything\n        # left of the 0 should be equivalent.\n        zero_index = np.where(sorted_ix == 0)[0][0]\n        if zero_index != 0:\n            sorted_ix[0], sorted_ix[zero_index] = sorted_ix[zero_index], sorted_ix[0]\n        denominator = np.max(similarities) - np.min(similarities) + 1e-6\n        normalized_similarities = (similarities - np.min(similarities)) / denominator\n        top_k_results = []\n        for row in sorted_ix[1 : self.k + 1]:\n            if (\n                self.relevancy_threshold is None\n                or normalized_similarities[row] >= self.relevancy_threshold\n            ):\n                top_k_results.append(Document(page_content=self.texts[row - 1]))\n        return top_k_results\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/svm.html"}177{"id": "89ab34c265d1-0", "text": "Source code for langchain.retrievers.wikipedia\nfrom typing import List\nfrom langchain.schema import BaseRetriever, Document\nfrom langchain.utilities.wikipedia import WikipediaAPIWrapper\n[docs]class WikipediaRetriever(BaseRetriever, WikipediaAPIWrapper):\n    \"\"\"\n    It is effectively a wrapper for WikipediaAPIWrapper.\n    It wraps load() to get_relevant_documents().\n    It uses all WikipediaAPIWrapper arguments without any change.\n    \"\"\"\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        return self.load(query=query)\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/wikipedia.html"}178{"id": "711b7ff05023-0", "text": "Source code for langchain.retrievers.azure_cognitive_search\n\"\"\"Retriever wrapper for Azure Cognitive Search.\"\"\"\nfrom __future__ import annotations\nimport json\nfrom typing import Dict, List, Optional\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.schema import BaseRetriever, Document\nfrom langchain.utils import get_from_dict_or_env\n[docs]class AzureCognitiveSearchRetriever(BaseRetriever, BaseModel):\n    \"\"\"Wrapper around Azure Cognitive Search.\"\"\"\n    service_name: str = \"\"\n    \"\"\"Name of Azure Cognitive Search service\"\"\"\n    index_name: str = \"\"\n    \"\"\"Name of Index inside Azure Cognitive Search service\"\"\"\n    api_key: str = \"\"\n    \"\"\"API Key. Both Admin and Query keys work, but for reading data it's\n    recommended to use a Query key.\"\"\"\n    api_version: str = \"2020-06-30\"\n    \"\"\"API version\"\"\"\n    aiosession: Optional[aiohttp.ClientSession] = None\n    \"\"\"ClientSession, in case we want to reuse connection for better performance.\"\"\"\n    content_key: str = \"content\"\n    \"\"\"Key in a retrieved result to set as the Document page_content.\"\"\"\n    class Config:\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that service name, index name and api key exists in environment.\"\"\"\n        values[\"service_name\"] = get_from_dict_or_env(\n            values, \"service_name\", \"AZURE_COGNITIVE_SEARCH_SERVICE_NAME\"\n        )\n        values[\"index_name\"] = get_from_dict_or_env(\n            values, \"index_name\", \"AZURE_COGNITIVE_SEARCH_INDEX_NAME\"\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/azure_cognitive_search.html"}179{"id": "711b7ff05023-1", "text": ")\n        values[\"api_key\"] = get_from_dict_or_env(\n            values, \"api_key\", \"AZURE_COGNITIVE_SEARCH_API_KEY\"\n        )\n        return values\n    def _build_search_url(self, query: str) -> str:\n        base_url = f\"https://{self.service_name}.search.windows.net/\"\n        endpoint_path = f\"indexes/{self.index_name}/docs?api-version={self.api_version}\"\n        return base_url + endpoint_path + f\"&search={query}\"\n    @property\n    def _headers(self) -> Dict[str, str]:\n        return {\n            \"Content-Type\": \"application/json\",\n            \"api-key\": self.api_key,\n        }\n    def _search(self, query: str) -> List[dict]:\n        search_url = self._build_search_url(query)\n        response = requests.get(search_url, headers=self._headers)\n        if response.status_code != 200:\n            raise Exception(f\"Error in search request: {response}\")\n        return json.loads(response.text)[\"value\"]\n    async def _asearch(self, query: str) -> List[dict]:\n        search_url = self._build_search_url(query)\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.get(search_url, headers=self._headers) as response:\n                    response_json = await response.json()\n        else:\n            async with self.aiosession.get(\n                search_url, headers=self._headers\n            ) as response:\n                response_json = await response.json()\n        return response_json[\"value\"]\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        search_results = self._search(query)\n        return [", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/azure_cognitive_search.html"}180{"id": "711b7ff05023-2", "text": "search_results = self._search(query)\n        return [\n            Document(page_content=result.pop(self.content_key), metadata=result)\n            for result in search_results\n        ]\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        search_results = await self._asearch(query)\n        return [\n            Document(page_content=result.pop(self.content_key), metadata=result)\n            for result in search_results\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/azure_cognitive_search.html"}181{"id": "18697f24da25-0", "text": "Source code for langchain.retrievers.elastic_search_bm25\n\"\"\"Wrapper around Elasticsearch vector database.\"\"\"\nfrom __future__ import annotations\nimport uuid\nfrom typing import Any, Iterable, List\nfrom langchain.docstore.document import Document\nfrom langchain.schema import BaseRetriever\n[docs]class ElasticSearchBM25Retriever(BaseRetriever):\n    \"\"\"Wrapper around Elasticsearch using BM25 as a retrieval method.\n    To connect to an Elasticsearch instance that requires login credentials,\n    including Elastic Cloud, use the Elasticsearch URL format\n    https://username:password@es_host:9243. For example, to connect to Elastic\n    Cloud, create the Elasticsearch URL with the required authentication details and\n    pass it to the ElasticVectorSearch constructor as the named parameter\n    elasticsearch_url.\n    You can obtain your Elastic Cloud URL and login credentials by logging in to the\n    Elastic Cloud console at https://cloud.elastic.co, selecting your deployment, and\n    navigating to the \"Deployments\" page.\n    To obtain your Elastic Cloud password for the default \"elastic\" user:\n    1. Log in to the Elastic Cloud console at https://cloud.elastic.co\n    2. Go to \"Security\" > \"Users\"\n    3. Locate the \"elastic\" user and click \"Edit\"\n    4. Click \"Reset password\"\n    5. Follow the prompts to reset the password\n    The format for Elastic Cloud URLs is\n    https://username:password@cluster_id.region_id.gcp.cloud.es.io:9243.\n    \"\"\"\n    def __init__(self, client: Any, index_name: str):\n        self.client = client\n        self.index_name = index_name\n[docs]    @classmethod\n    def create(", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/elastic_search_bm25.html"}182{"id": "18697f24da25-1", "text": "self.index_name = index_name\n[docs]    @classmethod\n    def create(\n        cls, elasticsearch_url: str, index_name: str, k1: float = 2.0, b: float = 0.75\n    ) -> ElasticSearchBM25Retriever:\n        from elasticsearch import Elasticsearch\n        # Create an Elasticsearch client instance\n        es = Elasticsearch(elasticsearch_url)\n        # Define the index settings and mappings\n        settings = {\n            \"analysis\": {\"analyzer\": {\"default\": {\"type\": \"standard\"}}},\n            \"similarity\": {\n                \"custom_bm25\": {\n                    \"type\": \"BM25\",\n                    \"k1\": k1,\n                    \"b\": b,\n                }\n            },\n        }\n        mappings = {\n            \"properties\": {\n                \"content\": {\n                    \"type\": \"text\",\n                    \"similarity\": \"custom_bm25\",  # Use the custom BM25 similarity\n                }\n            }\n        }\n        # Create the index with the specified settings and mappings\n        es.indices.create(index=index_name, mappings=mappings, settings=settings)\n        return cls(es, index_name)\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        refresh_indices: bool = True,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the retriver.\n        Args:\n            texts: Iterable of strings to add to the retriever.\n            refresh_indices: bool to refresh ElasticSearch indices\n        Returns:\n            List of ids from adding the texts into the retriever.\n        \"\"\"\n        try:\n            from elasticsearch.helpers import bulk\n        except ImportError:\n            raise ValueError(\n                \"Could not import elasticsearch python package. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/elastic_search_bm25.html"}183{"id": "18697f24da25-2", "text": "raise ValueError(\n                \"Could not import elasticsearch python package. \"\n                \"Please install it with `pip install elasticsearch`.\"\n            )\n        requests = []\n        ids = []\n        for i, text in enumerate(texts):\n            _id = str(uuid.uuid4())\n            request = {\n                \"_op_type\": \"index\",\n                \"_index\": self.index_name,\n                \"content\": text,\n                \"_id\": _id,\n            }\n            ids.append(_id)\n            requests.append(request)\n        bulk(self.client, requests)\n        if refresh_indices:\n            self.client.indices.refresh(index=self.index_name)\n        return ids\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        query_dict = {\"query\": {\"match\": {\"content\": query}}}\n        res = self.client.search(index=self.index_name, body=query_dict)\n        docs = []\n        for r in res[\"hits\"][\"hits\"]:\n            docs.append(Document(page_content=r[\"_source\"][\"content\"]))\n        return docs\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/elastic_search_bm25.html"}184{"id": "2254e056aa07-0", "text": "Source code for langchain.retrievers.knn\n\"\"\"KNN Retriever.\nLargely based on\nhttps://github.com/karpathy/randomfun/blob/master/knn_vs_svm.ipynb\"\"\"\nfrom __future__ import annotations\nimport concurrent.futures\nfrom typing import Any, List, Optional\nimport numpy as np\nfrom pydantic import BaseModel\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.schema import BaseRetriever, Document\ndef create_index(contexts: List[str], embeddings: Embeddings) -> np.ndarray:\n    with concurrent.futures.ThreadPoolExecutor() as executor:\n        return np.array(list(executor.map(embeddings.embed_query, contexts)))\n[docs]class KNNRetriever(BaseRetriever, BaseModel):\n    embeddings: Embeddings\n    index: Any\n    texts: List[str]\n    k: int = 4\n    relevancy_threshold: Optional[float] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    @classmethod\n    def from_texts(\n        cls, texts: List[str], embeddings: Embeddings, **kwargs: Any\n    ) -> KNNRetriever:\n        index = create_index(texts, embeddings)\n        return cls(embeddings=embeddings, index=index, texts=texts, **kwargs)\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        query_embeds = np.array(self.embeddings.embed_query(query))\n        # calc L2 norm\n        index_embeds = self.index / np.sqrt((self.index**2).sum(1, keepdims=True))\n        query_embeds = query_embeds / np.sqrt((query_embeds**2).sum())\n        similarities = index_embeds.dot(query_embeds)", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/knn.html"}185{"id": "2254e056aa07-1", "text": "similarities = index_embeds.dot(query_embeds)\n        sorted_ix = np.argsort(-similarities)\n        denominator = np.max(similarities) - np.min(similarities) + 1e-6\n        normalized_similarities = (similarities - np.min(similarities)) / denominator\n        top_k_results = []\n        for row in sorted_ix[0 : self.k]:\n            if (\n                self.relevancy_threshold is None\n                or normalized_similarities[row] >= self.relevancy_threshold\n            ):\n                top_k_results.append(Document(page_content=self.texts[row]))\n        return top_k_results\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/knn.html"}186{"id": "83e72c2224a6-0", "text": "Source code for langchain.retrievers.remote_retriever\nfrom typing import List, Optional\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel\nfrom langchain.schema import BaseRetriever, Document\n[docs]class RemoteLangChainRetriever(BaseRetriever, BaseModel):\n    url: str\n    headers: Optional[dict] = None\n    input_key: str = \"message\"\n    response_key: str = \"response\"\n    page_content_key: str = \"page_content\"\n    metadata_key: str = \"metadata\"\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        response = requests.post(\n            self.url, json={self.input_key: query}, headers=self.headers\n        )\n        result = response.json()\n        return [\n            Document(\n                page_content=r[self.page_content_key], metadata=r[self.metadata_key]\n            )\n            for r in result[self.response_key]\n        ]\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        async with aiohttp.ClientSession() as session:\n            async with session.request(\n                \"POST\", self.url, headers=self.headers, json={self.input_key: query}\n            ) as response:\n                result = await response.json()\n        return [\n            Document(\n                page_content=r[self.page_content_key], metadata=r[self.metadata_key]\n            )\n            for r in result[self.response_key]\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/remote_retriever.html"}187{"id": "4444fd9c5540-0", "text": "Source code for langchain.retrievers.zep\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, List, Optional\nfrom langchain.schema import BaseRetriever, Document\nif TYPE_CHECKING:\n    from zep_python import SearchResult\n[docs]class ZepRetriever(BaseRetriever):\n    \"\"\"A Retriever implementation for the Zep long-term memory store. Search your\n    user's long-term chat history with Zep.\n    Note: You will need to provide the user's `session_id` to use this retriever.\n    More on Zep:\n    Zep provides long-term conversation storage for LLM apps. The server stores,\n    summarizes, embeds, indexes, and enriches conversational AI chat\n    histories, and exposes them via simple, low-latency APIs.\n    For server installation instructions, see:\n    https://getzep.github.io/deployment/quickstart/\n    \"\"\"\n    def __init__(\n        self,\n        session_id: str,\n        url: str,\n        top_k: Optional[int] = None,\n    ):\n        try:\n            from zep_python import ZepClient\n        except ImportError:\n            raise ValueError(\n                \"Could not import zep-python package. \"\n                \"Please install it with `pip install zep-python`.\"\n            )\n        self.zep_client = ZepClient(base_url=url)\n        self.session_id = session_id\n        self.top_k = top_k\n    def _search_result_to_doc(self, results: List[SearchResult]) -> List[Document]:\n        return [\n            Document(\n                page_content=r.message.pop(\"content\"),\n                metadata={\"score\": r.dist, **r.message},\n            )\n            for r in results\n            if r.message\n        ]", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/zep.html"}188{"id": "4444fd9c5540-1", "text": ")\n            for r in results\n            if r.message\n        ]\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        from zep_python import SearchPayload\n        payload: SearchPayload = SearchPayload(text=query)\n        results: List[SearchResult] = self.zep_client.search_memory(\n            self.session_id, payload, limit=self.top_k\n        )\n        return self._search_result_to_doc(results)\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        from zep_python import SearchPayload\n        payload: SearchPayload = SearchPayload(text=query)\n        results: List[SearchResult] = await self.zep_client.asearch_memory(\n            self.session_id, payload, limit=self.top_k\n        )\n        return self._search_result_to_doc(results)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/zep.html"}189{"id": "85ee7db33c36-0", "text": "Source code for langchain.retrievers.contextual_compression\n\"\"\"Retriever that wraps a base retriever and filters the results.\"\"\"\nfrom typing import List\nfrom pydantic import BaseModel, Extra\nfrom langchain.retrievers.document_compressors.base import (\n    BaseDocumentCompressor,\n)\nfrom langchain.schema import BaseRetriever, Document\n[docs]class ContextualCompressionRetriever(BaseRetriever, BaseModel):\n    \"\"\"Retriever that wraps a base retriever and compresses the results.\"\"\"\n    base_compressor: BaseDocumentCompressor\n    \"\"\"Compressor for compressing retrieved documents.\"\"\"\n    base_retriever: BaseRetriever\n    \"\"\"Base Retriever to use for getting relevant documents.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        \"\"\"Get documents relevant for a query.\n        Args:\n            query: string to find relevant documents for\n        Returns:\n            Sequence of relevant documents\n        \"\"\"\n        docs = self.base_retriever.get_relevant_documents(query)\n        compressed_docs = self.base_compressor.compress_documents(docs, query)\n        return list(compressed_docs)\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        \"\"\"Get documents relevant for a query.\n        Args:\n            query: string to find relevant documents for\n        Returns:\n            List of relevant documents\n        \"\"\"\n        docs = await self.base_retriever.aget_relevant_documents(query)\n        compressed_docs = await self.base_compressor.acompress_documents(docs, query)\n        return list(compressed_docs)\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/contextual_compression.html"}190{"id": "85ee7db33c36-1", "text": "return list(compressed_docs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/contextual_compression.html"}191{"id": "9044fcb1e0d8-0", "text": "Source code for langchain.retrievers.weaviate_hybrid_search\n\"\"\"Wrapper around weaviate vector database.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, Dict, List, Optional\nfrom uuid import uuid4\nfrom pydantic import Extra\nfrom langchain.docstore.document import Document\nfrom langchain.schema import BaseRetriever\n[docs]class WeaviateHybridSearchRetriever(BaseRetriever):\n    def __init__(\n        self,\n        client: Any,\n        index_name: str,\n        text_key: str,\n        alpha: float = 0.5,\n        k: int = 4,\n        attributes: Optional[List[str]] = None,\n        create_schema_if_missing: bool = True,\n    ):\n        try:\n            import weaviate\n        except ImportError:\n            raise ImportError(\n                \"Could not import weaviate python package. \"\n                \"Please install it with `pip install weaviate-client`.\"\n            )\n        if not isinstance(client, weaviate.Client):\n            raise ValueError(\n                f\"client should be an instance of weaviate.Client, got {type(client)}\"\n            )\n        self._client = client\n        self.k = k\n        self.alpha = alpha\n        self._index_name = index_name\n        self._text_key = text_key\n        self._query_attrs = [self._text_key]\n        if attributes is not None:\n            self._query_attrs.extend(attributes)\n        if create_schema_if_missing:\n            self._create_schema_if_missing()\n    def _create_schema_if_missing(self) -> None:\n        class_obj = {\n            \"class\": self._index_name,\n            \"properties\": [{\"name\": self._text_key, \"dataType\": [\"text\"]}],", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/weaviate_hybrid_search.html"}192{"id": "9044fcb1e0d8-1", "text": "\"properties\": [{\"name\": self._text_key, \"dataType\": [\"text\"]}],\n            \"vectorizer\": \"text2vec-openai\",\n        }\n        if not self._client.schema.exists(self._index_name):\n            self._client.schema.create_class(class_obj)\n[docs]    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    # added text_key\n[docs]    def add_documents(self, docs: List[Document], **kwargs: Any) -> List[str]:\n        \"\"\"Upload documents to Weaviate.\"\"\"\n        from weaviate.util import get_valid_uuid\n        with self._client.batch as batch:\n            ids = []\n            for i, doc in enumerate(docs):\n                metadata = doc.metadata or {}\n                data_properties = {self._text_key: doc.page_content, **metadata}\n                # If the UUID of one of the objects already exists\n                # then the existing objectwill be replaced by the new object.\n                if \"uuids\" in kwargs:\n                    _id = kwargs[\"uuids\"][i]\n                else:\n                    _id = get_valid_uuid(uuid4())\n                batch.add_data_object(data_properties, self._index_name, _id)\n                ids.append(_id)\n        return ids\n[docs]    def get_relevant_documents(\n        self, query: str, where_filter: Optional[Dict[str, object]] = None\n    ) -> List[Document]:\n        \"\"\"Look up similar documents in Weaviate.\"\"\"\n        query_obj = self._client.query.get(self._index_name, self._query_attrs)\n        if where_filter:\n            query_obj = query_obj.with_where(where_filter)\n        result = query_obj.with_hybrid(query, alpha=self.alpha).with_limit(self.k).do()", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/weaviate_hybrid_search.html"}193{"id": "9044fcb1e0d8-2", "text": "if \"errors\" in result:\n            raise ValueError(f\"Error during query: {result['errors']}\")\n        docs = []\n        for res in result[\"data\"][\"Get\"][self._index_name]:\n            text = res.pop(self._text_key)\n            docs.append(Document(page_content=text, metadata=res))\n        return docs\n[docs]    async def aget_relevant_documents(\n        self, query: str, where_filter: Optional[Dict[str, object]] = None\n    ) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/weaviate_hybrid_search.html"}194{"id": "5a232a2ccbde-0", "text": "Source code for langchain.retrievers.databerry\nfrom typing import List, Optional\nimport aiohttp\nimport requests\nfrom langchain.schema import BaseRetriever, Document\n[docs]class DataberryRetriever(BaseRetriever):\n    datastore_url: str\n    top_k: Optional[int]\n    api_key: Optional[str]\n    def __init__(\n        self,\n        datastore_url: str,\n        top_k: Optional[int] = None,\n        api_key: Optional[str] = None,\n    ):\n        self.datastore_url = datastore_url\n        self.api_key = api_key\n        self.top_k = top_k\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        response = requests.post(\n            self.datastore_url,\n            json={\n                \"query\": query,\n                **({\"topK\": self.top_k} if self.top_k is not None else {}),\n            },\n            headers={\n                \"Content-Type\": \"application/json\",\n                **(\n                    {\"Authorization\": f\"Bearer {self.api_key}\"}\n                    if self.api_key is not None\n                    else {}\n                ),\n            },\n        )\n        data = response.json()\n        return [\n            Document(\n                page_content=r[\"text\"],\n                metadata={\"source\": r[\"source\"], \"score\": r[\"score\"]},\n            )\n            for r in data[\"results\"]\n        ]\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        async with aiohttp.ClientSession() as session:\n            async with session.request(\n                \"POST\",\n                self.datastore_url,\n                json={\n                    \"query\": query,", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/databerry.html"}195{"id": "5a232a2ccbde-1", "text": "self.datastore_url,\n                json={\n                    \"query\": query,\n                    **({\"topK\": self.top_k} if self.top_k is not None else {}),\n                },\n                headers={\n                    \"Content-Type\": \"application/json\",\n                    **(\n                        {\"Authorization\": f\"Bearer {self.api_key}\"}\n                        if self.api_key is not None\n                        else {}\n                    ),\n                },\n            ) as response:\n                data = await response.json()\n        return [\n            Document(\n                page_content=r[\"text\"],\n                metadata={\"source\": r[\"source\"], \"score\": r[\"score\"]},\n            )\n            for r in data[\"results\"]\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/databerry.html"}196{"id": "d0874245e600-0", "text": "Source code for langchain.retrievers.chatgpt_plugin_retriever\nfrom __future__ import annotations\nfrom typing import List, Optional\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel\nfrom langchain.schema import BaseRetriever, Document\n[docs]class ChatGPTPluginRetriever(BaseRetriever, BaseModel):\n    url: str\n    bearer_token: str\n    top_k: int = 3\n    filter: Optional[dict] = None\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        url, json, headers = self._create_request(query)\n        response = requests.post(url, json=json, headers=headers)\n        results = response.json()[\"results\"][0][\"results\"]\n        docs = []\n        for d in results:\n            content = d.pop(\"text\")\n            docs.append(Document(page_content=content, metadata=d))\n        return docs\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        url, json, headers = self._create_request(query)\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.post(url, headers=headers, json=json) as response:\n                    res = await response.json()\n        else:\n            async with self.aiosession.post(\n                url, headers=headers, json=json\n            ) as response:\n                res = await response.json()\n        results = res[\"results\"][0][\"results\"]\n        docs = []\n        for d in results:\n            content = d.pop(\"text\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/chatgpt_plugin_retriever.html"}197{"id": "d0874245e600-1", "text": "docs = []\n        for d in results:\n            content = d.pop(\"text\")\n            docs.append(Document(page_content=content, metadata=d))\n        return docs\n    def _create_request(self, query: str) -> tuple[str, dict, dict]:\n        url = f\"{self.url}/query\"\n        json = {\n            \"queries\": [\n                {\n                    \"query\": query,\n                    \"filter\": self.filter,\n                    \"top_k\": self.top_k,\n                }\n            ]\n        }\n        headers = {\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {self.bearer_token}\",\n        }\n        return url, json, headers\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/chatgpt_plugin_retriever.html"}198{"id": "ab341c603baf-0", "text": "Source code for langchain.retrievers.arxiv\nfrom typing import List\nfrom langchain.schema import BaseRetriever, Document\nfrom langchain.utilities.arxiv import ArxivAPIWrapper\n[docs]class ArxivRetriever(BaseRetriever, ArxivAPIWrapper):\n    \"\"\"\n    It is effectively a wrapper for ArxivAPIWrapper.\n    It wraps load() to get_relevant_documents().\n    It uses all ArxivAPIWrapper arguments without any change.\n    \"\"\"\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        return self.load(query=query)\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/arxiv.html"}199{"id": "cfe06c149ede-0", "text": "Source code for langchain.retrievers.metal\nfrom typing import Any, List, Optional\nfrom langchain.schema import BaseRetriever, Document\n[docs]class MetalRetriever(BaseRetriever):\n    def __init__(self, client: Any, params: Optional[dict] = None):\n        from metal_sdk.metal import Metal\n        if not isinstance(client, Metal):\n            raise ValueError(\n                \"Got unexpected client, should be of type metal_sdk.metal.Metal. \"\n                f\"Instead, got {type(client)}\"\n            )\n        self.client: Metal = client\n        self.params = params or {}\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        results = self.client.search({\"text\": query}, **self.params)\n        final_results = []\n        for r in results[\"data\"]:\n            metadata = {k: v for k, v in r.items() if k != \"text\"}\n            final_results.append(Document(page_content=r[\"text\"], metadata=metadata))\n        return final_results\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/metal.html"}200{"id": "d3af39275c7b-0", "text": "Source code for langchain.retrievers.document_compressors.embeddings_filter\n\"\"\"Document compressor that uses embeddings to drop documents unrelated to the query.\"\"\"\nfrom typing import Callable, Dict, Optional, Sequence\nimport numpy as np\nfrom pydantic import root_validator\nfrom langchain.document_transformers import (\n    _get_embeddings_from_stateful_docs,\n    get_stateful_documents,\n)\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.math_utils import cosine_similarity\nfrom langchain.retrievers.document_compressors.base import (\n    BaseDocumentCompressor,\n)\nfrom langchain.schema import Document\n[docs]class EmbeddingsFilter(BaseDocumentCompressor):\n    embeddings: Embeddings\n    \"\"\"Embeddings to use for embedding document contents and queries.\"\"\"\n    similarity_fn: Callable = cosine_similarity\n    \"\"\"Similarity function for comparing documents. Function expected to take as input\n    two matrices (List[List[float]]) and return a matrix of scores where higher values\n    indicate greater similarity.\"\"\"\n    k: Optional[int] = 20\n    \"\"\"The number of relevant documents to return. Can be set to None, in which case\n    `similarity_threshold` must be specified. Defaults to 20.\"\"\"\n    similarity_threshold: Optional[float]\n    \"\"\"Threshold for determining when two documents are similar enough\n    to be considered redundant. Defaults to None, must be specified if `k` is set\n    to None.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_params(cls, values: Dict) -> Dict:\n        \"\"\"Validate similarity parameters.\"\"\"\n        if values[\"k\"] is None and values[\"similarity_threshold\"] is None:\n            raise ValueError(\"Must specify one of `k` or `similarity_threshold`.\")\n        return values", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/embeddings_filter.html"}201{"id": "d3af39275c7b-1", "text": "return values\n[docs]    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Filter documents based on similarity of their embeddings to the query.\"\"\"\n        stateful_documents = get_stateful_documents(documents)\n        embedded_documents = _get_embeddings_from_stateful_docs(\n            self.embeddings, stateful_documents\n        )\n        embedded_query = self.embeddings.embed_query(query)\n        similarity = self.similarity_fn([embedded_query], embedded_documents)[0]\n        included_idxs = np.arange(len(embedded_documents))\n        if self.k is not None:\n            included_idxs = np.argsort(similarity)[::-1][: self.k]\n        if self.similarity_threshold is not None:\n            similar_enough = np.where(\n                similarity[included_idxs] > self.similarity_threshold\n            )\n            included_idxs = included_idxs[similar_enough]\n        return [stateful_documents[i] for i in included_idxs]\n[docs]    async def acompress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Filter down documents.\"\"\"\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/embeddings_filter.html"}202{"id": "109f1ead7d38-0", "text": "Source code for langchain.retrievers.document_compressors.base\n\"\"\"Interface for retrieved document compressors.\"\"\"\nfrom abc import ABC, abstractmethod\nfrom typing import List, Sequence, Union\nfrom pydantic import BaseModel\nfrom langchain.schema import BaseDocumentTransformer, Document\nclass BaseDocumentCompressor(BaseModel, ABC):\n    \"\"\"Base abstraction interface for document compression.\"\"\"\n    @abstractmethod\n    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Compress retrieved documents given the query context.\"\"\"\n    @abstractmethod\n    async def acompress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Compress retrieved documents given the query context.\"\"\"\n[docs]class DocumentCompressorPipeline(BaseDocumentCompressor):\n    \"\"\"Document compressor that uses a pipeline of transformers.\"\"\"\n    transformers: List[Union[BaseDocumentTransformer, BaseDocumentCompressor]]\n    \"\"\"List of document filters that are chained together and run in sequence.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Transform a list of documents.\"\"\"\n        for _transformer in self.transformers:\n            if isinstance(_transformer, BaseDocumentCompressor):\n                documents = _transformer.compress_documents(documents, query)\n            elif isinstance(_transformer, BaseDocumentTransformer):\n                documents = _transformer.transform_documents(documents)\n            else:\n                raise ValueError(f\"Got unexpected transformer type: {_transformer}\")\n        return documents\n[docs]    async def acompress_documents(\n        self, documents: Sequence[Document], query: str", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/base.html"}203{"id": "109f1ead7d38-1", "text": "self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Compress retrieved documents given the query context.\"\"\"\n        for _transformer in self.transformers:\n            if isinstance(_transformer, BaseDocumentCompressor):\n                documents = await _transformer.acompress_documents(documents, query)\n            elif isinstance(_transformer, BaseDocumentTransformer):\n                documents = await _transformer.atransform_documents(documents)\n            else:\n                raise ValueError(f\"Got unexpected transformer type: {_transformer}\")\n        return documents\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/base.html"}204{"id": "198471a5c9ae-0", "text": "Source code for langchain.retrievers.document_compressors.chain_filter\n\"\"\"Filter that uses an LLM to drop documents that aren't relevant to the query.\"\"\"\nfrom typing import Any, Callable, Dict, Optional, Sequence\nfrom langchain import BasePromptTemplate, LLMChain, PromptTemplate\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.output_parsers.boolean import BooleanOutputParser\nfrom langchain.retrievers.document_compressors.base import BaseDocumentCompressor\nfrom langchain.retrievers.document_compressors.chain_filter_prompt import (\n    prompt_template,\n)\nfrom langchain.schema import Document\ndef _get_default_chain_prompt() -> PromptTemplate:\n    return PromptTemplate(\n        template=prompt_template,\n        input_variables=[\"question\", \"context\"],\n        output_parser=BooleanOutputParser(),\n    )\ndef default_get_input(query: str, doc: Document) -> Dict[str, Any]:\n    \"\"\"Return the compression chain input.\"\"\"\n    return {\"question\": query, \"context\": doc.page_content}\n[docs]class LLMChainFilter(BaseDocumentCompressor):\n    \"\"\"Filter that drops documents that aren't relevant to the query.\"\"\"\n    llm_chain: LLMChain\n    \"\"\"LLM wrapper to use for filtering documents. \n    The chain prompt is expected to have a BooleanOutputParser.\"\"\"\n    get_input: Callable[[str, Document], dict] = default_get_input\n    \"\"\"Callable for constructing the chain input from the query and a Document.\"\"\"\n[docs]    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Filter down documents based on their relevance to the query.\"\"\"\n        filtered_docs = []\n        for doc in documents:\n            _input = self.get_input(query, doc)\n            include_doc = self.llm_chain.predict_and_parse(**_input)", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/chain_filter.html"}205{"id": "198471a5c9ae-1", "text": "include_doc = self.llm_chain.predict_and_parse(**_input)\n            if include_doc:\n                filtered_docs.append(doc)\n        return filtered_docs\n[docs]    async def acompress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Filter down documents.\"\"\"\n        raise NotImplementedError\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        prompt: Optional[BasePromptTemplate] = None,\n        **kwargs: Any\n    ) -> \"LLMChainFilter\":\n        _prompt = prompt if prompt is not None else _get_default_chain_prompt()\n        llm_chain = LLMChain(llm=llm, prompt=_prompt)\n        return cls(llm_chain=llm_chain, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/chain_filter.html"}206{"id": "5ce3e16a2d9d-0", "text": "Source code for langchain.retrievers.document_compressors.cohere_rerank\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, Dict, Sequence\nfrom pydantic import Extra, root_validator\nfrom langchain.retrievers.document_compressors.base import BaseDocumentCompressor\nfrom langchain.schema import Document\nfrom langchain.utils import get_from_dict_or_env\nif TYPE_CHECKING:\n    from cohere import Client\nelse:\n    # We do to avoid pydantic annotation issues when actually instantiating\n    # while keeping this import optional\n    try:\n        from cohere import Client\n    except ImportError:\n        pass\n[docs]class CohereRerank(BaseDocumentCompressor):\n    client: Client\n    top_n: int = 3\n    model: str = \"rerank-english-v2.0\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        cohere_api_key = get_from_dict_or_env(\n            values, \"cohere_api_key\", \"COHERE_API_KEY\"\n        )\n        try:\n            import cohere\n            values[\"client\"] = cohere.Client(cohere_api_key)\n        except ImportError:\n            raise ImportError(\n                \"Could not import cohere python package. \"\n                \"Please install it with `pip install cohere`.\"\n            )\n        return values\n[docs]    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        if len(documents) == 0:  # to avoid empty api call\n            return []", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/cohere_rerank.html"}207{"id": "5ce3e16a2d9d-1", "text": "return []\n        doc_list = list(documents)\n        _docs = [d.page_content for d in doc_list]\n        results = self.client.rerank(\n            model=self.model, query=query, documents=_docs, top_n=self.top_n\n        )\n        final_results = []\n        for r in results:\n            doc = doc_list[r.index]\n            doc.metadata[\"relevance_score\"] = r.relevance_score\n            final_results.append(doc)\n        return final_results\n[docs]    async def acompress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/cohere_rerank.html"}208{"id": "1174c1843903-0", "text": "Source code for langchain.retrievers.document_compressors.chain_extract\n\"\"\"DocumentFilter that uses an LLM chain to extract the relevant parts of documents.\"\"\"\nfrom __future__ import annotations\nimport asyncio\nfrom typing import Any, Callable, Dict, Optional, Sequence\nfrom langchain import LLMChain, PromptTemplate\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.retrievers.document_compressors.base import BaseDocumentCompressor\nfrom langchain.retrievers.document_compressors.chain_extract_prompt import (\n    prompt_template,\n)\nfrom langchain.schema import BaseOutputParser, Document\ndef default_get_input(query: str, doc: Document) -> Dict[str, Any]:\n    \"\"\"Return the compression chain input.\"\"\"\n    return {\"question\": query, \"context\": doc.page_content}\nclass NoOutputParser(BaseOutputParser[str]):\n    \"\"\"Parse outputs that could return a null string of some sort.\"\"\"\n    no_output_str: str = \"NO_OUTPUT\"\n    def parse(self, text: str) -> str:\n        cleaned_text = text.strip()\n        if cleaned_text == self.no_output_str:\n            return \"\"\n        return cleaned_text\ndef _get_default_chain_prompt() -> PromptTemplate:\n    output_parser = NoOutputParser()\n    template = prompt_template.format(no_output_str=output_parser.no_output_str)\n    return PromptTemplate(\n        template=template,\n        input_variables=[\"question\", \"context\"],\n        output_parser=output_parser,\n    )\n[docs]class LLMChainExtractor(BaseDocumentCompressor):\n    llm_chain: LLMChain\n    \"\"\"LLM wrapper to use for compressing documents.\"\"\"\n    get_input: Callable[[str, Document], dict] = default_get_input\n    \"\"\"Callable for constructing the chain input from the query and a Document.\"\"\"\n[docs]    def compress_documents(", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/chain_extract.html"}209{"id": "1174c1843903-1", "text": "[docs]    def compress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Compress page content of raw documents.\"\"\"\n        compressed_docs = []\n        for doc in documents:\n            _input = self.get_input(query, doc)\n            output = self.llm_chain.predict_and_parse(**_input)\n            if len(output) == 0:\n                continue\n            compressed_docs.append(Document(page_content=output, metadata=doc.metadata))\n        return compressed_docs\n[docs]    async def acompress_documents(\n        self, documents: Sequence[Document], query: str\n    ) -> Sequence[Document]:\n        \"\"\"Compress page content of raw documents asynchronously.\"\"\"\n        outputs = await asyncio.gather(\n            *[\n                self.llm_chain.apredict_and_parse(**self.get_input(query, doc))\n                for doc in documents\n            ]\n        )\n        compressed_docs = []\n        for i, doc in enumerate(documents):\n            if len(outputs[i]) == 0:\n                continue\n            compressed_docs.append(\n                Document(page_content=outputs[i], metadata=doc.metadata)\n            )\n        return compressed_docs\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        prompt: Optional[PromptTemplate] = None,\n        get_input: Optional[Callable[[str, Document], str]] = None,\n        llm_chain_kwargs: Optional[dict] = None,\n    ) -> LLMChainExtractor:\n        \"\"\"Initialize from LLM.\"\"\"\n        _prompt = prompt if prompt is not None else _get_default_chain_prompt()\n        _get_input = get_input if get_input is not None else default_get_input", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/chain_extract.html"}210{"id": "1174c1843903-2", "text": "_get_input = get_input if get_input is not None else default_get_input\n        llm_chain = LLMChain(llm=llm, prompt=_prompt, **(llm_chain_kwargs or {}))\n        return cls(llm_chain=llm_chain, get_input=_get_input)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/document_compressors/chain_extract.html"}211{"id": "7fdcf1b946e8-0", "text": "Source code for langchain.retrievers.self_query.base\n\"\"\"Retriever that generates and executes structured queries over its own data source.\"\"\"\nfrom typing import Any, Dict, List, Optional, Type, cast\nfrom pydantic import BaseModel, Field, root_validator\nfrom langchain import LLMChain\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.query_constructor.base import load_query_constructor_chain\nfrom langchain.chains.query_constructor.ir import StructuredQuery, Visitor\nfrom langchain.chains.query_constructor.schema import AttributeInfo\nfrom langchain.retrievers.self_query.chroma import ChromaTranslator\nfrom langchain.retrievers.self_query.pinecone import PineconeTranslator\nfrom langchain.retrievers.self_query.weaviate import WeaviateTranslator\nfrom langchain.schema import BaseRetriever, Document\nfrom langchain.vectorstores import Chroma, Pinecone, VectorStore, Weaviate\ndef _get_builtin_translator(vectorstore_cls: Type[VectorStore]) -> Visitor:\n    \"\"\"Get the translator class corresponding to the vector store class.\"\"\"\n    BUILTIN_TRANSLATORS: Dict[Type[VectorStore], Type[Visitor]] = {\n        Pinecone: PineconeTranslator,\n        Chroma: ChromaTranslator,\n        Weaviate: WeaviateTranslator,\n    }\n    if vectorstore_cls not in BUILTIN_TRANSLATORS:\n        raise ValueError(\n            f\"Self query retriever with Vector Store type {vectorstore_cls}\"\n            f\" not supported.\"\n        )\n    return BUILTIN_TRANSLATORS[vectorstore_cls]()\n[docs]class SelfQueryRetriever(BaseRetriever, BaseModel):\n    \"\"\"Retriever that wraps around a vector store and uses an LLM to generate\n    the vector store queries.\"\"\"\n    vectorstore: VectorStore\n    \"\"\"The underlying vector store from which documents will be retrieved.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/self_query/base.html"}212{"id": "7fdcf1b946e8-1", "text": "vectorstore: VectorStore\n    \"\"\"The underlying vector store from which documents will be retrieved.\"\"\"\n    llm_chain: LLMChain\n    \"\"\"The LLMChain for generating the vector store queries.\"\"\"\n    search_type: str = \"similarity\"\n    \"\"\"The search type to perform on the vector store.\"\"\"\n    search_kwargs: dict = Field(default_factory=dict)\n    \"\"\"Keyword arguments to pass in to the vector store search.\"\"\"\n    structured_query_translator: Visitor\n    \"\"\"Translator for turning internal query language into vectorstore search params.\"\"\"\n    verbose: bool = False\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @root_validator(pre=True)\n    def validate_translator(cls, values: Dict) -> Dict:\n        \"\"\"Validate translator.\"\"\"\n        if \"structured_query_translator\" not in values:\n            vectorstore_cls = values[\"vectorstore\"].__class__\n            values[\"structured_query_translator\"] = _get_builtin_translator(\n                vectorstore_cls\n            )\n        return values\n[docs]    def get_relevant_documents(self, query: str) -> List[Document]:\n        \"\"\"Get documents relevant for a query.\n        Args:\n            query: string to find relevant documents for\n        Returns:\n            List of relevant documents\n        \"\"\"\n        inputs = self.llm_chain.prep_inputs({\"query\": query})\n        structured_query = cast(\n            StructuredQuery, self.llm_chain.predict_and_parse(callbacks=None, **inputs)\n        )\n        if self.verbose:\n            print(structured_query)\n        new_query, new_kwargs = self.structured_query_translator.visit_structured_query(\n            structured_query\n        )\n        if structured_query.limit is not None:\n            new_kwargs[\"k\"] = structured_query.limit", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/self_query/base.html"}213{"id": "7fdcf1b946e8-2", "text": "if structured_query.limit is not None:\n            new_kwargs[\"k\"] = structured_query.limit\n        search_kwargs = {**self.search_kwargs, **new_kwargs}\n        docs = self.vectorstore.search(new_query, self.search_type, **search_kwargs)\n        return docs\n[docs]    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        vectorstore: VectorStore,\n        document_contents: str,\n        metadata_field_info: List[AttributeInfo],\n        structured_query_translator: Optional[Visitor] = None,\n        chain_kwargs: Optional[Dict] = None,\n        enable_limit: bool = False,\n        **kwargs: Any,\n    ) -> \"SelfQueryRetriever\":\n        if structured_query_translator is None:\n            structured_query_translator = _get_builtin_translator(vectorstore.__class__)\n        chain_kwargs = chain_kwargs or {}\n        if \"allowed_comparators\" not in chain_kwargs:\n            chain_kwargs[\n                \"allowed_comparators\"\n            ] = structured_query_translator.allowed_comparators\n        if \"allowed_operators\" not in chain_kwargs:\n            chain_kwargs[\n                \"allowed_operators\"\n            ] = structured_query_translator.allowed_operators\n        llm_chain = load_query_constructor_chain(\n            llm,\n            document_contents,\n            metadata_field_info,\n            enable_limit=enable_limit,\n            **chain_kwargs,\n        )\n        return cls(\n            llm_chain=llm_chain,\n            vectorstore=vectorstore,\n            structured_query_translator=structured_query_translator,\n            **kwargs,\n        )\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/self_query/base.html"}214{"id": "7fdcf1b946e8-3", "text": "**kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/retrievers/self_query/base.html"}215{"id": "f7fac1d6b36c-0", "text": "Source code for langchain.tools.base\n\"\"\"Base implementation for tools or skills.\"\"\"\nfrom __future__ import annotations\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom inspect import signature\nfrom typing import Any, Awaitable, Callable, Dict, Optional, Tuple, Type, Union\nfrom pydantic import (\n    BaseModel,\n    Extra,\n    Field,\n    create_model,\n    root_validator,\n    validate_arguments,\n)\nfrom pydantic.main import ModelMetaclass\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManager,\n    AsyncCallbackManagerForToolRun,\n    CallbackManager,\n    CallbackManagerForToolRun,\n    Callbacks,\n)\nclass SchemaAnnotationError(TypeError):\n    \"\"\"Raised when 'args_schema' is missing or has an incorrect type annotation.\"\"\"\nclass ToolMetaclass(ModelMetaclass):\n    \"\"\"Metaclass for BaseTool to ensure the provided args_schema\n    doesn't silently ignored.\"\"\"\n    def __new__(\n        cls: Type[ToolMetaclass], name: str, bases: Tuple[Type, ...], dct: dict\n    ) -> ToolMetaclass:\n        \"\"\"Create the definition of the new tool class.\"\"\"\n        schema_type: Optional[Type[BaseModel]] = dct.get(\"args_schema\")\n        if schema_type is not None:\n            schema_annotations = dct.get(\"__annotations__\", {})\n            args_schema_type = schema_annotations.get(\"args_schema\", None)\n            if args_schema_type is None or args_schema_type == BaseModel:\n                # Throw errors for common mis-annotations.\n                # TODO: Use get_args / get_origin and fully\n                # specify valid annotations.\n                typehint_mandate = \"\"\"\nclass ChildTool(BaseTool):\n    ...\n    args_schema: Type[BaseModel] = SchemaClass\n    ...\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}216{"id": "f7fac1d6b36c-1", "text": "...\n    args_schema: Type[BaseModel] = SchemaClass\n    ...\"\"\"\n                raise SchemaAnnotationError(\n                    f\"Tool definition for {name} must include valid type annotations\"\n                    f\" for argument 'args_schema' to behave as expected.\\n\"\n                    f\"Expected annotation of 'Type[BaseModel]'\"\n                    f\" but got '{args_schema_type}'.\\n\"\n                    f\"Expected class looks like:\\n\"\n                    f\"{typehint_mandate}\"\n                )\n        # Pass through to Pydantic's metaclass\n        return super().__new__(cls, name, bases, dct)\ndef _create_subset_model(\n    name: str, model: BaseModel, field_names: list\n) -> Type[BaseModel]:\n    \"\"\"Create a pydantic model with only a subset of model's fields.\"\"\"\n    fields = {\n        field_name: (\n            model.__fields__[field_name].type_,\n            model.__fields__[field_name].default,\n        )\n        for field_name in field_names\n        if field_name in model.__fields__\n    }\n    return create_model(name, **fields)  # type: ignore\ndef get_filtered_args(\n    inferred_model: Type[BaseModel],\n    func: Callable,\n) -> dict:\n    \"\"\"Get the arguments from a function's signature.\"\"\"\n    schema = inferred_model.schema()[\"properties\"]\n    valid_keys = signature(func).parameters\n    return {k: schema[k] for k in valid_keys if k != \"run_manager\"}\nclass _SchemaConfig:\n    \"\"\"Configuration for the pydantic model.\"\"\"\n    extra = Extra.forbid\n    arbitrary_types_allowed = True\ndef create_schema_from_function(\n    model_name: str,\n    func: Callable,\n) -> Type[BaseModel]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}217{"id": "f7fac1d6b36c-2", "text": "model_name: str,\n    func: Callable,\n) -> Type[BaseModel]:\n    \"\"\"Create a pydantic schema from a function's signature.\"\"\"\n    validated = validate_arguments(func, config=_SchemaConfig)  # type: ignore\n    inferred_model = validated.model  # type: ignore\n    if \"run_manager\" in inferred_model.__fields__:\n        del inferred_model.__fields__[\"run_manager\"]\n    # Pydantic adds placeholder virtual fields we need to strip\n    filtered_args = get_filtered_args(inferred_model, func)\n    return _create_subset_model(\n        f\"{model_name}Schema\", inferred_model, list(filtered_args)\n    )\n[docs]class BaseTool(ABC, BaseModel, metaclass=ToolMetaclass):\n    \"\"\"Interface LangChain tools must implement.\"\"\"\n    name: str\n    \"\"\"The unique name of the tool that clearly communicates its purpose.\"\"\"\n    description: str\n    \"\"\"Used to tell the model how/when/why to use the tool.\n    \n    You can provide few-shot examples as a part of the description.\n    \"\"\"\n    args_schema: Optional[Type[BaseModel]] = None\n    \"\"\"Pydantic model class to validate and parse the tool's input arguments.\"\"\"\n    return_direct: bool = False\n    \"\"\"Whether to return the tool's output directly. Setting this to True means\n    \n    that after the tool is called, the AgentExecutor will stop looping.\n    \"\"\"\n    verbose: bool = False\n    \"\"\"Whether to log the tool's progress.\"\"\"\n    callbacks: Callbacks = Field(default=None, exclude=True)\n    \"\"\"Callbacks to be called during tool execution.\"\"\"\n    callback_manager: Optional[BaseCallbackManager] = Field(default=None, exclude=True)\n    \"\"\"Deprecated. Please use callbacks instead.\"\"\"\n    class Config:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}218{"id": "f7fac1d6b36c-3", "text": "\"\"\"Deprecated. Please use callbacks instead.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @property\n    def is_single_input(self) -> bool:\n        \"\"\"Whether the tool only accepts a single input.\"\"\"\n        keys = {k for k in self.args if k != \"kwargs\"}\n        return len(keys) == 1\n    @property\n    def args(self) -> dict:\n        if self.args_schema is not None:\n            return self.args_schema.schema()[\"properties\"]\n        else:\n            schema = create_schema_from_function(self.name, self._run)\n            return schema.schema()[\"properties\"]\n    def _parse_input(\n        self,\n        tool_input: Union[str, Dict],\n    ) -> Union[str, Dict[str, Any]]:\n        \"\"\"Convert tool input to pydantic model.\"\"\"\n        input_args = self.args_schema\n        if isinstance(tool_input, str):\n            if input_args is not None:\n                key_ = next(iter(input_args.__fields__.keys()))\n                input_args.validate({key_: tool_input})\n            return tool_input\n        else:\n            if input_args is not None:\n                result = input_args.parse_obj(tool_input)\n                return {k: v for k, v in result.dict().items() if k in tool_input}\n        return tool_input\n    @root_validator()\n    def raise_deprecation(cls, values: Dict) -> Dict:\n        \"\"\"Raise deprecation warning if callback_manager is used.\"\"\"\n        if values.get(\"callback_manager\") is not None:\n            warnings.warn(\n                \"callback_manager is deprecated. Please use callbacks instead.\",\n                DeprecationWarning,\n            )\n            values[\"callbacks\"] = values.pop(\"callback_manager\", None)\n        return values", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}219{"id": "f7fac1d6b36c-4", "text": "values[\"callbacks\"] = values.pop(\"callback_manager\", None)\n        return values\n    @abstractmethod\n    def _run(\n        self,\n        *args: Any,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Use the tool.\n        Add run_manager: Optional[CallbackManagerForToolRun] = None\n        to child implementations to enable tracing,\n        \"\"\"\n    @abstractmethod\n    async def _arun(\n        self,\n        *args: Any,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Use the tool asynchronously.\n        Add run_manager: Optional[AsyncCallbackManagerForToolRun] = None\n        to child implementations to enable tracing,\n        \"\"\"\n    def _to_args_and_kwargs(self, tool_input: Union[str, Dict]) -> Tuple[Tuple, Dict]:\n        # For backwards compatibility, if run_input is a string,\n        # pass as a positional argument.\n        if isinstance(tool_input, str):\n            return (tool_input,), {}\n        else:\n            return (), tool_input\n[docs]    def run(\n        self,\n        tool_input: Union[str, Dict],\n        verbose: Optional[bool] = None,\n        start_color: Optional[str] = \"green\",\n        color: Optional[str] = \"green\",\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Run the tool.\"\"\"\n        parsed_input = self._parse_input(tool_input)\n        if not self.verbose and verbose is not None:\n            verbose_ = verbose\n        else:\n            verbose_ = self.verbose\n        callback_manager = CallbackManager.configure(\n            callbacks, self.callbacks, verbose=verbose_\n        )\n        # TODO: maybe also pass through run_manager is _run supports kwargs", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}220{"id": "f7fac1d6b36c-5", "text": ")\n        # TODO: maybe also pass through run_manager is _run supports kwargs\n        new_arg_supported = signature(self._run).parameters.get(\"run_manager\")\n        run_manager = callback_manager.on_tool_start(\n            {\"name\": self.name, \"description\": self.description},\n            tool_input if isinstance(tool_input, str) else str(tool_input),\n            color=start_color,\n            **kwargs,\n        )\n        try:\n            tool_args, tool_kwargs = self._to_args_and_kwargs(parsed_input)\n            observation = (\n                self._run(*tool_args, run_manager=run_manager, **tool_kwargs)\n                if new_arg_supported\n                else self._run(*tool_args, **tool_kwargs)\n            )\n        except (Exception, KeyboardInterrupt) as e:\n            run_manager.on_tool_error(e)\n            raise e\n        run_manager.on_tool_end(str(observation), color=color, name=self.name, **kwargs)\n        return observation\n[docs]    async def arun(\n        self,\n        tool_input: Union[str, Dict],\n        verbose: Optional[bool] = None,\n        start_color: Optional[str] = \"green\",\n        color: Optional[str] = \"green\",\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Run the tool asynchronously.\"\"\"\n        parsed_input = self._parse_input(tool_input)\n        if not self.verbose and verbose is not None:\n            verbose_ = verbose\n        else:\n            verbose_ = self.verbose\n        callback_manager = AsyncCallbackManager.configure(\n            callbacks, self.callbacks, verbose=verbose_\n        )\n        new_arg_supported = signature(self._arun).parameters.get(\"run_manager\")\n        run_manager = await callback_manager.on_tool_start(", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}221{"id": "f7fac1d6b36c-6", "text": "run_manager = await callback_manager.on_tool_start(\n            {\"name\": self.name, \"description\": self.description},\n            tool_input if isinstance(tool_input, str) else str(tool_input),\n            color=start_color,\n            **kwargs,\n        )\n        try:\n            # We then call the tool on the tool input to get an observation\n            tool_args, tool_kwargs = self._to_args_and_kwargs(parsed_input)\n            observation = (\n                await self._arun(*tool_args, run_manager=run_manager, **tool_kwargs)\n                if new_arg_supported\n                else await self._arun(*tool_args, **tool_kwargs)\n            )\n        except (Exception, KeyboardInterrupt) as e:\n            await run_manager.on_tool_error(e)\n            raise e\n        await run_manager.on_tool_end(\n            str(observation), color=color, name=self.name, **kwargs\n        )\n        return observation\n    def __call__(self, tool_input: str, callbacks: Callbacks = None) -> str:\n        \"\"\"Make tool callable.\"\"\"\n        return self.run(tool_input, callbacks=callbacks)\n[docs]class Tool(BaseTool):\n    \"\"\"Tool that takes in function or coroutine directly.\"\"\"\n    description: str = \"\"\n    func: Callable[..., str]\n    \"\"\"The function to run when the tool is called.\"\"\"\n    coroutine: Optional[Callable[..., Awaitable[str]]] = None\n    \"\"\"The asynchronous version of the function.\"\"\"\n    @property\n    def args(self) -> dict:\n        \"\"\"The tool's input arguments.\"\"\"\n        if self.args_schema is not None:\n            return self.args_schema.schema()[\"properties\"]\n        # For backwards compatibility, if the function signature is ambiguous,\n        # assume it takes a single string input.\n        return {\"tool_input\": {\"type\": \"string\"}}", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}222{"id": "f7fac1d6b36c-7", "text": "return {\"tool_input\": {\"type\": \"string\"}}\n    def _to_args_and_kwargs(self, tool_input: Union[str, Dict]) -> Tuple[Tuple, Dict]:\n        \"\"\"Convert tool input to pydantic model.\"\"\"\n        args, kwargs = super()._to_args_and_kwargs(tool_input)\n        # For backwards compatibility. The tool must be run with a single input\n        all_args = list(args) + list(kwargs.values())\n        if len(all_args) != 1:\n            raise ValueError(\n                f\"Too many arguments to single-input tool {self.name}.\"\n                f\" Args: {all_args}\"\n            )\n        return tuple(all_args), {}\n    def _run(\n        self,\n        *args: Any,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Use the tool.\"\"\"\n        new_argument_supported = signature(self.func).parameters.get(\"callbacks\")\n        return (\n            self.func(\n                *args,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **kwargs,\n            )\n            if new_argument_supported\n            else self.func(*args, **kwargs)\n        )\n    async def _arun(\n        self,\n        *args: Any,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        if self.coroutine:\n            new_argument_supported = signature(self.coroutine).parameters.get(\n                \"callbacks\"\n            )\n            return (\n                await self.coroutine(\n                    *args,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **kwargs,\n                )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}223{"id": "f7fac1d6b36c-8", "text": "**kwargs,\n                )\n                if new_argument_supported\n                else await self.coroutine(*args, **kwargs)\n            )\n        raise NotImplementedError(\"Tool does not support async\")\n    # TODO: this is for backwards compatibility, remove in future\n    def __init__(\n        self, name: str, func: Callable, description: str, **kwargs: Any\n    ) -> None:\n        \"\"\"Initialize tool.\"\"\"\n        super(Tool, self).__init__(\n            name=name, func=func, description=description, **kwargs\n        )\n[docs]    @classmethod\n    def from_function(\n        cls,\n        func: Callable,\n        name: str,  # We keep these required to support backwards compatibility\n        description: str,\n        return_direct: bool = False,\n        args_schema: Optional[Type[BaseModel]] = None,\n        **kwargs: Any,\n    ) -> Tool:\n        \"\"\"Initialize tool from a function.\"\"\"\n        return cls(\n            name=name,\n            func=func,\n            description=description,\n            return_direct=return_direct,\n            args_schema=args_schema,\n            **kwargs,\n        )\n[docs]class StructuredTool(BaseTool):\n    \"\"\"Tool that can operate on any number of inputs.\"\"\"\n    description: str = \"\"\n    args_schema: Type[BaseModel] = Field(..., description=\"The tool schema.\")\n    \"\"\"The input arguments' schema.\"\"\"\n    func: Callable[..., Any]\n    \"\"\"The function to run when the tool is called.\"\"\"\n    coroutine: Optional[Callable[..., Awaitable[Any]]] = None\n    \"\"\"The asynchronous version of the function.\"\"\"\n    @property\n    def args(self) -> dict:\n        \"\"\"The tool's input arguments.\"\"\"\n        return self.args_schema.schema()[\"properties\"]\n    def _run(", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}224{"id": "f7fac1d6b36c-9", "text": "return self.args_schema.schema()[\"properties\"]\n    def _run(\n        self,\n        *args: Any,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Use the tool.\"\"\"\n        new_argument_supported = signature(self.func).parameters.get(\"callbacks\")\n        return (\n            self.func(\n                *args,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **kwargs,\n            )\n            if new_argument_supported\n            else self.func(*args, **kwargs)\n        )\n    async def _arun(\n        self,\n        *args: Any,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        if self.coroutine:\n            new_argument_supported = signature(self.coroutine).parameters.get(\n                \"callbacks\"\n            )\n            return (\n                await self.coroutine(\n                    *args,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **kwargs,\n                )\n                if new_argument_supported\n                else await self.coroutine(*args, **kwargs)\n            )\n        raise NotImplementedError(\"Tool does not support async\")\n[docs]    @classmethod\n    def from_function(\n        cls,\n        func: Callable,\n        name: Optional[str] = None,\n        description: Optional[str] = None,\n        return_direct: bool = False,\n        args_schema: Optional[Type[BaseModel]] = None,\n        infer_schema: bool = True,\n        **kwargs: Any,\n    ) -> StructuredTool:\n        name = name or func.__name__", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}225{"id": "f7fac1d6b36c-10", "text": ") -> StructuredTool:\n        name = name or func.__name__\n        description = description or func.__doc__\n        assert (\n            description is not None\n        ), \"Function must have a docstring if description not provided.\"\n        # Description example:\n        # search_api(query: str) - Searches the API for the query.\n        description = f\"{name}{signature(func)} - {description.strip()}\"\n        _args_schema = args_schema\n        if _args_schema is None and infer_schema:\n            _args_schema = create_schema_from_function(f\"{name}Schema\", func)\n        return cls(\n            name=name,\n            func=func,\n            args_schema=_args_schema,\n            description=description,\n            return_direct=return_direct,\n            **kwargs,\n        )\n[docs]def tool(\n    *args: Union[str, Callable],\n    return_direct: bool = False,\n    args_schema: Optional[Type[BaseModel]] = None,\n    infer_schema: bool = True,\n) -> Callable:\n    \"\"\"Make tools out of functions, can be used with or without arguments.\n    Args:\n        *args: The arguments to the tool.\n        return_direct: Whether to return directly from the tool rather\n            than continuing the agent loop.\n        args_schema: optional argument schema for user to specify\n        infer_schema: Whether to infer the schema of the arguments from\n            the function's signature. This also makes the resultant tool\n            accept a dictionary input to its `run()` function.\n    Requires:\n        - Function must be of type (str) -> str\n        - Function must have a docstring\n    Examples:\n        .. code-block:: python\n            @tool\n            def search_api(query: str) -> str:\n                # Searches the API for the query.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}226{"id": "f7fac1d6b36c-11", "text": "# Searches the API for the query.\n                return\n            @tool(\"search\", return_direct=True)\n            def search_api(query: str) -> str:\n                # Searches the API for the query.\n                return\n    \"\"\"\n    def _make_with_name(tool_name: str) -> Callable:\n        def _make_tool(func: Callable) -> BaseTool:\n            if infer_schema or args_schema is not None:\n                return StructuredTool.from_function(\n                    func,\n                    name=tool_name,\n                    return_direct=return_direct,\n                    args_schema=args_schema,\n                    infer_schema=infer_schema,\n                )\n            # If someone doesn't want a schema applied, we must treat it as\n            # a simple string->string function\n            assert func.__doc__ is not None, \"Function must have a docstring\"\n            return Tool(\n                name=tool_name,\n                func=func,\n                description=f\"{tool_name} tool\",\n                return_direct=return_direct,\n            )\n        return _make_tool\n    if len(args) == 1 and isinstance(args[0], str):\n        # if the argument is a string, then we use the string as the tool name\n        # Example usage: @tool(\"search\", return_direct=True)\n        return _make_with_name(args[0])\n    elif len(args) == 1 and callable(args[0]):\n        # if the argument is a function, then we use the function name as the tool name\n        # Example usage: @tool\n        return _make_with_name(args[0].__name__)(args[0])\n    elif len(args) == 0:\n        # if there are no arguments, then we use the function name as the tool name\n        # Example usage: @tool(return_direct=True)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}227{"id": "f7fac1d6b36c-12", "text": "# Example usage: @tool(return_direct=True)\n        def _partial(func: Callable[[str], str]) -> BaseTool:\n            return _make_with_name(func.__name__)(func)\n        return _partial\n    else:\n        raise ValueError(\"Too many arguments for tool decorator\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/base.html"}228{"id": "cc2cb5d2bed4-0", "text": "Source code for langchain.tools.plugin\nfrom __future__ import annotations\nimport json\nfrom typing import Optional, Type\nimport requests\nimport yaml\nfrom pydantic import BaseModel\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nclass ApiConfig(BaseModel):\n    type: str\n    url: str\n    has_user_authentication: Optional[bool] = False\nclass AIPlugin(BaseModel):\n    \"\"\"AI Plugin Definition.\"\"\"\n    schema_version: str\n    name_for_model: str\n    name_for_human: str\n    description_for_model: str\n    description_for_human: str\n    auth: Optional[dict] = None\n    api: ApiConfig\n    logo_url: Optional[str]\n    contact_email: Optional[str]\n    legal_info_url: Optional[str]\n    @classmethod\n    def from_url(cls, url: str) -> AIPlugin:\n        \"\"\"Instantiate AIPlugin from a URL.\"\"\"\n        response = requests.get(url).json()\n        return cls(**response)\ndef marshal_spec(txt: str) -> dict:\n    \"\"\"Convert the yaml or json serialized spec to a dict.\"\"\"\n    try:\n        return json.loads(txt)\n    except json.JSONDecodeError:\n        return yaml.safe_load(txt)\nclass AIPluginToolSchema(BaseModel):\n    \"\"\"AIPLuginToolSchema.\"\"\"\n    tool_input: Optional[str] = \"\"\n[docs]class AIPluginTool(BaseTool):\n    plugin: AIPlugin\n    api_spec: str\n    args_schema: Type[AIPluginToolSchema] = AIPluginToolSchema\n[docs]    @classmethod\n    def from_plugin_url(cls, url: str) -> AIPluginTool:\n        plugin = AIPlugin.from_url(url)\n        description = (", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/plugin.html"}229{"id": "cc2cb5d2bed4-1", "text": "plugin = AIPlugin.from_url(url)\n        description = (\n            f\"Call this tool to get the OpenAPI spec (and usage guide) \"\n            f\"for interacting with the {plugin.name_for_human} API. \"\n            f\"You should only call this ONCE! What is the \"\n            f\"{plugin.name_for_human} API useful for? \"\n        ) + plugin.description_for_human\n        open_api_spec_str = requests.get(plugin.api.url).text\n        open_api_spec = marshal_spec(open_api_spec_str)\n        api_spec = (\n            f\"Usage Guide: {plugin.description_for_model}\\n\\n\"\n            f\"OpenAPI Spec: {open_api_spec}\"\n        )\n        return cls(\n            name=plugin.name_for_model,\n            description=description,\n            plugin=plugin,\n            api_spec=api_spec,\n        )\n    def _run(\n        self,\n        tool_input: Optional[str] = \"\",\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_spec\n    async def _arun(\n        self,\n        tool_input: Optional[str] = None,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        return self.api_spec\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/plugin.html"}230{"id": "4414a5007c31-0", "text": "Source code for langchain.tools.ifttt\n\"\"\"From https://github.com/SidU/teams-langchain-js/wiki/Connecting-IFTTT-Services.\n# Creating a webhook\n- Go to https://ifttt.com/create\n# Configuring the \"If This\"\n- Click on the \"If This\" button in the IFTTT interface.\n- Search for \"Webhooks\" in the search bar.\n- Choose the first option for \"Receive a web request with a JSON payload.\"\n- Choose an Event Name that is specific to the service you plan to connect to.\nThis will make it easier for you to manage the webhook URL.\nFor example, if you're connecting to Spotify, you could use \"Spotify\" as your\nEvent Name.\n- Click the \"Create Trigger\" button to save your settings and create your webhook.\n# Configuring the \"Then That\"\n- Tap on the \"Then That\" button in the IFTTT interface.\n- Search for the service you want to connect, such as Spotify.\n- Choose an action from the service, such as \"Add track to a playlist\".\n- Configure the action by specifying the necessary details, such as the playlist name,\ne.g., \"Songs from AI\".\n- Reference the JSON Payload received by the Webhook in your action. For the Spotify\nscenario, choose \"{{JsonPayload}}\" as your search query.\n- Tap the \"Create Action\" button to save your action settings.\n- Once you have finished configuring your action, click the \"Finish\" button to\ncomplete the setup.\n- Congratulations! You have successfully connected the Webhook to the desired\nservice, and you're ready to start receiving data and triggering actions \ud83c\udf89\n# Finishing up\n- To get your webhook URL go to https://ifttt.com/maker_webhooks/settings", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/ifttt.html"}231{"id": "4414a5007c31-1", "text": "- To get your webhook URL go to https://ifttt.com/maker_webhooks/settings\n- Copy the IFTTT key value from there. The URL is of the form\nhttps://maker.ifttt.com/use/YOUR_IFTTT_KEY. Grab the YOUR_IFTTT_KEY value.\n\"\"\"\nfrom typing import Optional\nimport requests\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\n[docs]class IFTTTWebhook(BaseTool):\n    \"\"\"IFTTT Webhook.\n    Args:\n        name: name of the tool\n        description: description of the tool\n        url: url to hit with the json event.\n    \"\"\"\n    url: str\n    def _run(\n        self,\n        tool_input: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        body = {\"this\": tool_input}\n        response = requests.post(self.url, data=body)\n        return response.text\n    async def _arun(\n        self,\n        tool_input: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        raise NotImplementedError(\"Not implemented.\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/ifttt.html"}232{"id": "461f35cd67b5-0", "text": "Source code for langchain.tools.wikipedia.tool\n\"\"\"Tool for the Wikipedia API.\"\"\"\nfrom typing import Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.wikipedia import WikipediaAPIWrapper\n[docs]class WikipediaQueryRun(BaseTool):\n    \"\"\"Tool that adds the capability to search using the Wikipedia API.\"\"\"\n    name = \"Wikipedia\"\n    description = (\n        \"A wrapper around Wikipedia. \"\n        \"Useful for when you need to answer general questions about \"\n        \"people, places, companies, facts, historical events, or other subjects. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: WikipediaAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Wikipedia tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Wikipedia tool asynchronously.\"\"\"\n        raise NotImplementedError(\"WikipediaQueryRun does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/wikipedia/tool.html"}233{"id": "3c11ff521557-0", "text": "Source code for langchain.tools.shell.tool\nimport asyncio\nimport platform\nimport warnings\nfrom typing import List, Optional, Type, Union\nfrom pydantic import BaseModel, Field, root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.bash import BashProcess\nclass ShellInput(BaseModel):\n    \"\"\"Commands for the Bash Shell tool.\"\"\"\n    commands: Union[str, List[str]] = Field(\n        ...,\n        description=\"List of shell commands to run. Deserialized using json.loads\",\n    )\n    \"\"\"List of shell commands to run.\"\"\"\n    @root_validator\n    def _validate_commands(cls, values: dict) -> dict:\n        \"\"\"Validate commands.\"\"\"\n        # TODO: Add real validators\n        commands = values.get(\"commands\")\n        if not isinstance(commands, list):\n            values[\"commands\"] = [commands]\n        # Warn that the bash tool is not safe\n        warnings.warn(\n            \"The shell tool has no safeguards by default. Use at your own risk.\"\n        )\n        return values\ndef _get_default_bash_processs() -> BashProcess:\n    \"\"\"Get file path from string.\"\"\"\n    return BashProcess(return_err_output=True)\ndef _get_platform() -> str:\n    \"\"\"Get platform.\"\"\"\n    system = platform.system()\n    if system == \"Darwin\":\n        return \"MacOS\"\n    return system\n[docs]class ShellTool(BaseTool):\n    \"\"\"Tool to run shell commands.\"\"\"\n    process: BashProcess = Field(default_factory=_get_default_bash_processs)\n    \"\"\"Bash process to run commands.\"\"\"\n    name: str = \"terminal\"\n    \"\"\"Name of tool.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/shell/tool.html"}234{"id": "3c11ff521557-1", "text": "name: str = \"terminal\"\n    \"\"\"Name of tool.\"\"\"\n    description: str = f\"Run shell commands on this {_get_platform()} machine.\"\n    \"\"\"Description of tool.\"\"\"\n    args_schema: Type[BaseModel] = ShellInput\n    \"\"\"Schema for input arguments.\"\"\"\n    def _run(\n        self,\n        commands: Union[str, List[str]],\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Run commands and return final output.\"\"\"\n        return self.process.run(commands)\n    async def _arun(\n        self,\n        commands: Union[str, List[str]],\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Run commands asynchronously and return final output.\"\"\"\n        return await asyncio.get_event_loop().run_in_executor(\n            None, self.process.run, commands\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/shell/tool.html"}235{"id": "1b630b27b6df-0", "text": "Source code for langchain.tools.zapier.tool\n\"\"\"## Zapier Natural Language Actions API\n\\\nFull docs here: https://nla.zapier.com/api/v1/docs\n**Zapier Natural Language Actions** gives you access to the 5k+ apps, 20k+ actions\non Zapier's platform through a natural language API interface.\nNLA supports apps like Gmail, Salesforce, Trello, Slack, Asana, HubSpot, Google Sheets,\nMicrosoft Teams, and thousands more apps: https://zapier.com/apps\nZapier NLA handles ALL the underlying API auth and translation from\nnatural language --> underlying API call --> return simplified output for LLMs\nThe key idea is you, or your users, expose a set of actions via an oauth-like setup\nwindow, which you can then query and execute via a REST API.\nNLA offers both API Key and OAuth for signing NLA API requests.\n1. Server-side (API Key): for quickly getting started, testing, and production scenarios\n    where LangChain will only use actions exposed in the developer's Zapier account\n    (and will use the developer's connected accounts on Zapier.com)\n2. User-facing (Oauth): for production scenarios where you are deploying an end-user\n    facing application and LangChain needs access to end-user's exposed actions and\n    connected accounts on Zapier.com\nThis quick start will focus on the server-side use case for brevity.\nReview [full docs](https://nla.zapier.com/api/v1/docs) or reach out to\nnla@zapier.com for user-facing oauth developer support.\nTypically you'd use SequentialChain, here's a basic example:\n    1. Use NLA to find an email in Gmail\n    2. Use LLMChain to generate a draft reply to (1)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/zapier/tool.html"}236{"id": "1b630b27b6df-1", "text": "2. Use LLMChain to generate a draft reply to (1)\n    3. Use NLA to send the draft reply (2) to someone in Slack via direct message\nIn code, below:\n```python\nimport os\n# get from https://platform.openai.com/\nos.environ[\"OPENAI_API_KEY\"] = os.environ.get(\"OPENAI_API_KEY\", \"\")\n# get from https://nla.zapier.com/demo/provider/debug\n# (under User Information, after logging in):\nos.environ[\"ZAPIER_NLA_API_KEY\"] = os.environ.get(\"ZAPIER_NLA_API_KEY\", \"\")\nfrom langchain.llms import OpenAI\nfrom langchain.agents import initialize_agent\nfrom langchain.agents.agent_toolkits import ZapierToolkit\nfrom langchain.utilities.zapier import ZapierNLAWrapper\n## step 0. expose gmail 'find email' and slack 'send channel message' actions\n# first go here, log in, expose (enable) the two actions:\n#    https://nla.zapier.com/demo/start\n#    -- for this example, can leave all fields \"Have AI guess\"\n# in an oauth scenario, you'd get your own <provider> id (instead of 'demo')\n# which you route your users through first\nllm = OpenAI(temperature=0)\nzapier = ZapierNLAWrapper()\n## To leverage a nla_oauth_access_token you may pass the value to the ZapierNLAWrapper\n## If you do this there is no need to initialize the ZAPIER_NLA_API_KEY env variable\n# zapier = ZapierNLAWrapper(zapier_nla_oauth_access_token=\"TOKEN_HERE\")\ntoolkit = ZapierToolkit.from_zapier_nla_wrapper(zapier)\nagent = initialize_agent(\n    toolkit.get_tools(),\n    llm,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/zapier/tool.html"}237{"id": "1b630b27b6df-2", "text": "agent = initialize_agent(\n    toolkit.get_tools(),\n    llm,\n    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n    verbose=True\n)\nagent.run((\"Summarize the last email I received regarding Silicon Valley Bank. \"\n    \"Send the summary to the #test-zapier channel in slack.\"))\n```\n\"\"\"\nfrom typing import Any, Dict, Optional\nfrom pydantic import Field, root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.zapier.prompt import BASE_ZAPIER_TOOL_PROMPT\nfrom langchain.utilities.zapier import ZapierNLAWrapper\n[docs]class ZapierNLARunAction(BaseTool):\n    \"\"\"\n    Args:\n        action_id: a specific action ID (from list actions) of the action to execute\n            (the set api_key must be associated with the action owner)\n        instructions: a natural language instruction string for using the action\n            (eg. \"get the latest email from Mike Knoop\" for \"Gmail: find email\" action)\n        params: a dict, optional. Any params provided will *override* AI guesses\n            from `instructions` (see \"understanding the AI guessing flow\" here:\n            https://nla.zapier.com/api/v1/docs)\n    \"\"\"\n    api_wrapper: ZapierNLAWrapper = Field(default_factory=ZapierNLAWrapper)\n    action_id: str\n    params: Optional[dict] = None\n    base_prompt: str = BASE_ZAPIER_TOOL_PROMPT\n    zapier_description: str\n    params_schema: Dict[str, str] = Field(default_factory=dict)\n    name = \"\"\n    description = \"\"\n    @root_validator", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/zapier/tool.html"}238{"id": "1b630b27b6df-3", "text": "name = \"\"\n    description = \"\"\n    @root_validator\n    def set_name_description(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        zapier_description = values[\"zapier_description\"]\n        params_schema = values[\"params_schema\"]\n        if \"instructions\" in params_schema:\n            del params_schema[\"instructions\"]\n        # Ensure base prompt (if overrided) contains necessary input fields\n        necessary_fields = {\"{zapier_description}\", \"{params}\"}\n        if not all(field in values[\"base_prompt\"] for field in necessary_fields):\n            raise ValueError(\n                \"Your custom base Zapier prompt must contain input fields for \"\n                \"{zapier_description} and {params}.\"\n            )\n        values[\"name\"] = zapier_description\n        values[\"description\"] = values[\"base_prompt\"].format(\n            zapier_description=zapier_description,\n            params=str(list(params_schema.keys())),\n        )\n        return values\n    def _run(\n        self, instructions: str, run_manager: Optional[CallbackManagerForToolRun] = None\n    ) -> str:\n        \"\"\"Use the Zapier NLA tool to return a list of all exposed user actions.\"\"\"\n        return self.api_wrapper.run_as_str(self.action_id, instructions, self.params)\n    async def _arun(\n        self,\n        _: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Zapier NLA tool to return a list of all exposed user actions.\"\"\"\n        raise NotImplementedError(\"ZapierNLAListActions does not support async\")\nZapierNLARunAction.__doc__ = (\n    ZapierNLAWrapper.run.__doc__ + ZapierNLARunAction.__doc__  # type: ignore\n)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/zapier/tool.html"}239{"id": "1b630b27b6df-4", "text": ")\n# other useful actions\n[docs]class ZapierNLAListActions(BaseTool):\n    \"\"\"\n    Args:\n        None\n    \"\"\"\n    name = \"Zapier NLA: List Actions\"\n    description = BASE_ZAPIER_TOOL_PROMPT + (\n        \"This tool returns a list of the user's exposed actions.\"\n    )\n    api_wrapper: ZapierNLAWrapper = Field(default_factory=ZapierNLAWrapper)\n    def _run(\n        self,\n        _: str = \"\",\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Zapier NLA tool to return a list of all exposed user actions.\"\"\"\n        return self.api_wrapper.list_as_str()\n    async def _arun(\n        self,\n        _: str = \"\",\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Zapier NLA tool to return a list of all exposed user actions.\"\"\"\n        raise NotImplementedError(\"ZapierNLAListActions does not support async\")\nZapierNLAListActions.__doc__ = (\n    ZapierNLAWrapper.list.__doc__ + ZapierNLAListActions.__doc__  # type: ignore\n)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/zapier/tool.html"}240{"id": "863f81a98a87-0", "text": "Source code for langchain.tools.google_places.tool\n\"\"\"Tool for the Google search API.\"\"\"\nfrom typing import Optional\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.google_places_api import GooglePlacesAPIWrapper\nclass GooglePlacesSchema(BaseModel):\n    query: str = Field(..., description=\"Query for goole maps\")\n[docs]class GooglePlacesTool(BaseTool):\n    \"\"\"Tool that adds the capability to query the Google places API.\"\"\"\n    name = \"Google Places\"\n    description = (\n        \"A wrapper around Google Places. \"\n        \"Useful for when you need to validate or \"\n        \"discover addressed from ambiguous text. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: GooglePlacesAPIWrapper = Field(default_factory=GooglePlacesAPIWrapper)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"GooglePlacesRun does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/google_places/tool.html"}241{"id": "06a4d7d92ffd-0", "text": "Source code for langchain.tools.azure_cognitive_services.image_analysis\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, Dict, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.azure_cognitive_services.utils import detect_file_src_type\nfrom langchain.tools.base import BaseTool\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class AzureCogsImageAnalysisTool(BaseTool):\n    \"\"\"Tool that queries the Azure Cognitive Services Image Analysis API.\n    In order to set this up, follow instructions at:\n    https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/image-analysis-client-library-40\n    \"\"\"\n    azure_cogs_key: str = \"\"  #: :meta private:\n    azure_cogs_endpoint: str = \"\"  #: :meta private:\n    vision_service: Any  #: :meta private:\n    analysis_options: Any  #: :meta private:\n    name = \"Azure Cognitive Services Image Analysis\"\n    description = (\n        \"A wrapper around Azure Cognitive Services Image Analysis. \"\n        \"Useful for when you need to analyze images. \"\n        \"Input should be a url to an image.\"\n    )\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        azure_cogs_key = get_from_dict_or_env(\n            values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )\n        azure_cogs_endpoint = get_from_dict_or_env(\n            values, \"azure_cogs_endpoint\", \"AZURE_COGS_ENDPOINT\"\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/image_analysis.html"}242{"id": "06a4d7d92ffd-1", "text": "values, \"azure_cogs_endpoint\", \"AZURE_COGS_ENDPOINT\"\n        )\n        try:\n            import azure.ai.vision as sdk\n            values[\"vision_service\"] = sdk.VisionServiceOptions(\n                endpoint=azure_cogs_endpoint, key=azure_cogs_key\n            )\n            values[\"analysis_options\"] = sdk.ImageAnalysisOptions()\n            values[\"analysis_options\"].features = (\n                sdk.ImageAnalysisFeature.CAPTION\n                | sdk.ImageAnalysisFeature.OBJECTS\n                | sdk.ImageAnalysisFeature.TAGS\n                | sdk.ImageAnalysisFeature.TEXT\n            )\n        except ImportError:\n            raise ImportError(\n                \"azure-ai-vision is not installed. \"\n                \"Run `pip install azure-ai-vision` to install.\"\n            )\n        return values\n    def _image_analysis(self, image_path: str) -> Dict:\n        try:\n            import azure.ai.vision as sdk\n        except ImportError:\n            pass\n        image_src_type = detect_file_src_type(image_path)\n        if image_src_type == \"local\":\n            vision_source = sdk.VisionSource(filename=image_path)\n        elif image_src_type == \"remote\":\n            vision_source = sdk.VisionSource(url=image_path)\n        else:\n            raise ValueError(f\"Invalid image path: {image_path}\")\n        image_analyzer = sdk.ImageAnalyzer(\n            self.vision_service, vision_source, self.analysis_options\n        )\n        result = image_analyzer.analyze()\n        res_dict = {}\n        if result.reason == sdk.ImageAnalysisResultReason.ANALYZED:\n            if result.caption is not None:\n                res_dict[\"caption\"] = result.caption.content\n            if result.objects is not None:\n                res_dict[\"objects\"] = [obj.name for obj in result.objects]\n            if result.tags is not None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/image_analysis.html"}243{"id": "06a4d7d92ffd-2", "text": "if result.tags is not None:\n                res_dict[\"tags\"] = [tag.name for tag in result.tags]\n            if result.text is not None:\n                res_dict[\"text\"] = [line.content for line in result.text.lines]\n        else:\n            error_details = sdk.ImageAnalysisErrorDetails.from_result(result)\n            raise RuntimeError(\n                f\"Image analysis failed.\\n\"\n                f\"Reason: {error_details.reason}\\n\"\n                f\"Details: {error_details.message}\"\n            )\n        return res_dict\n    def _format_image_analysis_result(self, image_analysis_result: Dict) -> str:\n        formatted_result = []\n        if \"caption\" in image_analysis_result:\n            formatted_result.append(\"Caption: \" + image_analysis_result[\"caption\"])\n        if (\n            \"objects\" in image_analysis_result\n            and len(image_analysis_result[\"objects\"]) > 0\n        ):\n            formatted_result.append(\n                \"Objects: \" + \", \".join(image_analysis_result[\"objects\"])\n            )\n        if \"tags\" in image_analysis_result and len(image_analysis_result[\"tags\"]) > 0:\n            formatted_result.append(\"Tags: \" + \", \".join(image_analysis_result[\"tags\"]))\n        if \"text\" in image_analysis_result and len(image_analysis_result[\"text\"]) > 0:\n            formatted_result.append(\"Text: \" + \", \".join(image_analysis_result[\"text\"]))\n        return \"\\n\".join(formatted_result)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        try:\n            image_analysis_result = self._image_analysis(query)\n            if not image_analysis_result:\n                return \"No good image analysis result was found\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/image_analysis.html"}244{"id": "06a4d7d92ffd-3", "text": "if not image_analysis_result:\n                return \"No good image analysis result was found\"\n            return self._format_image_analysis_result(image_analysis_result)\n        except Exception as e:\n            raise RuntimeError(f\"Error while running AzureCogsImageAnalysisTool: {e}\")\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"AzureCogsImageAnalysisTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/image_analysis.html"}245{"id": "fe0d7446fea4-0", "text": "Source code for langchain.tools.azure_cognitive_services.form_recognizer\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.azure_cognitive_services.utils import detect_file_src_type\nfrom langchain.tools.base import BaseTool\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class AzureCogsFormRecognizerTool(BaseTool):\n    \"\"\"Tool that queries the Azure Cognitive Services Form Recognizer API.\n    In order to set this up, follow instructions at:\n    https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/quickstarts/get-started-sdks-rest-api?view=form-recog-3.0.0&pivots=programming-language-python\n    \"\"\"\n    azure_cogs_key: str = \"\"  #: :meta private:\n    azure_cogs_endpoint: str = \"\"  #: :meta private:\n    doc_analysis_client: Any  #: :meta private:\n    name = \"Azure Cognitive Services Form Recognizer\"\n    description = (\n        \"A wrapper around Azure Cognitive Services Form Recognizer. \"\n        \"Useful for when you need to \"\n        \"extract text, tables, and key-value pairs from documents. \"\n        \"Input should be a url to a document.\"\n    )\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        azure_cogs_key = get_from_dict_or_env(\n            values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/form_recognizer.html"}246{"id": "fe0d7446fea4-1", "text": "values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )\n        azure_cogs_endpoint = get_from_dict_or_env(\n            values, \"azure_cogs_endpoint\", \"AZURE_COGS_ENDPOINT\"\n        )\n        try:\n            from azure.ai.formrecognizer import DocumentAnalysisClient\n            from azure.core.credentials import AzureKeyCredential\n            values[\"doc_analysis_client\"] = DocumentAnalysisClient(\n                endpoint=azure_cogs_endpoint,\n                credential=AzureKeyCredential(azure_cogs_key),\n            )\n        except ImportError:\n            raise ImportError(\n                \"azure-ai-formrecognizer is not installed. \"\n                \"Run `pip install azure-ai-formrecognizer` to install.\"\n            )\n        return values\n    def _parse_tables(self, tables: List[Any]) -> List[Any]:\n        result = []\n        for table in tables:\n            rc, cc = table.row_count, table.column_count\n            _table = [[\"\" for _ in range(cc)] for _ in range(rc)]\n            for cell in table.cells:\n                _table[cell.row_index][cell.column_index] = cell.content\n            result.append(_table)\n        return result\n    def _parse_kv_pairs(self, kv_pairs: List[Any]) -> List[Any]:\n        result = []\n        for kv_pair in kv_pairs:\n            key = kv_pair.key.content if kv_pair.key else \"\"\n            value = kv_pair.value.content if kv_pair.value else \"\"\n            result.append((key, value))\n        return result\n    def _document_analysis(self, document_path: str) -> Dict:\n        document_src_type = detect_file_src_type(document_path)\n        if document_src_type == \"local\":\n            with open(document_path, \"rb\") as document:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/form_recognizer.html"}247{"id": "fe0d7446fea4-2", "text": "with open(document_path, \"rb\") as document:\n                poller = self.doc_analysis_client.begin_analyze_document(\n                    \"prebuilt-document\", document\n                )\n        elif document_src_type == \"remote\":\n            poller = self.doc_analysis_client.begin_analyze_document_from_url(\n                \"prebuilt-document\", document_path\n            )\n        else:\n            raise ValueError(f\"Invalid document path: {document_path}\")\n        result = poller.result()\n        res_dict = {}\n        if result.content is not None:\n            res_dict[\"content\"] = result.content\n        if result.tables is not None:\n            res_dict[\"tables\"] = self._parse_tables(result.tables)\n        if result.key_value_pairs is not None:\n            res_dict[\"key_value_pairs\"] = self._parse_kv_pairs(result.key_value_pairs)\n        return res_dict\n    def _format_document_analysis_result(self, document_analysis_result: Dict) -> str:\n        formatted_result = []\n        if \"content\" in document_analysis_result:\n            formatted_result.append(\n                f\"Content: {document_analysis_result['content']}\".replace(\"\\n\", \" \")\n            )\n        if \"tables\" in document_analysis_result:\n            for i, table in enumerate(document_analysis_result[\"tables\"]):\n                formatted_result.append(f\"Table {i}: {table}\".replace(\"\\n\", \" \"))\n        if \"key_value_pairs\" in document_analysis_result:\n            for kv_pair in document_analysis_result[\"key_value_pairs\"]:\n                formatted_result.append(\n                    f\"{kv_pair[0]}: {kv_pair[1]}\".replace(\"\\n\", \" \")\n                )\n        return \"\\n\".join(formatted_result)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/form_recognizer.html"}248{"id": "fe0d7446fea4-3", "text": "run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        try:\n            document_analysis_result = self._document_analysis(query)\n            if not document_analysis_result:\n                return \"No good document analysis result was found\"\n            return self._format_document_analysis_result(document_analysis_result)\n        except Exception as e:\n            raise RuntimeError(f\"Error while running AzureCogsFormRecognizerTool: {e}\")\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"AzureCogsFormRecognizerTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/form_recognizer.html"}249{"id": "6c926b6cfb39-0", "text": "Source code for langchain.tools.azure_cognitive_services.speech2text\nfrom __future__ import annotations\nimport logging\nimport time\nfrom typing import Any, Dict, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.azure_cognitive_services.utils import (\n    detect_file_src_type,\n    download_audio_from_url,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class AzureCogsSpeech2TextTool(BaseTool):\n    \"\"\"Tool that queries the Azure Cognitive Services Speech2Text API.\n    In order to set this up, follow instructions at:\n    https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/get-started-speech-to-text?pivots=programming-language-python\n    \"\"\"\n    azure_cogs_key: str = \"\"  #: :meta private:\n    azure_cogs_region: str = \"\"  #: :meta private:\n    speech_language: str = \"en-US\"  #: :meta private:\n    speech_config: Any  #: :meta private:\n    name = \"Azure Cognitive Services Speech2Text\"\n    description = (\n        \"A wrapper around Azure Cognitive Services Speech2Text. \"\n        \"Useful for when you need to transcribe audio to text. \"\n        \"Input should be a url to an audio file.\"\n    )\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        azure_cogs_key = get_from_dict_or_env(\n            values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/speech2text.html"}250{"id": "6c926b6cfb39-1", "text": "values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )\n        azure_cogs_region = get_from_dict_or_env(\n            values, \"azure_cogs_region\", \"AZURE_COGS_REGION\"\n        )\n        try:\n            import azure.cognitiveservices.speech as speechsdk\n            values[\"speech_config\"] = speechsdk.SpeechConfig(\n                subscription=azure_cogs_key, region=azure_cogs_region\n            )\n        except ImportError:\n            raise ImportError(\n                \"azure-cognitiveservices-speech is not installed. \"\n                \"Run `pip install azure-cognitiveservices-speech` to install.\"\n            )\n        return values\n    def _continuous_recognize(self, speech_recognizer: Any) -> str:\n        done = False\n        text = \"\"\n        def stop_cb(evt: Any) -> None:\n            \"\"\"callback that stop continuous recognition\"\"\"\n            speech_recognizer.stop_continuous_recognition_async()\n            nonlocal done\n            done = True\n        def retrieve_cb(evt: Any) -> None:\n            \"\"\"callback that retrieves the intermediate recognition results\"\"\"\n            nonlocal text\n            text += evt.result.text\n        # retrieve text on recognized events\n        speech_recognizer.recognized.connect(retrieve_cb)\n        # stop continuous recognition on either session stopped or canceled events\n        speech_recognizer.session_stopped.connect(stop_cb)\n        speech_recognizer.canceled.connect(stop_cb)\n        # Start continuous speech recognition\n        speech_recognizer.start_continuous_recognition_async()\n        while not done:\n            time.sleep(0.5)\n        return text\n    def _speech2text(self, audio_path: str, speech_language: str) -> str:\n        try:\n            import azure.cognitiveservices.speech as speechsdk", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/speech2text.html"}251{"id": "6c926b6cfb39-2", "text": "try:\n            import azure.cognitiveservices.speech as speechsdk\n        except ImportError:\n            pass\n        audio_src_type = detect_file_src_type(audio_path)\n        if audio_src_type == \"local\":\n            audio_config = speechsdk.AudioConfig(filename=audio_path)\n        elif audio_src_type == \"remote\":\n            tmp_audio_path = download_audio_from_url(audio_path)\n            audio_config = speechsdk.AudioConfig(filename=tmp_audio_path)\n        else:\n            raise ValueError(f\"Invalid audio path: {audio_path}\")\n        self.speech_config.speech_recognition_language = speech_language\n        speech_recognizer = speechsdk.SpeechRecognizer(self.speech_config, audio_config)\n        return self._continuous_recognize(speech_recognizer)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        try:\n            text = self._speech2text(query, self.speech_language)\n            return text\n        except Exception as e:\n            raise RuntimeError(f\"Error while running AzureCogsSpeech2TextTool: {e}\")\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"AzureCogsSpeech2TextTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/speech2text.html"}252{"id": "b4f7d3a1edb1-0", "text": "Source code for langchain.tools.azure_cognitive_services.text2speech\nfrom __future__ import annotations\nimport logging\nimport tempfile\nfrom typing import Any, Dict, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class AzureCogsText2SpeechTool(BaseTool):\n    \"\"\"Tool that queries the Azure Cognitive Services Text2Speech API.\n    In order to set this up, follow instructions at:\n    https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/get-started-text-to-speech?pivots=programming-language-python\n    \"\"\"\n    azure_cogs_key: str = \"\"  #: :meta private:\n    azure_cogs_region: str = \"\"  #: :meta private:\n    speech_language: str = \"en-US\"  #: :meta private:\n    speech_config: Any  #: :meta private:\n    name = \"Azure Cognitive Services Text2Speech\"\n    description = (\n        \"A wrapper around Azure Cognitive Services Text2Speech. \"\n        \"Useful for when you need to convert text to speech. \"\n    )\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        azure_cogs_key = get_from_dict_or_env(\n            values, \"azure_cogs_key\", \"AZURE_COGS_KEY\"\n        )\n        azure_cogs_region = get_from_dict_or_env(\n            values, \"azure_cogs_region\", \"AZURE_COGS_REGION\"\n        )\n        try:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/text2speech.html"}253{"id": "b4f7d3a1edb1-1", "text": ")\n        try:\n            import azure.cognitiveservices.speech as speechsdk\n            values[\"speech_config\"] = speechsdk.SpeechConfig(\n                subscription=azure_cogs_key, region=azure_cogs_region\n            )\n        except ImportError:\n            raise ImportError(\n                \"azure-cognitiveservices-speech is not installed. \"\n                \"Run `pip install azure-cognitiveservices-speech` to install.\"\n            )\n        return values\n    def _text2speech(self, text: str, speech_language: str) -> str:\n        try:\n            import azure.cognitiveservices.speech as speechsdk\n        except ImportError:\n            pass\n        self.speech_config.speech_synthesis_language = speech_language\n        speech_synthesizer = speechsdk.SpeechSynthesizer(\n            speech_config=self.speech_config, audio_config=None\n        )\n        result = speech_synthesizer.speak_text(text)\n        if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:\n            stream = speechsdk.AudioDataStream(result)\n            with tempfile.NamedTemporaryFile(\n                mode=\"wb\", suffix=\".wav\", delete=False\n            ) as f:\n                stream.save_to_wav_file(f.name)\n            return f.name\n        elif result.reason == speechsdk.ResultReason.Canceled:\n            cancellation_details = result.cancellation_details\n            logger.debug(f\"Speech synthesis canceled: {cancellation_details.reason}\")\n            if cancellation_details.reason == speechsdk.CancellationReason.Error:\n                raise RuntimeError(\n                    f\"Speech synthesis error: {cancellation_details.error_details}\"\n                )\n            return \"Speech synthesis canceled.\"\n        else:\n            return f\"Speech synthesis failed: {result.reason}\"\n    def _run(\n        self,\n        query: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/text2speech.html"}254{"id": "b4f7d3a1edb1-2", "text": "def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        try:\n            speech_file = self._text2speech(query, self.speech_language)\n            return speech_file\n        except Exception as e:\n            raise RuntimeError(f\"Error while running AzureCogsText2SpeechTool: {e}\")\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"AzureCogsText2SpeechTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/azure_cognitive_services/text2speech.html"}255{"id": "27d5be0c8cb2-0", "text": "Source code for langchain.tools.file_management.read\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass ReadFileInput(BaseModel):\n    \"\"\"Input for ReadFileTool.\"\"\"\n    file_path: str = Field(..., description=\"name of file\")\n[docs]class ReadFileTool(BaseFileToolMixin, BaseTool):\n    name: str = \"read_file\"\n    args_schema: Type[BaseModel] = ReadFileInput\n    description: str = \"Read file from disk\"\n    def _run(\n        self,\n        file_path: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            read_path = self.get_relative_path(file_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(arg_name=\"file_path\", value=file_path)\n        if not read_path.exists():\n            return f\"Error: no such file or directory: {file_path}\"\n        try:\n            with read_path.open(\"r\", encoding=\"utf-8\") as f:\n                content = f.read()\n            return content\n        except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        file_path: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/read.html"}256{"id": "27d5be0c8cb2-1", "text": "# TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/read.html"}257{"id": "a26683cc34a0-0", "text": "Source code for langchain.tools.file_management.file_search\nimport fnmatch\nimport os\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass FileSearchInput(BaseModel):\n    \"\"\"Input for FileSearchTool.\"\"\"\n    dir_path: str = Field(\n        default=\".\",\n        description=\"Subdirectory to search in.\",\n    )\n    pattern: str = Field(\n        ...,\n        description=\"Unix shell regex, where * matches everything.\",\n    )\n[docs]class FileSearchTool(BaseFileToolMixin, BaseTool):\n    name: str = \"file_search\"\n    args_schema: Type[BaseModel] = FileSearchInput\n    description: str = (\n        \"Recursively search for files in a subdirectory that match the regex pattern\"\n    )\n    def _run(\n        self,\n        pattern: str,\n        dir_path: str = \".\",\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            dir_path_ = self.get_relative_path(dir_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(arg_name=\"dir_path\", value=dir_path)\n        matches = []\n        try:\n            for root, _, filenames in os.walk(dir_path_):\n                for filename in fnmatch.filter(filenames, pattern):\n                    absolute_path = os.path.join(root, filename)\n                    relative_path = os.path.relpath(absolute_path, dir_path_)\n                    matches.append(relative_path)\n            if matches:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/file_search.html"}258{"id": "a26683cc34a0-1", "text": "matches.append(relative_path)\n            if matches:\n                return \"\\n\".join(matches)\n            else:\n                return f\"No files found for pattern {pattern} in directory {dir_path}\"\n        except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        dir_path: str,\n        pattern: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/file_search.html"}259{"id": "fe6d423f5f3d-0", "text": "Source code for langchain.tools.file_management.list_dir\nimport os\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass DirectoryListingInput(BaseModel):\n    \"\"\"Input for ListDirectoryTool.\"\"\"\n    dir_path: str = Field(default=\".\", description=\"Subdirectory to list.\")\n[docs]class ListDirectoryTool(BaseFileToolMixin, BaseTool):\n    name: str = \"list_directory\"\n    args_schema: Type[BaseModel] = DirectoryListingInput\n    description: str = \"List files and directories in a specified folder\"\n    def _run(\n        self,\n        dir_path: str = \".\",\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            dir_path_ = self.get_relative_path(dir_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(arg_name=\"dir_path\", value=dir_path)\n        try:\n            entries = os.listdir(dir_path_)\n            if entries:\n                return \"\\n\".join(entries)\n            else:\n                return f\"No files found in directory {dir_path}\"\n        except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        dir_path: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/list_dir.html"}260{"id": "fe6d423f5f3d-1", "text": "raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/list_dir.html"}261{"id": "8e03746c2efc-0", "text": "Source code for langchain.tools.file_management.write\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass WriteFileInput(BaseModel):\n    \"\"\"Input for WriteFileTool.\"\"\"\n    file_path: str = Field(..., description=\"name of file\")\n    text: str = Field(..., description=\"text to write to file\")\n    append: bool = Field(\n        default=False, description=\"Whether to append to an existing file.\"\n    )\n[docs]class WriteFileTool(BaseFileToolMixin, BaseTool):\n    name: str = \"write_file\"\n    args_schema: Type[BaseModel] = WriteFileInput\n    description: str = \"Write file to disk\"\n    def _run(\n        self,\n        file_path: str,\n        text: str,\n        append: bool = False,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            write_path = self.get_relative_path(file_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(arg_name=\"file_path\", value=file_path)\n        try:\n            write_path.parent.mkdir(exist_ok=True, parents=False)\n            mode = \"a\" if append else \"w\"\n            with write_path.open(mode, encoding=\"utf-8\") as f:\n                f.write(text)\n            return f\"File written successfully to {file_path}.\"\n        except Exception as e:\n            return \"Error: \" + str(e)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/write.html"}262{"id": "8e03746c2efc-1", "text": "except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        file_path: str,\n        text: str,\n        append: bool = False,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/write.html"}263{"id": "f7d76796fce1-0", "text": "Source code for langchain.tools.file_management.copy\nimport shutil\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass FileCopyInput(BaseModel):\n    \"\"\"Input for CopyFileTool.\"\"\"\n    source_path: str = Field(..., description=\"Path of the file to copy\")\n    destination_path: str = Field(..., description=\"Path to save the copied file\")\n[docs]class CopyFileTool(BaseFileToolMixin, BaseTool):\n    name: str = \"copy_file\"\n    args_schema: Type[BaseModel] = FileCopyInput\n    description: str = \"Create a copy of a file in a specified location\"\n    def _run(\n        self,\n        source_path: str,\n        destination_path: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            source_path_ = self.get_relative_path(source_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(\n                arg_name=\"source_path\", value=source_path\n            )\n        try:\n            destination_path_ = self.get_relative_path(destination_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(\n                arg_name=\"destination_path\", value=destination_path\n            )\n        try:\n            shutil.copy2(source_path_, destination_path_, follow_symlinks=False)\n            return f\"File copied successfully from {source_path} to {destination_path}.\"\n        except Exception as e:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/copy.html"}264{"id": "f7d76796fce1-1", "text": "except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        source_path: str,\n        destination_path: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/copy.html"}265{"id": "08010246c557-0", "text": "Source code for langchain.tools.file_management.move\nimport shutil\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass FileMoveInput(BaseModel):\n    \"\"\"Input for MoveFileTool.\"\"\"\n    source_path: str = Field(..., description=\"Path of the file to move\")\n    destination_path: str = Field(..., description=\"New path for the moved file\")\n[docs]class MoveFileTool(BaseFileToolMixin, BaseTool):\n    name: str = \"move_file\"\n    args_schema: Type[BaseModel] = FileMoveInput\n    description: str = \"Move or rename a file from one location to another\"\n    def _run(\n        self,\n        source_path: str,\n        destination_path: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            source_path_ = self.get_relative_path(source_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(\n                arg_name=\"source_path\", value=source_path\n            )\n        try:\n            destination_path_ = self.get_relative_path(destination_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(\n                arg_name=\"destination_path_\", value=destination_path_\n            )\n        if not source_path_.exists():\n            return f\"Error: no such file or directory {source_path}\"\n        try:\n            # shutil.move expects str args in 3.8\n            shutil.move(str(source_path_), destination_path_)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/move.html"}266{"id": "08010246c557-1", "text": "shutil.move(str(source_path_), destination_path_)\n            return f\"File moved successfully from {source_path} to {destination_path}.\"\n        except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        source_path: str,\n        destination_path: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/move.html"}267{"id": "8a9430181825-0", "text": "Source code for langchain.tools.file_management.delete\nimport os\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.file_management.utils import (\n    INVALID_PATH_TEMPLATE,\n    BaseFileToolMixin,\n    FileValidationError,\n)\nclass FileDeleteInput(BaseModel):\n    \"\"\"Input for DeleteFileTool.\"\"\"\n    file_path: str = Field(..., description=\"Path of the file to delete\")\n[docs]class DeleteFileTool(BaseFileToolMixin, BaseTool):\n    name: str = \"file_delete\"\n    args_schema: Type[BaseModel] = FileDeleteInput\n    description: str = \"Delete a file\"\n    def _run(\n        self,\n        file_path: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            file_path_ = self.get_relative_path(file_path)\n        except FileValidationError:\n            return INVALID_PATH_TEMPLATE.format(arg_name=\"file_path\", value=file_path)\n        if not file_path_.exists():\n            return f\"Error: no such file or directory: {file_path}\"\n        try:\n            os.remove(file_path_)\n            return f\"File deleted successfully: {file_path}.\"\n        except Exception as e:\n            return \"Error: \" + str(e)\n    async def _arun(\n        self,\n        file_path: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        # TODO: Add aiofiles method\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/delete.html"}268{"id": "8a9430181825-1", "text": "raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/file_management/delete.html"}269{"id": "4569e340b66d-0", "text": "Source code for langchain.tools.playwright.navigate_back\nfrom __future__ import annotations\nfrom typing import Optional, Type\nfrom pydantic import BaseModel\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import (\n    aget_current_page,\n    get_current_page,\n)\n[docs]class NavigateBackTool(BaseBrowserTool):\n    \"\"\"Navigate back to the previous page in the browser history.\"\"\"\n    name: str = \"previous_webpage\"\n    description: str = \"Navigate back to the previous page in the browser history\"\n    args_schema: Type[BaseModel] = BaseModel\n    def _run(self, run_manager: Optional[CallbackManagerForToolRun] = None) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        response = page.go_back()\n        if response:\n            return (\n                f\"Navigated back to the previous page with URL '{response.url}'.\"\n                f\" Status code {response.status}\"\n            )\n        else:\n            return \"Unable to navigate back; no previous page in the history\"\n    async def _arun(\n        self,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        response = await page.go_back()\n        if response:\n            return (", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/navigate_back.html"}270{"id": "4569e340b66d-1", "text": "response = await page.go_back()\n        if response:\n            return (\n                f\"Navigated back to the previous page with URL '{response.url}'.\"\n                f\" Status code {response.status}\"\n            )\n        else:\n            return \"Unable to navigate back; no previous page in the history\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/navigate_back.html"}271{"id": "977bb43d0c2a-0", "text": "Source code for langchain.tools.playwright.navigate\nfrom __future__ import annotations\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import (\n    aget_current_page,\n    get_current_page,\n)\nclass NavigateToolInput(BaseModel):\n    \"\"\"Input for NavigateToolInput.\"\"\"\n    url: str = Field(..., description=\"url to navigate to\")\n[docs]class NavigateTool(BaseBrowserTool):\n    name: str = \"navigate_browser\"\n    description: str = \"Navigate a browser to the specified URL\"\n    args_schema: Type[BaseModel] = NavigateToolInput\n    def _run(\n        self,\n        url: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        response = page.goto(url)\n        status = response.status if response else \"unknown\"\n        return f\"Navigating to {url} returned status code {status}\"\n    async def _arun(\n        self,\n        url: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        response = await page.goto(url)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/navigate.html"}272{"id": "977bb43d0c2a-1", "text": "response = await page.goto(url)\n        status = response.status if response else \"unknown\"\n        return f\"Navigating to {url} returned status code {status}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/navigate.html"}273{"id": "3783a1a2f034-0", "text": "Source code for langchain.tools.playwright.extract_hyperlinks\nfrom __future__ import annotations\nimport json\nfrom typing import TYPE_CHECKING, Any, Optional, Type\nfrom pydantic import BaseModel, Field, root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import aget_current_page, get_current_page\nif TYPE_CHECKING:\n    pass\nclass ExtractHyperlinksToolInput(BaseModel):\n    \"\"\"Input for ExtractHyperlinksTool.\"\"\"\n    absolute_urls: bool = Field(\n        default=False,\n        description=\"Return absolute URLs instead of relative URLs\",\n    )\n[docs]class ExtractHyperlinksTool(BaseBrowserTool):\n    \"\"\"Extract all hyperlinks on the page.\"\"\"\n    name: str = \"extract_hyperlinks\"\n    description: str = \"Extract all hyperlinks on the current webpage\"\n    args_schema: Type[BaseModel] = ExtractHyperlinksToolInput\n    @root_validator\n    def check_bs_import(cls, values: dict) -> dict:\n        \"\"\"Check that the arguments are valid.\"\"\"\n        try:\n            from bs4 import BeautifulSoup  # noqa: F401\n        except ImportError:\n            raise ValueError(\n                \"The 'beautifulsoup4' package is required to use this tool.\"\n                \" Please install it with 'pip install beautifulsoup4'.\"\n            )\n        return values\n[docs]    @staticmethod\n    def scrape_page(page: Any, html_content: str, absolute_urls: bool) -> str:\n        from urllib.parse import urljoin\n        from bs4 import BeautifulSoup\n        # Parse the HTML content with BeautifulSoup\n        soup = BeautifulSoup(html_content, \"lxml\")\n        # Find all the anchor elements and extract their href attributes", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/extract_hyperlinks.html"}274{"id": "3783a1a2f034-1", "text": "# Find all the anchor elements and extract their href attributes\n        anchors = soup.find_all(\"a\")\n        if absolute_urls:\n            base_url = page.url\n            links = [urljoin(base_url, anchor.get(\"href\", \"\")) for anchor in anchors]\n        else:\n            links = [anchor.get(\"href\", \"\") for anchor in anchors]\n        # Return the list of links as a JSON string\n        return json.dumps(links)\n    def _run(\n        self,\n        absolute_urls: bool = False,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        html_content = page.content()\n        return self.scrape_page(page, html_content, absolute_urls)\n    async def _arun(\n        self,\n        absolute_urls: bool = False,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        html_content = await page.content()\n        return self.scrape_page(page, html_content, absolute_urls)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/extract_hyperlinks.html"}275{"id": "a9aea442cc7b-0", "text": "Source code for langchain.tools.playwright.current_page\nfrom __future__ import annotations\nfrom typing import Optional, Type\nfrom pydantic import BaseModel\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import aget_current_page, get_current_page\n[docs]class CurrentWebPageTool(BaseBrowserTool):\n    name: str = \"current_webpage\"\n    description: str = \"Returns the URL of the current page\"\n    args_schema: Type[BaseModel] = BaseModel\n    def _run(\n        self,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        return str(page.url)\n    async def _arun(\n        self,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        return str(page.url)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/current_page.html"}276{"id": "bd3bd66536a0-0", "text": "Source code for langchain.tools.playwright.click\nfrom __future__ import annotations\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import (\n    aget_current_page,\n    get_current_page,\n)\nclass ClickToolInput(BaseModel):\n    \"\"\"Input for ClickTool.\"\"\"\n    selector: str = Field(..., description=\"CSS selector for the element to click\")\n[docs]class ClickTool(BaseBrowserTool):\n    name: str = \"click_element\"\n    description: str = \"Click on an element with the given CSS selector\"\n    args_schema: Type[BaseModel] = ClickToolInput\n    visible_only: bool = True\n    \"\"\"Whether to consider only visible elements.\"\"\"\n    playwright_strict: bool = False\n    \"\"\"Whether to employ Playwright's strict mode when clicking on elements.\"\"\"\n    playwright_timeout: float = 1_000\n    \"\"\"Timeout (in ms) for Playwright to wait for element to be ready.\"\"\"\n    def _selector_effective(self, selector: str) -> str:\n        if not self.visible_only:\n            return selector\n        return f\"{selector} >> visible=1\"\n    def _run(\n        self,\n        selector: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        # Navigate to the desired webpage before using this tool", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/click.html"}277{"id": "bd3bd66536a0-1", "text": "# Navigate to the desired webpage before using this tool\n        selector_effective = self._selector_effective(selector=selector)\n        from playwright.sync_api import TimeoutError as PlaywrightTimeoutError\n        try:\n            page.click(\n                selector_effective,\n                strict=self.playwright_strict,\n                timeout=self.playwright_timeout,\n            )\n        except PlaywrightTimeoutError:\n            return f\"Unable to click on element '{selector}'\"\n        return f\"Clicked element '{selector}'\"\n    async def _arun(\n        self,\n        selector: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        # Navigate to the desired webpage before using this tool\n        selector_effective = self._selector_effective(selector=selector)\n        from playwright.async_api import TimeoutError as PlaywrightTimeoutError\n        try:\n            await page.click(\n                selector_effective,\n                strict=self.playwright_strict,\n                timeout=self.playwright_timeout,\n            )\n        except PlaywrightTimeoutError:\n            return f\"Unable to click on element '{selector}'\"\n        return f\"Clicked element '{selector}'\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/click.html"}278{"id": "727634890b3a-0", "text": "Source code for langchain.tools.playwright.extract_text\nfrom __future__ import annotations\nfrom typing import Optional, Type\nfrom pydantic import BaseModel, root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import aget_current_page, get_current_page\n[docs]class ExtractTextTool(BaseBrowserTool):\n    name: str = \"extract_text\"\n    description: str = \"Extract all the text on the current webpage\"\n    args_schema: Type[BaseModel] = BaseModel\n    @root_validator\n    def check_acheck_bs_importrgs(cls, values: dict) -> dict:\n        \"\"\"Check that the arguments are valid.\"\"\"\n        try:\n            from bs4 import BeautifulSoup  # noqa: F401\n        except ImportError:\n            raise ValueError(\n                \"The 'beautifulsoup4' package is required to use this tool.\"\n                \" Please install it with 'pip install beautifulsoup4'.\"\n            )\n        return values\n    def _run(self, run_manager: Optional[CallbackManagerForToolRun] = None) -> str:\n        \"\"\"Use the tool.\"\"\"\n        # Use Beautiful Soup since it's faster than looping through the elements\n        from bs4 import BeautifulSoup\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        html_content = page.content()\n        # Parse the HTML content with BeautifulSoup\n        soup = BeautifulSoup(html_content, \"lxml\")\n        return \" \".join(text for text in soup.stripped_strings)\n    async def _arun(\n        self, run_manager: Optional[AsyncCallbackManagerForToolRun] = None", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/extract_text.html"}279{"id": "727634890b3a-1", "text": "self, run_manager: Optional[AsyncCallbackManagerForToolRun] = None\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        # Use Beautiful Soup since it's faster than looping through the elements\n        from bs4 import BeautifulSoup\n        page = await aget_current_page(self.async_browser)\n        html_content = await page.content()\n        # Parse the HTML content with BeautifulSoup\n        soup = BeautifulSoup(html_content, \"lxml\")\n        return \" \".join(text for text in soup.stripped_strings)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/extract_text.html"}280{"id": "6f3e6ebd7945-0", "text": "Source code for langchain.tools.playwright.get_elements\nfrom __future__ import annotations\nimport json\nfrom typing import TYPE_CHECKING, List, Optional, Sequence, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.playwright.base import BaseBrowserTool\nfrom langchain.tools.playwright.utils import aget_current_page, get_current_page\nif TYPE_CHECKING:\n    from playwright.async_api import Page as AsyncPage\n    from playwright.sync_api import Page as SyncPage\nclass GetElementsToolInput(BaseModel):\n    \"\"\"Input for GetElementsTool.\"\"\"\n    selector: str = Field(\n        ...,\n        description=\"CSS selector, such as '*', 'div', 'p', 'a', #id, .classname\",\n    )\n    attributes: List[str] = Field(\n        default_factory=lambda: [\"innerText\"],\n        description=\"Set of attributes to retrieve for each element\",\n    )\nasync def _aget_elements(\n    page: AsyncPage, selector: str, attributes: Sequence[str]\n) -> List[dict]:\n    \"\"\"Get elements matching the given CSS selector.\"\"\"\n    elements = await page.query_selector_all(selector)\n    results = []\n    for element in elements:\n        result = {}\n        for attribute in attributes:\n            if attribute == \"innerText\":\n                val: Optional[str] = await element.inner_text()\n            else:\n                val = await element.get_attribute(attribute)\n            if val is not None and val.strip() != \"\":\n                result[attribute] = val\n        if result:\n            results.append(result)\n    return results\ndef _get_elements(\n    page: SyncPage, selector: str, attributes: Sequence[str]\n) -> List[dict]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/get_elements.html"}281{"id": "6f3e6ebd7945-1", "text": ") -> List[dict]:\n    \"\"\"Get elements matching the given CSS selector.\"\"\"\n    elements = page.query_selector_all(selector)\n    results = []\n    for element in elements:\n        result = {}\n        for attribute in attributes:\n            if attribute == \"innerText\":\n                val: Optional[str] = element.inner_text()\n            else:\n                val = element.get_attribute(attribute)\n            if val is not None and val.strip() != \"\":\n                result[attribute] = val\n        if result:\n            results.append(result)\n    return results\n[docs]class GetElementsTool(BaseBrowserTool):\n    name: str = \"get_elements\"\n    description: str = (\n        \"Retrieve elements in the current web page matching the given CSS selector\"\n    )\n    args_schema: Type[BaseModel] = GetElementsToolInput\n    def _run(\n        self,\n        selector: str,\n        attributes: Sequence[str] = [\"innerText\"],\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.sync_browser is None:\n            raise ValueError(f\"Synchronous browser not provided to {self.name}\")\n        page = get_current_page(self.sync_browser)\n        # Navigate to the desired webpage before using this tool\n        results = _get_elements(page, selector, attributes)\n        return json.dumps(results, ensure_ascii=False)\n    async def _arun(\n        self,\n        selector: str,\n        attributes: Sequence[str] = [\"innerText\"],\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        if self.async_browser is None:\n            raise ValueError(f\"Asynchronous browser not provided to {self.name}\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/get_elements.html"}282{"id": "6f3e6ebd7945-2", "text": "raise ValueError(f\"Asynchronous browser not provided to {self.name}\")\n        page = await aget_current_page(self.async_browser)\n        # Navigate to the desired webpage before using this tool\n        results = await _aget_elements(page, selector, attributes)\n        return json.dumps(results, ensure_ascii=False)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/playwright/get_elements.html"}283{"id": "4932086af754-0", "text": "Source code for langchain.tools.youtube.search\n\"\"\"\nAdapted from https://github.com/venuv/langchain_yt_tools\nCustomYTSearchTool searches YouTube videos related to a person\nand returns a specified number of video URLs.\nInput to this tool should be a comma separated list,\n - the first part contains a person name\n - and the second(optional) a number that is the\n    maximum number of video results to return\n \"\"\"\nimport json\nfrom typing import Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools import BaseTool\n[docs]class YouTubeSearchTool(BaseTool):\n    name = \"YouTubeSearch\"\n    description = (\n        \"search for youtube videos associated with a person. \"\n        \"the input to this tool should be a comma separated list, \"\n        \"the first part contains a person name and the second a \"\n        \"number that is the maximum number of video results \"\n        \"to return aka num_results. the second part is optional\"\n    )\n    def _search(self, person: str, num_results: int) -> str:\n        from youtube_search import YoutubeSearch\n        results = YoutubeSearch(person, num_results).to_json()\n        data = json.loads(results)\n        url_suffix_list = [video[\"url_suffix\"] for video in data[\"videos\"]]\n        return str(url_suffix_list)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        values = query.split(\",\")\n        person = values[0]\n        if len(values) > 1:\n            num_results = int(values[1])\n        else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/youtube/search.html"}284{"id": "4932086af754-1", "text": "num_results = int(values[1])\n        else:\n            num_results = 2\n        return self._search(person, num_results)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"YouTubeSearchTool  does not yet support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/youtube/search.html"}285{"id": "249a8376736c-0", "text": "Source code for langchain.tools.openapi.utils.api_models\n\"\"\"Pydantic models for parsing an OpenAPI spec.\"\"\"\nimport logging\nfrom enum import Enum\nfrom typing import Any, Dict, List, Optional, Sequence, Tuple, Type, Union\nfrom openapi_schema_pydantic import MediaType, Parameter, Reference, RequestBody, Schema\nfrom pydantic import BaseModel, Field\nfrom langchain.tools.openapi.utils.openapi_utils import HTTPVerb, OpenAPISpec\nlogger = logging.getLogger(__name__)\nPRIMITIVE_TYPES = {\n    \"integer\": int,\n    \"number\": float,\n    \"string\": str,\n    \"boolean\": bool,\n    \"array\": List,\n    \"object\": Dict,\n    \"null\": None,\n}\n# See https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md#parameterIn\n# for more info.\nclass APIPropertyLocation(Enum):\n    \"\"\"The location of the property.\"\"\"\n    QUERY = \"query\"\n    PATH = \"path\"\n    HEADER = \"header\"\n    COOKIE = \"cookie\"  # Not yet supported\n    @classmethod\n    def from_str(cls, location: str) -> \"APIPropertyLocation\":\n        \"\"\"Parse an APIPropertyLocation.\"\"\"\n        try:\n            return cls(location)\n        except ValueError:\n            raise ValueError(\n                f\"Invalid APIPropertyLocation. Valid values are {cls.__members__}\"\n            )\n_SUPPORTED_MEDIA_TYPES = (\"application/json\",)\nSUPPORTED_LOCATIONS = {\n    APIPropertyLocation.QUERY,\n    APIPropertyLocation.PATH,\n}\nINVALID_LOCATION_TEMPL = (\n    'Unsupported APIPropertyLocation \"{location}\"'\n    \" for parameter {name}. \"\n    + f\"Valid values are {[loc.value for loc in SUPPORTED_LOCATIONS]}\"\n)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}286{"id": "249a8376736c-1", "text": "+ f\"Valid values are {[loc.value for loc in SUPPORTED_LOCATIONS]}\"\n)\nSCHEMA_TYPE = Union[str, Type, tuple, None, Enum]\nclass APIPropertyBase(BaseModel):\n    \"\"\"Base model for an API property.\"\"\"\n    # The name of the parameter is required and is case sensitive.\n    # If \"in\" is \"path\", the \"name\" field must correspond to a template expression\n    # within the path field in the Paths Object.\n    # If \"in\" is \"header\" and the \"name\" field is \"Accept\", \"Content-Type\",\n    # or \"Authorization\", the parameter definition is ignored.\n    # For all other cases, the \"name\" corresponds to the parameter\n    # name used by the \"in\" property.\n    name: str = Field(alias=\"name\")\n    \"\"\"The name of the property.\"\"\"\n    required: bool = Field(alias=\"required\")\n    \"\"\"Whether the property is required.\"\"\"\n    type: SCHEMA_TYPE = Field(alias=\"type\")\n    \"\"\"The type of the property.\n    \n    Either a primitive type, a component/parameter type,\n    or an array or 'object' (dict) of the above.\"\"\"\n    default: Optional[Any] = Field(alias=\"default\", default=None)\n    \"\"\"The default value of the property.\"\"\"\n    description: Optional[str] = Field(alias=\"description\", default=None)\n    \"\"\"The description of the property.\"\"\"\nclass APIProperty(APIPropertyBase):\n    \"\"\"A model for a property in the query, path, header, or cookie params.\"\"\"\n    location: APIPropertyLocation = Field(alias=\"location\")\n    \"\"\"The path/how it's being passed to the endpoint.\"\"\"\n    @staticmethod\n    def _cast_schema_list_type(schema: Schema) -> Optional[Union[str, Tuple[str, ...]]]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}287{"id": "249a8376736c-2", "text": "type_ = schema.type\n        if not isinstance(type_, list):\n            return type_\n        else:\n            return tuple(type_)\n    @staticmethod\n    def _get_schema_type_for_enum(parameter: Parameter, schema: Schema) -> Enum:\n        \"\"\"Get the schema type when the parameter is an enum.\"\"\"\n        param_name = f\"{parameter.name}Enum\"\n        return Enum(param_name, {str(v): v for v in schema.enum})\n    @staticmethod\n    def _get_schema_type_for_array(\n        schema: Schema,\n    ) -> Optional[Union[str, Tuple[str, ...]]]:\n        items = schema.items\n        if isinstance(items, Schema):\n            schema_type = APIProperty._cast_schema_list_type(items)\n        elif isinstance(items, Reference):\n            ref_name = items.ref.split(\"/\")[-1]\n            schema_type = ref_name  # TODO: Add ref definitions to make his valid\n        else:\n            raise ValueError(f\"Unsupported array items: {items}\")\n        if isinstance(schema_type, str):\n            # TODO: recurse\n            schema_type = (schema_type,)\n        return schema_type\n    @staticmethod\n    def _get_schema_type(parameter: Parameter, schema: Optional[Schema]) -> SCHEMA_TYPE:\n        if schema is None:\n            return None\n        schema_type: SCHEMA_TYPE = APIProperty._cast_schema_list_type(schema)\n        if schema_type == \"array\":\n            schema_type = APIProperty._get_schema_type_for_array(schema)\n        elif schema_type == \"object\":\n            # TODO: Resolve array and object types to components.\n            raise NotImplementedError(\"Objects not yet supported\")\n        elif schema_type in PRIMITIVE_TYPES:\n            if schema.enum:\n                schema_type = APIProperty._get_schema_type_for_enum(parameter, schema)\n            else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}288{"id": "249a8376736c-3", "text": "schema_type = APIProperty._get_schema_type_for_enum(parameter, schema)\n            else:\n                # Directly use the primitive type\n                pass\n        else:\n            raise NotImplementedError(f\"Unsupported type: {schema_type}\")\n        return schema_type\n    @staticmethod\n    def _validate_location(location: APIPropertyLocation, name: str) -> None:\n        if location not in SUPPORTED_LOCATIONS:\n            raise NotImplementedError(\n                INVALID_LOCATION_TEMPL.format(location=location, name=name)\n            )\n    @staticmethod\n    def _validate_content(content: Optional[Dict[str, MediaType]]) -> None:\n        if content:\n            raise ValueError(\n                \"API Properties with media content not supported. \"\n                \"Media content only supported within APIRequestBodyProperty's\"\n            )\n    @staticmethod\n    def _get_schema(parameter: Parameter, spec: OpenAPISpec) -> Optional[Schema]:\n        schema = parameter.param_schema\n        if isinstance(schema, Reference):\n            schema = spec.get_referenced_schema(schema)\n        elif schema is None:\n            return None\n        elif not isinstance(schema, Schema):\n            raise ValueError(f\"Error dereferencing schema: {schema}\")\n        return schema\n    @staticmethod\n    def is_supported_location(location: str) -> bool:\n        \"\"\"Return whether the provided location is supported.\"\"\"\n        try:\n            return APIPropertyLocation.from_str(location) in SUPPORTED_LOCATIONS\n        except ValueError:\n            return False\n    @classmethod\n    def from_parameter(cls, parameter: Parameter, spec: OpenAPISpec) -> \"APIProperty\":\n        \"\"\"Instantiate from an OpenAPI Parameter.\"\"\"\n        location = APIPropertyLocation.from_str(parameter.param_in)\n        cls._validate_location(\n            location,\n            parameter.name,\n        )\n        cls._validate_content(parameter.content)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}289{"id": "249a8376736c-4", "text": "location,\n            parameter.name,\n        )\n        cls._validate_content(parameter.content)\n        schema = cls._get_schema(parameter, spec)\n        schema_type = cls._get_schema_type(parameter, schema)\n        default_val = schema.default if schema is not None else None\n        return cls(\n            name=parameter.name,\n            location=location,\n            default=default_val,\n            description=parameter.description,\n            required=parameter.required,\n            type=schema_type,\n        )\nclass APIRequestBodyProperty(APIPropertyBase):\n    \"\"\"A model for a request body property.\"\"\"\n    properties: List[\"APIRequestBodyProperty\"] = Field(alias=\"properties\")\n    \"\"\"The sub-properties of the property.\"\"\"\n    # This is useful for handling nested property cycles.\n    # We can define separate types in that case.\n    references_used: List[str] = Field(alias=\"references_used\")\n    \"\"\"The references used by the property.\"\"\"\n    @classmethod\n    def _process_object_schema(\n        cls, schema: Schema, spec: OpenAPISpec, references_used: List[str]\n    ) -> Tuple[Union[str, List[str], None], List[\"APIRequestBodyProperty\"]]:\n        properties = []\n        required_props = schema.required or []\n        if schema.properties is None:\n            raise ValueError(\n                f\"No properties found when processing object schema: {schema}\"\n            )\n        for prop_name, prop_schema in schema.properties.items():\n            if isinstance(prop_schema, Reference):\n                ref_name = prop_schema.ref.split(\"/\")[-1]\n                if ref_name not in references_used:\n                    references_used.append(ref_name)\n                    prop_schema = spec.get_referenced_schema(prop_schema)\n                else:\n                    continue\n            properties.append(\n                cls.from_schema(\n                    schema=prop_schema,\n                    name=prop_name,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}290{"id": "249a8376736c-5", "text": "cls.from_schema(\n                    schema=prop_schema,\n                    name=prop_name,\n                    required=prop_name in required_props,\n                    spec=spec,\n                    references_used=references_used,\n                )\n            )\n        return schema.type, properties\n    @classmethod\n    def _process_array_schema(\n        cls, schema: Schema, name: str, spec: OpenAPISpec, references_used: List[str]\n    ) -> str:\n        items = schema.items\n        if items is not None:\n            if isinstance(items, Reference):\n                ref_name = items.ref.split(\"/\")[-1]\n                if ref_name not in references_used:\n                    references_used.append(ref_name)\n                    items = spec.get_referenced_schema(items)\n                else:\n                    pass\n                return f\"Array<{ref_name}>\"\n            else:\n                pass\n            if isinstance(items, Schema):\n                array_type = cls.from_schema(\n                    schema=items,\n                    name=f\"{name}Item\",\n                    required=True,  # TODO: Add required\n                    spec=spec,\n                    references_used=references_used,\n                )\n                return f\"Array<{array_type.type}>\"\n        return \"array\"\n    @classmethod\n    def from_schema(\n        cls,\n        schema: Schema,\n        name: str,\n        required: bool,\n        spec: OpenAPISpec,\n        references_used: Optional[List[str]] = None,\n    ) -> \"APIRequestBodyProperty\":\n        \"\"\"Recursively populate from an OpenAPI Schema.\"\"\"\n        if references_used is None:\n            references_used = []\n        schema_type = schema.type\n        properties: List[APIRequestBodyProperty] = []\n        if schema_type == \"object\" and schema.properties:\n            schema_type, properties = cls._process_object_schema(", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}291{"id": "249a8376736c-6", "text": "schema_type, properties = cls._process_object_schema(\n                schema, spec, references_used\n            )\n        elif schema_type == \"array\":\n            schema_type = cls._process_array_schema(schema, name, spec, references_used)\n        elif schema_type in PRIMITIVE_TYPES:\n            # Use the primitive type directly\n            pass\n        elif schema_type is None:\n            # No typing specified/parsed. WIll map to 'any'\n            pass\n        else:\n            raise ValueError(f\"Unsupported type: {schema_type}\")\n        return cls(\n            name=name,\n            required=required,\n            type=schema_type,\n            default=schema.default,\n            description=schema.description,\n            properties=properties,\n            references_used=references_used,\n        )\nclass APIRequestBody(BaseModel):\n    \"\"\"A model for a request body.\"\"\"\n    description: Optional[str] = Field(alias=\"description\")\n    \"\"\"The description of the request body.\"\"\"\n    properties: List[APIRequestBodyProperty] = Field(alias=\"properties\")\n    # E.g., application/json - we only support JSON at the moment.\n    media_type: str = Field(alias=\"media_type\")\n    \"\"\"The media type of the request body.\"\"\"\n    @classmethod\n    def _process_supported_media_type(\n        cls,\n        media_type_obj: MediaType,\n        spec: OpenAPISpec,\n    ) -> List[APIRequestBodyProperty]:\n        \"\"\"Process the media type of the request body.\"\"\"\n        references_used = []\n        schema = media_type_obj.media_type_schema\n        if isinstance(schema, Reference):\n            references_used.append(schema.ref.split(\"/\")[-1])\n            schema = spec.get_referenced_schema(schema)\n        if schema is None:\n            raise ValueError(\n                f\"Could not resolve schema for media type: {media_type_obj}\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}292{"id": "249a8376736c-7", "text": "f\"Could not resolve schema for media type: {media_type_obj}\"\n            )\n        api_request_body_properties = []\n        required_properties = schema.required or []\n        if schema.type == \"object\" and schema.properties:\n            for prop_name, prop_schema in schema.properties.items():\n                if isinstance(prop_schema, Reference):\n                    prop_schema = spec.get_referenced_schema(prop_schema)\n                api_request_body_properties.append(\n                    APIRequestBodyProperty.from_schema(\n                        schema=prop_schema,\n                        name=prop_name,\n                        required=prop_name in required_properties,\n                        spec=spec,\n                    )\n                )\n        else:\n            api_request_body_properties.append(\n                APIRequestBodyProperty(\n                    name=\"body\",\n                    required=True,\n                    type=schema.type,\n                    default=schema.default,\n                    description=schema.description,\n                    properties=[],\n                    references_used=references_used,\n                )\n            )\n        return api_request_body_properties\n    @classmethod\n    def from_request_body(\n        cls, request_body: RequestBody, spec: OpenAPISpec\n    ) -> \"APIRequestBody\":\n        \"\"\"Instantiate from an OpenAPI RequestBody.\"\"\"\n        properties = []\n        for media_type, media_type_obj in request_body.content.items():\n            if media_type not in _SUPPORTED_MEDIA_TYPES:\n                continue\n            api_request_body_properties = cls._process_supported_media_type(\n                media_type_obj,\n                spec,\n            )\n            properties.extend(api_request_body_properties)\n        return cls(\n            description=request_body.description,\n            properties=properties,\n            media_type=media_type,\n        )\n[docs]class APIOperation(BaseModel):\n    \"\"\"A model for a single API operation.\"\"\"\n    operation_id: str = Field(alias=\"operation_id\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}293{"id": "249a8376736c-8", "text": "operation_id: str = Field(alias=\"operation_id\")\n    \"\"\"The unique identifier of the operation.\"\"\"\n    description: Optional[str] = Field(alias=\"description\")\n    \"\"\"The description of the operation.\"\"\"\n    base_url: str = Field(alias=\"base_url\")\n    \"\"\"The base URL of the operation.\"\"\"\n    path: str = Field(alias=\"path\")\n    \"\"\"The path of the operation.\"\"\"\n    method: HTTPVerb = Field(alias=\"method\")\n    \"\"\"The HTTP method of the operation.\"\"\"\n    properties: Sequence[APIProperty] = Field(alias=\"properties\")\n    # TODO: Add parse in used components to be able to specify what type of\n    # referenced object it is.\n    # \"\"\"The properties of the operation.\"\"\"\n    # components: Dict[str, BaseModel] = Field(alias=\"components\")\n    request_body: Optional[APIRequestBody] = Field(alias=\"request_body\")\n    \"\"\"The request body of the operation.\"\"\"\n    @staticmethod\n    def _get_properties_from_parameters(\n        parameters: List[Parameter], spec: OpenAPISpec\n    ) -> List[APIProperty]:\n        \"\"\"Get the properties of the operation.\"\"\"\n        properties = []\n        for param in parameters:\n            if APIProperty.is_supported_location(param.param_in):\n                properties.append(APIProperty.from_parameter(param, spec))\n            elif param.required:\n                raise ValueError(\n                    INVALID_LOCATION_TEMPL.format(\n                        location=param.param_in, name=param.name\n                    )\n                )\n            else:\n                logger.warning(\n                    INVALID_LOCATION_TEMPL.format(\n                        location=param.param_in, name=param.name\n                    )\n                    + \" Ignoring optional parameter\"\n                )\n                pass\n        return properties\n[docs]    @classmethod\n    def from_openapi_url(\n        cls,\n        spec_url: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}294{"id": "249a8376736c-9", "text": "def from_openapi_url(\n        cls,\n        spec_url: str,\n        path: str,\n        method: str,\n    ) -> \"APIOperation\":\n        \"\"\"Create an APIOperation from an OpenAPI URL.\"\"\"\n        spec = OpenAPISpec.from_url(spec_url)\n        return cls.from_openapi_spec(spec, path, method)\n[docs]    @classmethod\n    def from_openapi_spec(\n        cls,\n        spec: OpenAPISpec,\n        path: str,\n        method: str,\n    ) -> \"APIOperation\":\n        \"\"\"Create an APIOperation from an OpenAPI spec.\"\"\"\n        operation = spec.get_operation(path, method)\n        parameters = spec.get_parameters_for_operation(operation)\n        properties = cls._get_properties_from_parameters(parameters, spec)\n        operation_id = OpenAPISpec.get_cleaned_operation_id(operation, path, method)\n        request_body = spec.get_request_body_for_operation(operation)\n        api_request_body = (\n            APIRequestBody.from_request_body(request_body, spec)\n            if request_body is not None\n            else None\n        )\n        description = operation.description or operation.summary\n        if not description and spec.paths is not None:\n            description = spec.paths[path].description or spec.paths[path].summary\n        return cls(\n            operation_id=operation_id,\n            description=description,\n            base_url=spec.base_url,\n            path=path,\n            method=method,\n            properties=properties,\n            request_body=api_request_body,\n        )\n[docs]    @staticmethod\n    def ts_type_from_python(type_: SCHEMA_TYPE) -> str:\n        if type_ is None:\n            # TODO: Handle Nones better. These often result when\n            # parsing specs that are < v3\n            return \"any\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}295{"id": "249a8376736c-10", "text": "# parsing specs that are < v3\n            return \"any\"\n        elif isinstance(type_, str):\n            return {\n                \"str\": \"string\",\n                \"integer\": \"number\",\n                \"float\": \"number\",\n                \"date-time\": \"string\",\n            }.get(type_, type_)\n        elif isinstance(type_, tuple):\n            return f\"Array<{APIOperation.ts_type_from_python(type_[0])}>\"\n        elif isinstance(type_, type) and issubclass(type_, Enum):\n            return \" | \".join([f\"'{e.value}'\" for e in type_])\n        else:\n            return str(type_)\n    def _format_nested_properties(\n        self, properties: List[APIRequestBodyProperty], indent: int = 2\n    ) -> str:\n        \"\"\"Format nested properties.\"\"\"\n        formatted_props = []\n        for prop in properties:\n            prop_name = prop.name\n            prop_type = self.ts_type_from_python(prop.type)\n            prop_required = \"\" if prop.required else \"?\"\n            prop_desc = f\"/* {prop.description} */\" if prop.description else \"\"\n            if prop.properties:\n                nested_props = self._format_nested_properties(\n                    prop.properties, indent + 2\n                )\n                prop_type = f\"{{\\n{nested_props}\\n{' ' * indent}}}\"\n            formatted_props.append(\n                f\"{prop_desc}\\n{' ' * indent}{prop_name}{prop_required}: {prop_type},\"\n            )\n        return \"\\n\".join(formatted_props)\n[docs]    def to_typescript(self) -> str:\n        \"\"\"Get typescript string representation of the operation.\"\"\"\n        operation_name = self.operation_id\n        params = []\n        if self.request_body:\n            formatted_request_body_props = self._format_nested_properties(", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}296{"id": "249a8376736c-11", "text": "if self.request_body:\n            formatted_request_body_props = self._format_nested_properties(\n                self.request_body.properties\n            )\n            params.append(formatted_request_body_props)\n        for prop in self.properties:\n            prop_name = prop.name\n            prop_type = self.ts_type_from_python(prop.type)\n            prop_required = \"\" if prop.required else \"?\"\n            prop_desc = f\"/* {prop.description} */\" if prop.description else \"\"\n            params.append(f\"{prop_desc}\\n\\t\\t{prop_name}{prop_required}: {prop_type},\")\n        formatted_params = \"\\n\".join(params).strip()\n        description_str = f\"/* {self.description} */\" if self.description else \"\"\n        typescript_definition = f\"\"\"\n{description_str}\ntype {operation_name} = (_: {{\n{formatted_params}\n}}) => any;\n\"\"\"\n        return typescript_definition.strip()\n    @property\n    def query_params(self) -> List[str]:\n        return [\n            property.name\n            for property in self.properties\n            if property.location == APIPropertyLocation.QUERY\n        ]\n    @property\n    def path_params(self) -> List[str]:\n        return [\n            property.name\n            for property in self.properties\n            if property.location == APIPropertyLocation.PATH\n        ]\n    @property\n    def body_params(self) -> List[str]:\n        if self.request_body is None:\n            return []\n        return [prop.name for prop in self.request_body.properties]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/api_models.html"}297{"id": "2ae81e0b731f-0", "text": "Source code for langchain.tools.openapi.utils.openapi_utils\n\"\"\"Utility functions for parsing an OpenAPI spec.\"\"\"\nimport copy\nimport json\nimport logging\nimport re\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import Dict, List, Optional, Union\nimport requests\nimport yaml\nfrom openapi_schema_pydantic import (\n    Components,\n    OpenAPI,\n    Operation,\n    Parameter,\n    PathItem,\n    Paths,\n    Reference,\n    RequestBody,\n    Schema,\n)\nfrom pydantic import ValidationError\nlogger = logging.getLogger(__name__)\nclass HTTPVerb(str, Enum):\n    \"\"\"HTTP verbs.\"\"\"\n    GET = \"get\"\n    PUT = \"put\"\n    POST = \"post\"\n    DELETE = \"delete\"\n    OPTIONS = \"options\"\n    HEAD = \"head\"\n    PATCH = \"patch\"\n    TRACE = \"trace\"\n    @classmethod\n    def from_str(cls, verb: str) -> \"HTTPVerb\":\n        \"\"\"Parse an HTTP verb.\"\"\"\n        try:\n            return cls(verb)\n        except ValueError:\n            raise ValueError(f\"Invalid HTTP verb. Valid values are {cls.__members__}\")\n[docs]class OpenAPISpec(OpenAPI):\n    \"\"\"OpenAPI Model that removes misformatted parts of the spec.\"\"\"\n    @property\n    def _paths_strict(self) -> Paths:\n        if not self.paths:\n            raise ValueError(\"No paths found in spec\")\n        return self.paths\n    def _get_path_strict(self, path: str) -> PathItem:\n        path_item = self._paths_strict.get(path)\n        if not path_item:\n            raise ValueError(f\"No path found for {path}\")\n        return path_item\n    @property\n    def _components_strict(self) -> Components:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}298{"id": "2ae81e0b731f-1", "text": "return path_item\n    @property\n    def _components_strict(self) -> Components:\n        \"\"\"Get components or err.\"\"\"\n        if self.components is None:\n            raise ValueError(\"No components found in spec. \")\n        return self.components\n    @property\n    def _parameters_strict(self) -> Dict[str, Union[Parameter, Reference]]:\n        \"\"\"Get parameters or err.\"\"\"\n        parameters = self._components_strict.parameters\n        if parameters is None:\n            raise ValueError(\"No parameters found in spec. \")\n        return parameters\n    @property\n    def _schemas_strict(self) -> Dict[str, Schema]:\n        \"\"\"Get the dictionary of schemas or err.\"\"\"\n        schemas = self._components_strict.schemas\n        if schemas is None:\n            raise ValueError(\"No schemas found in spec. \")\n        return schemas\n    @property\n    def _request_bodies_strict(self) -> Dict[str, Union[RequestBody, Reference]]:\n        \"\"\"Get the request body or err.\"\"\"\n        request_bodies = self._components_strict.requestBodies\n        if request_bodies is None:\n            raise ValueError(\"No request body found in spec. \")\n        return request_bodies\n    def _get_referenced_parameter(self, ref: Reference) -> Union[Parameter, Reference]:\n        \"\"\"Get a parameter (or nested reference) or err.\"\"\"\n        ref_name = ref.ref.split(\"/\")[-1]\n        parameters = self._parameters_strict\n        if ref_name not in parameters:\n            raise ValueError(f\"No parameter found for {ref_name}\")\n        return parameters[ref_name]\n    def _get_root_referenced_parameter(self, ref: Reference) -> Parameter:\n        \"\"\"Get the root reference or err.\"\"\"\n        parameter = self._get_referenced_parameter(ref)\n        while isinstance(parameter, Reference):", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}299{"id": "2ae81e0b731f-2", "text": "parameter = self._get_referenced_parameter(ref)\n        while isinstance(parameter, Reference):\n            parameter = self._get_referenced_parameter(parameter)\n        return parameter\n[docs]    def get_referenced_schema(self, ref: Reference) -> Schema:\n        \"\"\"Get a schema (or nested reference) or err.\"\"\"\n        ref_name = ref.ref.split(\"/\")[-1]\n        schemas = self._schemas_strict\n        if ref_name not in schemas:\n            raise ValueError(f\"No schema found for {ref_name}\")\n        return schemas[ref_name]\n    def _get_root_referenced_schema(self, ref: Reference) -> Schema:\n        \"\"\"Get the root reference or err.\"\"\"\n        schema = self.get_referenced_schema(ref)\n        while isinstance(schema, Reference):\n            schema = self.get_referenced_schema(schema)\n        return schema\n    def _get_referenced_request_body(\n        self, ref: Reference\n    ) -> Optional[Union[Reference, RequestBody]]:\n        \"\"\"Get a request body (or nested reference) or err.\"\"\"\n        ref_name = ref.ref.split(\"/\")[-1]\n        request_bodies = self._request_bodies_strict\n        if ref_name not in request_bodies:\n            raise ValueError(f\"No request body found for {ref_name}\")\n        return request_bodies[ref_name]\n    def _get_root_referenced_request_body(\n        self, ref: Reference\n    ) -> Optional[RequestBody]:\n        \"\"\"Get the root request Body or err.\"\"\"\n        request_body = self._get_referenced_request_body(ref)\n        while isinstance(request_body, Reference):\n            request_body = self._get_referenced_request_body(request_body)\n        return request_body\n    @staticmethod\n    def _alert_unsupported_spec(obj: dict) -> None:\n        \"\"\"Alert if the spec is not supported.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}300{"id": "2ae81e0b731f-3", "text": "\"\"\"Alert if the spec is not supported.\"\"\"\n        warning_message = (\n            \" This may result in degraded performance.\"\n            + \" Convert your OpenAPI spec to 3.1.* spec\"\n            + \" for better support.\"\n        )\n        swagger_version = obj.get(\"swagger\")\n        openapi_version = obj.get(\"openapi\")\n        if isinstance(openapi_version, str):\n            if openapi_version != \"3.1.0\":\n                logger.warning(\n                    f\"Attempting to load an OpenAPI {openapi_version}\"\n                    f\" spec. {warning_message}\"\n                )\n            else:\n                pass\n        elif isinstance(swagger_version, str):\n            logger.warning(\n                f\"Attempting to load a Swagger {swagger_version}\"\n                f\" spec. {warning_message}\"\n            )\n        else:\n            raise ValueError(\n                \"Attempting to load an unsupported spec:\"\n                f\"\\n\\n{obj}\\n{warning_message}\"\n            )\n[docs]    @classmethod\n    def parse_obj(cls, obj: dict) -> \"OpenAPISpec\":\n        try:\n            cls._alert_unsupported_spec(obj)\n            return super().parse_obj(obj)\n        except ValidationError as e:\n            # We are handling possibly misconfigured specs and want to do a best-effort\n            # job to get a reasonable interface out of it.\n            new_obj = copy.deepcopy(obj)\n            for error in e.errors():\n                keys = error[\"loc\"]\n                item = new_obj\n                for key in keys[:-1]:\n                    item = item[key]\n                item.pop(keys[-1], None)\n            return cls.parse_obj(new_obj)\n[docs]    @classmethod\n    def from_spec_dict(cls, spec_dict: dict) -> \"OpenAPISpec\":", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}301{"id": "2ae81e0b731f-4", "text": "def from_spec_dict(cls, spec_dict: dict) -> \"OpenAPISpec\":\n        \"\"\"Get an OpenAPI spec from a dict.\"\"\"\n        return cls.parse_obj(spec_dict)\n[docs]    @classmethod\n    def from_text(cls, text: str) -> \"OpenAPISpec\":\n        \"\"\"Get an OpenAPI spec from a text.\"\"\"\n        try:\n            spec_dict = json.loads(text)\n        except json.JSONDecodeError:\n            spec_dict = yaml.safe_load(text)\n        return cls.from_spec_dict(spec_dict)\n[docs]    @classmethod\n    def from_file(cls, path: Union[str, Path]) -> \"OpenAPISpec\":\n        \"\"\"Get an OpenAPI spec from a file path.\"\"\"\n        path_ = path if isinstance(path, Path) else Path(path)\n        if not path_.exists():\n            raise FileNotFoundError(f\"{path} does not exist\")\n        with path_.open(\"r\") as f:\n            return cls.from_text(f.read())\n[docs]    @classmethod\n    def from_url(cls, url: str) -> \"OpenAPISpec\":\n        \"\"\"Get an OpenAPI spec from a URL.\"\"\"\n        response = requests.get(url)\n        return cls.from_text(response.text)\n    @property\n    def base_url(self) -> str:\n        \"\"\"Get the base url.\"\"\"\n        return self.servers[0].url\n[docs]    def get_methods_for_path(self, path: str) -> List[str]:\n        \"\"\"Return a list of valid methods for the specified path.\"\"\"\n        path_item = self._get_path_strict(path)\n        results = []\n        for method in HTTPVerb:\n            operation = getattr(path_item, method.value, None)\n            if isinstance(operation, Operation):\n                results.append(method.value)\n        return results", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}302{"id": "2ae81e0b731f-5", "text": "if isinstance(operation, Operation):\n                results.append(method.value)\n        return results\n[docs]    def get_operation(self, path: str, method: str) -> Operation:\n        \"\"\"Get the operation object for a given path and HTTP method.\"\"\"\n        path_item = self._get_path_strict(path)\n        operation_obj = getattr(path_item, method, None)\n        if not isinstance(operation_obj, Operation):\n            raise ValueError(f\"No {method} method found for {path}\")\n        return operation_obj\n[docs]    def get_parameters_for_operation(self, operation: Operation) -> List[Parameter]:\n        \"\"\"Get the components for a given operation.\"\"\"\n        parameters = []\n        if operation.parameters:\n            for parameter in operation.parameters:\n                if isinstance(parameter, Reference):\n                    parameter = self._get_root_referenced_parameter(parameter)\n                parameters.append(parameter)\n        return parameters\n[docs]    def get_request_body_for_operation(\n        self, operation: Operation\n    ) -> Optional[RequestBody]:\n        \"\"\"Get the request body for a given operation.\"\"\"\n        request_body = operation.requestBody\n        if isinstance(request_body, Reference):\n            request_body = self._get_root_referenced_request_body(request_body)\n        return request_body\n[docs]    @staticmethod\n    def get_cleaned_operation_id(operation: Operation, path: str, method: str) -> str:\n        \"\"\"Get a cleaned operation id from an operation id.\"\"\"\n        operation_id = operation.operationId\n        if operation_id is None:\n            # Replace all punctuation of any kind with underscore\n            path = re.sub(r\"[^a-zA-Z0-9]\", \"_\", path.lstrip(\"/\"))\n            operation_id = f\"{path}_{method}\"\n        return operation_id.replace(\"-\", \"_\").replace(\".\", \"_\").replace(\"/\", \"_\")\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}303{"id": "2ae81e0b731f-6", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openapi/utils/openapi_utils.html"}304{"id": "cf2955f9f87c-0", "text": "Source code for langchain.tools.google_search.tool\n\"\"\"Tool for the Google search API.\"\"\"\nfrom typing import Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.google_search import GoogleSearchAPIWrapper\n[docs]class GoogleSearchRun(BaseTool):\n    \"\"\"Tool that adds the capability to query the Google search API.\"\"\"\n    name = \"Google Search\"\n    description = (\n        \"A wrapper around Google Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: GoogleSearchAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"GoogleSearchRun does not support async\")\n[docs]class GoogleSearchResults(BaseTool):\n    \"\"\"Tool that has capability to query the Google Search API and get back json.\"\"\"\n    name = \"Google Search Results JSON\"\n    description = (\n        \"A wrapper around Google Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query. Output is a JSON array of the query results\"\n    )\n    num_results: int = 4\n    api_wrapper: GoogleSearchAPIWrapper\n    def _run(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/google_search/tool.html"}305{"id": "cf2955f9f87c-1", "text": "api_wrapper: GoogleSearchAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return str(self.api_wrapper.results(query, self.num_results))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"GoogleSearchRun does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/google_search/tool.html"}306{"id": "991190c75301-0", "text": "Source code for langchain.tools.wolfram_alpha.tool\n\"\"\"Tool for the Wolfram Alpha API.\"\"\"\nfrom typing import Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.wolfram_alpha import WolframAlphaAPIWrapper\n[docs]class WolframAlphaQueryRun(BaseTool):\n    \"\"\"Tool that adds the capability to query using the Wolfram Alpha SDK.\"\"\"\n    name = \"Wolfram Alpha\"\n    description = (\n        \"A wrapper around Wolfram Alpha. \"\n        \"Useful for when you need to answer questions about Math, \"\n        \"Science, Technology, Culture, Society and Everyday Life. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: WolframAlphaAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the WolframAlpha tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the WolframAlpha tool asynchronously.\"\"\"\n        raise NotImplementedError(\"WolframAlphaQueryRun does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/wolfram_alpha/tool.html"}307{"id": "f35963c0a8ba-0", "text": "Source code for langchain.tools.ddg_search.tool\n\"\"\"Tool for the DuckDuckGo search API.\"\"\"\nimport warnings\nfrom typing import Any, Optional\nfrom pydantic import Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.duckduckgo_search import DuckDuckGoSearchAPIWrapper\n[docs]class DuckDuckGoSearchRun(BaseTool):\n    \"\"\"Tool that adds the capability to query the DuckDuckGo search API.\"\"\"\n    name = \"DuckDuckGo Search\"\n    description = (\n        \"A wrapper around DuckDuckGo Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: DuckDuckGoSearchAPIWrapper = Field(\n        default_factory=DuckDuckGoSearchAPIWrapper\n    )\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"DuckDuckGoSearch does not support async\")\n[docs]class DuckDuckGoSearchResults(BaseTool):\n    \"\"\"Tool that queries the Duck Duck Go Search API and get back json.\"\"\"\n    name = \"DuckDuckGo Results JSON\"\n    description = (\n        \"A wrapper around Duck Duck Go Search. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/ddg_search/tool.html"}308{"id": "f35963c0a8ba-1", "text": "description = (\n        \"A wrapper around Duck Duck Go Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query. Output is a JSON array of the query results\"\n    )\n    num_results: int = 4\n    api_wrapper: DuckDuckGoSearchAPIWrapper = Field(\n        default_factory=DuckDuckGoSearchAPIWrapper\n    )\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return str(self.api_wrapper.results(query, self.num_results))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"DuckDuckGoSearchResults does not support async\")\ndef DuckDuckGoSearchTool(*args: Any, **kwargs: Any) -> DuckDuckGoSearchRun:\n    warnings.warn(\n        \"DuckDuckGoSearchTool will be deprecated in the future. \"\n        \"Please use DuckDuckGoSearchRun instead.\",\n        DeprecationWarning,\n    )\n    return DuckDuckGoSearchRun(*args, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/ddg_search/tool.html"}309{"id": "f2ad2ae0af4d-0", "text": "Source code for langchain.tools.bing_search.tool\n\"\"\"Tool for the Bing search API.\"\"\"\nfrom typing import Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.bing_search import BingSearchAPIWrapper\n[docs]class BingSearchRun(BaseTool):\n    \"\"\"Tool that adds the capability to query the Bing search API.\"\"\"\n    name = \"Bing Search\"\n    description = (\n        \"A wrapper around Bing Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: BingSearchAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"BingSearchRun does not support async\")\n[docs]class BingSearchResults(BaseTool):\n    \"\"\"Tool that has capability to query the Bing Search API and get back json.\"\"\"\n    name = \"Bing Search Results JSON\"\n    description = (\n        \"A wrapper around Bing Search. \"\n        \"Useful for when you need to answer questions about current events. \"\n        \"Input should be a search query. Output is a JSON array of the query results\"\n    )\n    num_results: int = 4\n    api_wrapper: BingSearchAPIWrapper\n    def _run(", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/bing_search/tool.html"}310{"id": "f2ad2ae0af4d-1", "text": "api_wrapper: BingSearchAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return str(self.api_wrapper.results(query, self.num_results))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"BingSearchResults does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/bing_search/tool.html"}311{"id": "eb756f17b59d-0", "text": "Source code for langchain.tools.powerbi.tool\n\"\"\"Tools for interacting with a Power BI dataset.\"\"\"\nfrom typing import Any, Dict, Optional, Tuple\nfrom pydantic import Field, validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.chains.llm import LLMChain\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.powerbi.prompt import (\n    BAD_REQUEST_RESPONSE,\n    DEFAULT_FEWSHOT_EXAMPLES,\n    QUESTION_TO_QUERY,\n    RETRY_RESPONSE,\n)\nfrom langchain.utilities.powerbi import PowerBIDataset, json_to_md\n[docs]class QueryPowerBITool(BaseTool):\n    \"\"\"Tool for querying a Power BI Dataset.\"\"\"\n    name = \"query_powerbi\"\n    description = \"\"\"\n    Input to this tool is a detailed question about the dataset, output is a result from the dataset. It will try to answer the question using the dataset, and if it cannot, it will ask for clarification.\n    Example Input: \"How many rows are in table1?\"\n    \"\"\"  # noqa: E501\n    llm_chain: LLMChain\n    powerbi: PowerBIDataset = Field(exclude=True)\n    template: Optional[str] = QUESTION_TO_QUERY\n    examples: Optional[str] = DEFAULT_FEWSHOT_EXAMPLES\n    session_cache: Dict[str, Any] = Field(default_factory=dict, exclude=True)\n    max_iterations: int = 5\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @validator(\"llm_chain\")\n    def validate_llm_chain_input_variables(  # pylint: disable=E0213\n        cls, llm_chain: LLMChain\n    ) -> LLMChain:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}312{"id": "eb756f17b59d-1", "text": "cls, llm_chain: LLMChain\n    ) -> LLMChain:\n        \"\"\"Make sure the LLM chain has the correct input variables.\"\"\"\n        if llm_chain.prompt.input_variables != [\n            \"tool_input\",\n            \"tables\",\n            \"schemas\",\n            \"examples\",\n        ]:\n            raise ValueError(\n                \"LLM chain for QueryPowerBITool must have input variables ['tool_input', 'tables', 'schemas', 'examples'], found %s\",  # noqa: C0301 E501 # pylint: disable=C0301\n                llm_chain.prompt.input_variables,\n            )\n        return llm_chain\n    def _check_cache(self, tool_input: str) -> Optional[str]:\n        \"\"\"Check if the input is present in the cache.\n        If the value is a bad request, overwrite with the escalated version,\n        if not present return None.\"\"\"\n        if tool_input not in self.session_cache:\n            return None\n        return self.session_cache[tool_input]\n    def _run(\n        self,\n        tool_input: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> str:\n        \"\"\"Execute the query, return the results or an error message.\"\"\"\n        if cache := self._check_cache(tool_input):\n            return cache\n        try:\n            query = self.llm_chain.predict(\n                tool_input=tool_input,\n                tables=self.powerbi.get_table_names(),\n                schemas=self.powerbi.get_schemas(),\n                examples=self.examples,\n            )\n        except Exception as exc:  # pylint: disable=broad-except\n            self.session_cache[tool_input] = f\"Error on call to LLM: {exc}\"\n            return self.session_cache[tool_input]", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}313{"id": "eb756f17b59d-2", "text": "return self.session_cache[tool_input]\n        if query == \"I cannot answer this\":\n            self.session_cache[tool_input] = query\n            return self.session_cache[tool_input]\n        pbi_result = self.powerbi.run(command=query)\n        result, error = self._parse_output(pbi_result)\n        iterations = kwargs.get(\"iterations\", 0)\n        if error and iterations < self.max_iterations:\n            return self._run(\n                tool_input=RETRY_RESPONSE.format(\n                    tool_input=tool_input, query=query, error=error\n                ),\n                run_manager=run_manager,\n                iterations=iterations + 1,\n            )\n        self.session_cache[tool_input] = (\n            result if result else BAD_REQUEST_RESPONSE.format(error=error)\n        )\n        return self.session_cache[tool_input]\n    async def _arun(\n        self,\n        tool_input: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n        **kwargs: Any,\n    ) -> str:\n        \"\"\"Execute the query, return the results or an error message.\"\"\"\n        if cache := self._check_cache(tool_input):\n            return cache\n        try:\n            query = await self.llm_chain.apredict(\n                tool_input=tool_input,\n                tables=self.powerbi.get_table_names(),\n                schemas=self.powerbi.get_schemas(),\n                examples=self.examples,\n            )\n        except Exception as exc:  # pylint: disable=broad-except\n            self.session_cache[tool_input] = f\"Error on call to LLM: {exc}\"\n            return self.session_cache[tool_input]\n        if query == \"I cannot answer this\":\n            self.session_cache[tool_input] = query\n            return self.session_cache[tool_input]", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}314{"id": "eb756f17b59d-3", "text": "self.session_cache[tool_input] = query\n            return self.session_cache[tool_input]\n        pbi_result = await self.powerbi.arun(command=query)\n        result, error = self._parse_output(pbi_result)\n        iterations = kwargs.get(\"iterations\", 0)\n        if error and iterations < self.max_iterations:\n            return await self._arun(\n                tool_input=RETRY_RESPONSE.format(\n                    tool_input=tool_input, query=query, error=error\n                ),\n                run_manager=run_manager,\n                iterations=iterations + 1,\n            )\n        self.session_cache[tool_input] = (\n            result if result else BAD_REQUEST_RESPONSE.format(error=error)\n        )\n        return self.session_cache[tool_input]\n    def _parse_output(\n        self, pbi_result: Dict[str, Any]\n    ) -> Tuple[Optional[str], Optional[str]]:\n        \"\"\"Parse the output of the query to a markdown table.\"\"\"\n        if \"results\" in pbi_result:\n            return json_to_md(pbi_result[\"results\"][0][\"tables\"][0][\"rows\"]), None\n        if \"error\" in pbi_result:\n            if (\n                \"pbi.error\" in pbi_result[\"error\"]\n                and \"details\" in pbi_result[\"error\"][\"pbi.error\"]\n            ):\n                return None, pbi_result[\"error\"][\"pbi.error\"][\"details\"][0][\"detail\"]\n            return None, pbi_result[\"error\"]\n        return None, \"Unknown error\"\n[docs]class InfoPowerBITool(BaseTool):\n    \"\"\"Tool for getting metadata about a PowerBI Dataset.\"\"\"\n    name = \"schema_powerbi\"\n    description = \"\"\"\n    Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}315{"id": "eb756f17b59d-4", "text": "Be sure that the tables actually exist by calling list_tables_powerbi first!\n    Example Input: \"table1, table2, table3\"\n    \"\"\"  # noqa: E501\n    powerbi: PowerBIDataset = Field(exclude=True)\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    def _run(\n        self,\n        tool_input: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Get the schema for tables in a comma-separated list.\"\"\"\n        return self.powerbi.get_table_info(tool_input.split(\", \"))\n    async def _arun(\n        self,\n        tool_input: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        return await self.powerbi.aget_table_info(tool_input.split(\", \"))\n[docs]class ListPowerBITool(BaseTool):\n    \"\"\"Tool for getting tables names.\"\"\"\n    name = \"list_tables_powerbi\"\n    description = \"Input is an empty string, output is a comma separated list of tables in the database.\"  # noqa: E501 # pylint: disable=C0301\n    powerbi: PowerBIDataset = Field(exclude=True)\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    def _run(\n        self,\n        tool_input: Optional[str] = None,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Get the names of the tables.\"\"\"\n        return \", \".join(self.powerbi.get_table_names())\n    async def _arun(\n        self,\n        tool_input: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}316{"id": "eb756f17b59d-5", "text": "self,\n        tool_input: Optional[str] = None,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Get the names of the tables.\"\"\"\n        return \", \".join(self.powerbi.get_table_names())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/powerbi/tool.html"}317{"id": "1417cd8ec3fe-0", "text": "Source code for langchain.tools.steamship_image_generation.tool\n\"\"\"This tool allows agents to generate images using Steamship.\nSteamship offers access to different third party image generation APIs\nusing a single API key.\nToday the following models are supported:\n- Dall-E\n- Stable Diffusion\nTo use this tool, you must first set as environment variables:\n    STEAMSHIP_API_KEY\n```\n\"\"\"\nfrom __future__ import annotations\nfrom enum import Enum\nfrom typing import TYPE_CHECKING, Dict, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools import BaseTool\nfrom langchain.tools.steamship_image_generation.utils import make_image_public\nfrom langchain.utils import get_from_dict_or_env\nif TYPE_CHECKING:\n    pass\nclass ModelName(str, Enum):\n    \"\"\"Supported Image Models for generation.\"\"\"\n    DALL_E = \"dall-e\"\n    STABLE_DIFFUSION = \"stable-diffusion\"\nSUPPORTED_IMAGE_SIZES = {\n    ModelName.DALL_E: (\"256x256\", \"512x512\", \"1024x1024\"),\n    ModelName.STABLE_DIFFUSION: (\"512x512\", \"768x768\"),\n}\n[docs]class SteamshipImageGenerationTool(BaseTool):\n    try:\n        from steamship import Steamship\n    except ImportError:\n        pass\n    \"\"\"Tool used to generate images from a text-prompt.\"\"\"\n    model_name: ModelName\n    size: Optional[str] = \"512x512\"\n    steamship: Steamship\n    return_urls: Optional[bool] = False\n    name = \"GenerateImage\"\n    description = (\n        \"Useful for when you need to generate an image.\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/steamship_image_generation/tool.html"}318{"id": "1417cd8ec3fe-1", "text": "description = (\n        \"Useful for when you need to generate an image.\"\n        \"Input: A detailed text-2-image prompt describing an image\"\n        \"Output: the UUID of a generated image\"\n    )\n    @root_validator(pre=True)\n    def validate_size(cls, values: Dict) -> Dict:\n        if \"size\" in values:\n            size = values[\"size\"]\n            model_name = values[\"model_name\"]\n            if size not in SUPPORTED_IMAGE_SIZES[model_name]:\n                raise RuntimeError(f\"size {size} is not supported by {model_name}\")\n        return values\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        steamship_api_key = get_from_dict_or_env(\n            values, \"steamship_api_key\", \"STEAMSHIP_API_KEY\"\n        )\n        try:\n            from steamship import Steamship\n        except ImportError:\n            raise ImportError(\n                \"steamship is not installed. \"\n                \"Please install it with `pip install steamship`\"\n            )\n        steamship = Steamship(\n            api_key=steamship_api_key,\n        )\n        values[\"steamship\"] = steamship\n        if \"steamship_api_key\" in values:\n            del values[\"steamship_api_key\"]\n        return values\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        image_generator = self.steamship.use_plugin(\n            plugin_handle=self.model_name.value, config={\"n\": 1, \"size\": self.size}\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/steamship_image_generation/tool.html"}319{"id": "1417cd8ec3fe-2", "text": ")\n        task = image_generator.generate(text=query, append_output_to_file=True)\n        task.wait()\n        blocks = task.output.blocks\n        if len(blocks) > 0:\n            if self.return_urls:\n                return make_image_public(self.steamship, blocks[0])\n            else:\n                return blocks[0].id\n        raise RuntimeError(f\"[{self.name}] Tool unable to generate image!\")\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"GenerateImageTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/steamship_image_generation/tool.html"}320{"id": "85a03f1bf76f-0", "text": "Source code for langchain.tools.scenexplain.tool\n\"\"\"Tool for the SceneXplain API.\"\"\"\nfrom typing import Optional\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.scenexplain import SceneXplainAPIWrapper\nclass SceneXplainInput(BaseModel):\n    \"\"\"Input for SceneXplain.\"\"\"\n    query: str = Field(..., description=\"The link to the image to explain\")\n[docs]class SceneXplainTool(BaseTool):\n    \"\"\"Tool that adds the capability to explain images.\"\"\"\n    name = \"Image Explainer\"\n    description = (\n        \"An Image Captioning Tool: Use this tool to generate a detailed caption \"\n        \"for an image. The input can be an image file of any format, and \"\n        \"the output will be a text description that covers every detail of the image.\"\n    )\n    api_wrapper: SceneXplainAPIWrapper = Field(default_factory=SceneXplainAPIWrapper)\n    def _run(\n        self, query: str, run_manager: Optional[CallbackManagerForToolRun] = None\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return self.api_wrapper.run(query)\n    async def _arun(\n        self, query: str, run_manager: Optional[AsyncCallbackManagerForToolRun] = None\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"SceneXplainTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/scenexplain/tool.html"}321{"id": "793822bd8fce-0", "text": "Source code for langchain.tools.human.tool\n\"\"\"Tool for asking human input.\"\"\"\nfrom typing import Callable, Optional\nfrom pydantic import Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\ndef _print_func(text: str) -> None:\n    print(\"\\n\")\n    print(text)\n[docs]class HumanInputRun(BaseTool):\n    \"\"\"Tool that adds the capability to ask user for input.\"\"\"\n    name = \"Human\"\n    description = (\n        \"You can ask a human for guidance when you think you \"\n        \"got stuck or you are not sure what to do next. \"\n        \"The input should be a question for the human.\"\n    )\n    prompt_func: Callable[[str], None] = Field(default_factory=lambda: _print_func)\n    input_func: Callable = Field(default_factory=lambda: input)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Human input tool.\"\"\"\n        self.prompt_func(query)\n        return self.input_func()\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the Human tool asynchronously.\"\"\"\n        raise NotImplementedError(\"Human tool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/human/tool.html"}322{"id": "b3df77e13877-0", "text": "Source code for langchain.tools.openweathermap.tool\n\"\"\"Tool for the OpenWeatherMap API.\"\"\"\nfrom typing import Optional\nfrom pydantic import Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities import OpenWeatherMapAPIWrapper\n[docs]class OpenWeatherMapQueryRun(BaseTool):\n    \"\"\"Tool that adds the capability to query using the OpenWeatherMap API.\"\"\"\n    api_wrapper: OpenWeatherMapAPIWrapper = Field(\n        default_factory=OpenWeatherMapAPIWrapper\n    )\n    name = \"OpenWeatherMap\"\n    description = (\n        \"A wrapper around OpenWeatherMap API. \"\n        \"Useful for fetching current weather information for a specified location. \"\n        \"Input should be a location string (e.g. London,GB).\"\n    )\n    def _run(\n        self, location: str, run_manager: Optional[CallbackManagerForToolRun] = None\n    ) -> str:\n        \"\"\"Use the OpenWeatherMap tool.\"\"\"\n        return self.api_wrapper.run(location)\n    async def _arun(\n        self,\n        location: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the OpenWeatherMap tool asynchronously.\"\"\"\n        raise NotImplementedError(\"OpenWeatherMapQueryRun does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/openweathermap/tool.html"}323{"id": "b3b7aabd296d-0", "text": "Source code for langchain.tools.metaphor_search.tool\n\"\"\"Tool for the Metaphor search API.\"\"\"\nfrom typing import Dict, List, Optional, Union\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.metaphor_search import MetaphorSearchAPIWrapper\n[docs]class MetaphorSearchResults(BaseTool):\n    \"\"\"Tool that has capability to query the Metaphor Search API and get back json.\"\"\"\n    name = \"Metaphor Search Results JSON\"\n    description = (\n        \"A wrapper around Metaphor Search. \"\n        \"Input should be a Metaphor-optimized query. \"\n        \"Output is a JSON array of the query results\"\n    )\n    api_wrapper: MetaphorSearchAPIWrapper\n    def _run(\n        self,\n        query: str,\n        num_results: int,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> Union[List[Dict], str]:\n        \"\"\"Use the tool.\"\"\"\n        try:\n            return self.api_wrapper.results(query, num_results)\n        except Exception as e:\n            return repr(e)\n    async def _arun(\n        self,\n        query: str,\n        num_results: int,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> Union[List[Dict], str]:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        try:\n            return await self.api_wrapper.results_async(query, num_results)\n        except Exception as e:\n            return repr(e)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/metaphor_search/tool.html"}324{"id": "0bca7510fea9-0", "text": "Source code for langchain.tools.gmail.get_thread\nfrom typing import Dict, Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.gmail.base import GmailBaseTool\nclass GetThreadSchema(BaseModel):\n    # From https://support.google.com/mail/answer/7190?hl=en\n    thread_id: str = Field(\n        ...,\n        description=\"The thread ID.\",\n    )\n[docs]class GmailGetThread(GmailBaseTool):\n    name: str = \"get_gmail_thread\"\n    description: str = (\n        \"Use this tool to search for email messages.\"\n        \" The input must be a valid Gmail query.\"\n        \" The output is a JSON list of messages.\"\n    )\n    args_schema: Type[GetThreadSchema] = GetThreadSchema\n    def _run(\n        self,\n        thread_id: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> Dict:\n        \"\"\"Run the tool.\"\"\"\n        query = self.api_resource.users().threads().get(userId=\"me\", id=thread_id)\n        thread_data = query.execute()\n        if not isinstance(thread_data, dict):\n            raise ValueError(\"The output of the query must be a list.\")\n        messages = thread_data[\"messages\"]\n        thread_data[\"messages\"] = []\n        keys_to_keep = [\"id\", \"snippet\", \"snippet\"]\n        # TODO: Parse body.\n        for message in messages:\n            thread_data[\"messages\"].append(\n                {k: message[k] for k in keys_to_keep if k in message}\n            )\n        return thread_data\n    async def _arun(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/get_thread.html"}325{"id": "0bca7510fea9-1", "text": ")\n        return thread_data\n    async def _arun(\n        self,\n        thread_id: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> Dict:\n        \"\"\"Run the tool.\"\"\"\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/get_thread.html"}326{"id": "cf3527ff0108-0", "text": "Source code for langchain.tools.gmail.send_message\n\"\"\"Send Gmail messages.\"\"\"\nimport base64\nfrom email.mime.multipart import MIMEMultipart\nfrom email.mime.text import MIMEText\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.gmail.base import GmailBaseTool\nclass SendMessageSchema(BaseModel):\n    message: str = Field(\n        ...,\n        description=\"The message to send.\",\n    )\n    to: List[str] = Field(\n        ...,\n        description=\"The list of recipients.\",\n    )\n    subject: str = Field(\n        ...,\n        description=\"The subject of the message.\",\n    )\n    cc: Optional[List[str]] = Field(\n        None,\n        description=\"The list of CC recipients.\",\n    )\n    bcc: Optional[List[str]] = Field(\n        None,\n        description=\"The list of BCC recipients.\",\n    )\n[docs]class GmailSendMessage(GmailBaseTool):\n    name: str = \"send_gmail_message\"\n    description: str = (\n        \"Use this tool to send email messages.\" \" The input is the message, recipents\"\n    )\n    def _prepare_message(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n    ) -> Dict[str, Any]:\n        \"\"\"Create a message for an email.\"\"\"\n        mime_message = MIMEMultipart()\n        mime_message.attach(MIMEText(message, \"html\"))\n        mime_message[\"To\"] = \", \".join(to)", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/send_message.html"}327{"id": "cf3527ff0108-1", "text": "mime_message[\"To\"] = \", \".join(to)\n        mime_message[\"Subject\"] = subject\n        if cc is not None:\n            mime_message[\"Cc\"] = \", \".join(cc)\n        if bcc is not None:\n            mime_message[\"Bcc\"] = \", \".join(bcc)\n        encoded_message = base64.urlsafe_b64encode(mime_message.as_bytes()).decode()\n        return {\"raw\": encoded_message}\n    def _run(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Run the tool.\"\"\"\n        try:\n            create_message = self._prepare_message(message, to, subject, cc=cc, bcc=bcc)\n            send_message = (\n                self.api_resource.users()\n                .messages()\n                .send(userId=\"me\", body=create_message)\n            )\n            sent_message = send_message.execute()\n            return f'Message sent. Message Id: {sent_message[\"id\"]}'\n        except Exception as error:\n            raise Exception(f\"An error occurred: {error}\")\n    async def _arun(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Run the tool asynchronously.\"\"\"\n        raise NotImplementedError(f\"The tool {self.name} does not support async yet.\")\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/send_message.html"}328{"id": "cf3527ff0108-2", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/send_message.html"}329{"id": "051d16f6119b-0", "text": "Source code for langchain.tools.gmail.search\nimport base64\nimport email\nfrom enum import Enum\nfrom typing import Any, Dict, List, Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.gmail.base import GmailBaseTool\nfrom langchain.tools.gmail.utils import clean_email_body\nclass Resource(str, Enum):\n    THREADS = \"threads\"\n    MESSAGES = \"messages\"\nclass SearchArgsSchema(BaseModel):\n    # From https://support.google.com/mail/answer/7190?hl=en\n    query: str = Field(\n        ...,\n        description=\"The Gmail query. Example filters include from:sender,\"\n        \" to:recipient, subject:subject, -filtered_term,\"\n        \" in:folder, is:important|read|starred, after:year/mo/date, \"\n        \"before:year/mo/date, label:label_name\"\n        ' \"exact phrase\".'\n        \" Search newer/older than using d (day), m (month), and y (year): \"\n        \"newer_than:2d, older_than:1y.\"\n        \" Attachments with extension example: filename:pdf. Multiple term\"\n        \" matching example: from:amy OR from:david.\",\n    )\n    resource: Resource = Field(\n        default=Resource.MESSAGES,\n        description=\"Whether to search for threads or messages.\",\n    )\n    max_results: int = Field(\n        default=10,\n        description=\"The maximum number of results to return.\",\n    )\n[docs]class GmailSearch(GmailBaseTool):\n    name: str = \"search_gmail\"\n    description: str = (", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/search.html"}330{"id": "051d16f6119b-1", "text": "name: str = \"search_gmail\"\n    description: str = (\n        \"Use this tool to search for email messages or threads.\"\n        \" The input must be a valid Gmail query.\"\n        \" The output is a JSON list of the requested resource.\"\n    )\n    args_schema: Type[SearchArgsSchema] = SearchArgsSchema\n    def _parse_threads(self, threads: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n        # Add the thread message snippets to the thread results\n        results = []\n        for thread in threads:\n            thread_id = thread[\"id\"]\n            thread_data = (\n                self.api_resource.users()\n                .threads()\n                .get(userId=\"me\", id=thread_id)\n                .execute()\n            )\n            messages = thread_data[\"messages\"]\n            thread[\"messages\"] = []\n            for message in messages:\n                snippet = message[\"snippet\"]\n                thread[\"messages\"].append({\"snippet\": snippet, \"id\": message[\"id\"]})\n            results.append(thread)\n        return results\n    def _parse_messages(self, messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n        results = []\n        for message in messages:\n            message_id = message[\"id\"]\n            message_data = (\n                self.api_resource.users()\n                .messages()\n                .get(userId=\"me\", format=\"raw\", id=message_id)\n                .execute()\n            )\n            raw_message = base64.urlsafe_b64decode(message_data[\"raw\"])\n            email_msg = email.message_from_bytes(raw_message)\n            subject = email_msg[\"Subject\"]\n            sender = email_msg[\"From\"]\n            message_body = email_msg.get_payload()\n            body = clean_email_body(message_body)\n            results.append(\n                {", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/search.html"}331{"id": "051d16f6119b-2", "text": "body = clean_email_body(message_body)\n            results.append(\n                {\n                    \"id\": message[\"id\"],\n                    \"threadId\": message_data[\"threadId\"],\n                    \"snippet\": message_data[\"snippet\"],\n                    \"body\": body,\n                    \"subject\": subject,\n                    \"sender\": sender,\n                }\n            )\n        return results\n    def _run(\n        self,\n        query: str,\n        resource: Resource = Resource.MESSAGES,\n        max_results: int = 10,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> List[Dict[str, Any]]:\n        \"\"\"Run the tool.\"\"\"\n        results = (\n            self.api_resource.users()\n            .messages()\n            .list(userId=\"me\", q=query, maxResults=max_results)\n            .execute()\n            .get(resource.value, [])\n        )\n        if resource == Resource.THREADS:\n            return self._parse_threads(results)\n        elif resource == Resource.MESSAGES:\n            return self._parse_messages(results)\n        else:\n            raise NotImplementedError(f\"Resource of type {resource} not implemented.\")\n    async def _arun(\n        self,\n        query: str,\n        resource: Resource = Resource.MESSAGES,\n        max_results: int = 10,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> List[Dict[str, Any]]:\n        \"\"\"Run the tool.\"\"\"\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/search.html"}332{"id": "da221a340071-0", "text": "Source code for langchain.tools.gmail.get_message\nimport base64\nimport email\nfrom typing import Dict, Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.gmail.base import GmailBaseTool\nfrom langchain.tools.gmail.utils import clean_email_body\nclass SearchArgsSchema(BaseModel):\n    message_id: str = Field(\n        ...,\n        description=\"The unique ID of the email message, retrieved from a search.\",\n    )\n[docs]class GmailGetMessage(GmailBaseTool):\n    name: str = \"get_gmail_message\"\n    description: str = (\n        \"Use this tool to fetch an email by message ID.\"\n        \" Returns the thread ID, snipet, body, subject, and sender.\"\n    )\n    args_schema: Type[SearchArgsSchema] = SearchArgsSchema\n    def _run(\n        self,\n        message_id: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> Dict:\n        \"\"\"Run the tool.\"\"\"\n        query = (\n            self.api_resource.users()\n            .messages()\n            .get(userId=\"me\", format=\"raw\", id=message_id)\n        )\n        message_data = query.execute()\n        raw_message = base64.urlsafe_b64decode(message_data[\"raw\"])\n        email_msg = email.message_from_bytes(raw_message)\n        subject = email_msg[\"Subject\"]\n        sender = email_msg[\"From\"]\n        message_body = email_msg.get_payload()\n        body = clean_email_body(message_body)\n        return {\n            \"id\": message_id,\n            \"threadId\": message_data[\"threadId\"],\n            \"snippet\": message_data[\"snippet\"],", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/get_message.html"}333{"id": "da221a340071-1", "text": "\"snippet\": message_data[\"snippet\"],\n            \"body\": body,\n            \"subject\": subject,\n            \"sender\": sender,\n        }\n    async def _arun(\n        self,\n        message_id: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> Dict:\n        \"\"\"Run the tool.\"\"\"\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/get_message.html"}334{"id": "05b899c58e13-0", "text": "Source code for langchain.tools.gmail.create_draft\nimport base64\nfrom email.message import EmailMessage\nfrom typing import List, Optional, Type\nfrom pydantic import BaseModel, Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.gmail.base import GmailBaseTool\nclass CreateDraftSchema(BaseModel):\n    message: str = Field(\n        ...,\n        description=\"The message to include in the draft.\",\n    )\n    to: List[str] = Field(\n        ...,\n        description=\"The list of recipients.\",\n    )\n    subject: str = Field(\n        ...,\n        description=\"The subject of the message.\",\n    )\n    cc: Optional[List[str]] = Field(\n        None,\n        description=\"The list of CC recipients.\",\n    )\n    bcc: Optional[List[str]] = Field(\n        None,\n        description=\"The list of BCC recipients.\",\n    )\n[docs]class GmailCreateDraft(GmailBaseTool):\n    name: str = \"create_gmail_draft\"\n    description: str = (\n        \"Use this tool to create a draft email with the provided message fields.\"\n    )\n    args_schema: Type[CreateDraftSchema] = CreateDraftSchema\n    def _prepare_draft_message(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n    ) -> dict:\n        draft_message = EmailMessage()\n        draft_message.set_content(message)\n        draft_message[\"To\"] = \", \".join(to)\n        draft_message[\"Subject\"] = subject\n        if cc is not None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/create_draft.html"}335{"id": "05b899c58e13-1", "text": "draft_message[\"Subject\"] = subject\n        if cc is not None:\n            draft_message[\"Cc\"] = \", \".join(cc)\n        if bcc is not None:\n            draft_message[\"Bcc\"] = \", \".join(bcc)\n        encoded_message = base64.urlsafe_b64encode(draft_message.as_bytes()).decode()\n        return {\"message\": {\"raw\": encoded_message}}\n    def _run(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        try:\n            create_message = self._prepare_draft_message(message, to, subject, cc, bcc)\n            draft = (\n                self.api_resource.users()\n                .drafts()\n                .create(userId=\"me\", body=create_message)\n                .execute()\n            )\n            output = f'Draft created. Draft Id: {draft[\"id\"]}'\n            return output\n        except Exception as e:\n            raise Exception(f\"An error occurred: {e}\")\n    async def _arun(\n        self,\n        message: str,\n        to: List[str],\n        subject: str,\n        cc: Optional[List[str]] = None,\n        bcc: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        raise NotImplementedError(f\"The tool {self.name} does not support async yet.\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/gmail/create_draft.html"}336{"id": "c5a1dfe3c3cc-0", "text": "Source code for langchain.tools.vectorstore.tool\n\"\"\"Tools for interacting with vectorstores.\"\"\"\nimport json\nfrom typing import Any, Dict, Optional\nfrom pydantic import BaseModel, Field\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.chains import RetrievalQA, RetrievalQAWithSourcesChain\nfrom langchain.llms.openai import OpenAI\nfrom langchain.tools.base import BaseTool\nfrom langchain.vectorstores.base import VectorStore\nclass BaseVectorStoreTool(BaseModel):\n    \"\"\"Base class for tools that use a VectorStore.\"\"\"\n    vectorstore: VectorStore = Field(exclude=True)\n    llm: BaseLanguageModel = Field(default_factory=lambda: OpenAI(temperature=0))\n    class Config(BaseTool.Config):\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\ndef _create_description_from_template(values: Dict[str, Any]) -> Dict[str, Any]:\n    values[\"description\"] = values[\"template\"].format(name=values[\"name\"])\n    return values\n[docs]class VectorStoreQATool(BaseVectorStoreTool, BaseTool):\n    \"\"\"Tool for the VectorDBQA chain. To be initialized with name and chain.\"\"\"\n[docs]    @staticmethod\n    def get_description(name: str, description: str) -> str:\n        template: str = (\n            \"Useful for when you need to answer questions about {name}. \"\n            \"Whenever you need information about {description} \"\n            \"you should ALWAYS use this. \"\n            \"Input should be a fully formed question.\"\n        )\n        return template.format(name=name, description=description)\n    def _run(\n        self,\n        query: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/vectorstore/tool.html"}337{"id": "c5a1dfe3c3cc-1", "text": "def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        chain = RetrievalQA.from_chain_type(\n            self.llm, retriever=self.vectorstore.as_retriever()\n        )\n        return chain.run(query)\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"VectorStoreQATool does not support async\")\n[docs]class VectorStoreQAWithSourcesTool(BaseVectorStoreTool, BaseTool):\n    \"\"\"Tool for the VectorDBQAWithSources chain.\"\"\"\n[docs]    @staticmethod\n    def get_description(name: str, description: str) -> str:\n        template: str = (\n            \"Useful for when you need to answer questions about {name} and the sources \"\n            \"used to construct the answer. \"\n            \"Whenever you need information about {description} \"\n            \"you should ALWAYS use this. \"\n            \" Input should be a fully formed question. \"\n            \"Output is a json serialized dictionary with keys `answer` and `sources`. \"\n            \"Only use this tool if the user explicitly asks for sources.\"\n        )\n        return template.format(name=name, description=description)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        chain = RetrievalQAWithSourcesChain.from_chain_type(\n            self.llm, retriever=self.vectorstore.as_retriever()\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/vectorstore/tool.html"}338{"id": "c5a1dfe3c3cc-2", "text": "self.llm, retriever=self.vectorstore.as_retriever()\n        )\n        return json.dumps(chain({chain.question_key: query}, return_only_outputs=True))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        raise NotImplementedError(\"VectorStoreQAWithSourcesTool does not support async\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/vectorstore/tool.html"}339{"id": "20c79e9b359a-0", "text": "Source code for langchain.tools.google_serper.tool\n\"\"\"Tool for the Serper.dev Google Search API.\"\"\"\nfrom typing import Optional\nfrom pydantic.fields import Field\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForToolRun,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.google_serper import GoogleSerperAPIWrapper\n[docs]class GoogleSerperRun(BaseTool):\n    \"\"\"Tool that adds the capability to query the Serper.dev Google search API.\"\"\"\n    name = \"Google Serper\"\n    description = (\n        \"A low-cost Google Search API.\"\n        \"Useful for when you need to answer questions about current events.\"\n        \"Input should be a search query.\"\n    )\n    api_wrapper: GoogleSerperAPIWrapper\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return str(self.api_wrapper.run(query))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        return (await self.api_wrapper.arun(query)).__str__()\n[docs]class GoogleSerperResults(BaseTool):\n    \"\"\"Tool that has capability to query the Serper.dev Google Search API\n    and get back json.\"\"\"\n    name = \"Google Serrper Results JSON\"\n    description = (\n        \"A low-cost Google Search API.\"\n        \"Useful for when you need to answer questions about current events.\"\n        \"Input should be a search query. Output is a JSON object of the query results\"\n    )", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/google_serper/tool.html"}340{"id": "20c79e9b359a-1", "text": ")\n    api_wrapper: GoogleSerperAPIWrapper = Field(default_factory=GoogleSerperAPIWrapper)\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool.\"\"\"\n        return str(self.api_wrapper.results(query))\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        \"\"\"Use the tool asynchronously.\"\"\"\n        return (await self.api_wrapper.aresults(query)).__str__()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/tools/google_serper/tool.html"}341{"id": "603320a12952-0", "text": "Source code for langchain.embeddings.huggingface_hub\n\"\"\"Wrapper around HuggingFace Hub embedding models.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nDEFAULT_REPO_ID = \"sentence-transformers/all-mpnet-base-v2\"\nVALID_TASKS = (\"feature-extraction\",)\n[docs]class HuggingFaceHubEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around HuggingFaceHub embedding models.\n    To use, you should have the ``huggingface_hub`` python package installed, and the\n    environment variable ``HUGGINGFACEHUB_API_TOKEN`` set with your API token, or pass\n    it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import HuggingFaceHubEmbeddings\n            repo_id = \"sentence-transformers/all-mpnet-base-v2\"\n            hf = HuggingFaceHubEmbeddings(\n                repo_id=repo_id,\n                task=\"feature-extraction\",\n                huggingfacehub_api_token=\"my-api-key\",\n            )\n    \"\"\"\n    client: Any  #: :meta private:\n    repo_id: str = DEFAULT_REPO_ID\n    \"\"\"Model name to use.\"\"\"\n    task: Optional[str] = \"feature-extraction\"\n    \"\"\"Task to call the model with.\"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    huggingfacehub_api_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface_hub.html"}342{"id": "603320a12952-1", "text": "@root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        huggingfacehub_api_token = get_from_dict_or_env(\n            values, \"huggingfacehub_api_token\", \"HUGGINGFACEHUB_API_TOKEN\"\n        )\n        try:\n            from huggingface_hub.inference_api import InferenceApi\n            repo_id = values[\"repo_id\"]\n            if not repo_id.startswith(\"sentence-transformers\"):\n                raise ValueError(\n                    \"Currently only 'sentence-transformers' embedding models \"\n                    f\"are supported. Got invalid 'repo_id' {repo_id}.\"\n                )\n            client = InferenceApi(\n                repo_id=repo_id,\n                token=huggingfacehub_api_token,\n                task=values.get(\"task\"),\n            )\n            if client.task not in VALID_TASKS:\n                raise ValueError(\n                    f\"Got invalid task {client.task}, \"\n                    f\"currently only {VALID_TASKS} are supported\"\n                )\n            values[\"client\"] = client\n        except ImportError:\n            raise ValueError(\n                \"Could not import huggingface_hub python package. \"\n                \"Please install it with `pip install huggingface_hub`.\"\n            )\n        return values\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Call out to HuggingFaceHub's embedding endpoint for embedding search docs.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        # replace newlines, which can negatively affect performance.\n        texts = [text.replace(\"\\n\", \" \") for text in texts]", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface_hub.html"}343{"id": "603320a12952-2", "text": "texts = [text.replace(\"\\n\", \" \") for text in texts]\n        _model_kwargs = self.model_kwargs or {}\n        responses = self.client(inputs=texts, params=_model_kwargs)\n        return responses\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Call out to HuggingFaceHub's embedding endpoint for embedding query text.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        response = self.embed_documents([text])[0]\n        return response\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface_hub.html"}344{"id": "336388cb5cb1-0", "text": "Source code for langchain.embeddings.self_hosted\n\"\"\"Running custom embedding models on self-hosted remote hardware.\"\"\"\nfrom typing import Any, Callable, List\nfrom pydantic import Extra\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.llms import SelfHostedPipeline\ndef _embed_documents(pipeline: Any, *args: Any, **kwargs: Any) -> List[List[float]]:\n    \"\"\"Inference function to send to the remote hardware.\n    Accepts a sentence_transformer model_id and\n    returns a list of embeddings for each document in the batch.\n    \"\"\"\n    return pipeline(*args, **kwargs)\n[docs]class SelfHostedEmbeddings(SelfHostedPipeline, Embeddings):\n    \"\"\"Runs custom embedding models on self-hosted remote hardware.\n    Supported hardware includes auto-launched instances on AWS, GCP, Azure,\n    and Lambda, as well as servers specified\n    by IP address and SSH credentials (such as on-prem, or another\n    cloud like Paperspace, Coreweave, etc.).\n    To use, you should have the ``runhouse`` python package installed.\n    Example using a model load function:\n        .. code-block:: python\n            from langchain.embeddings import SelfHostedEmbeddings\n            from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n            import runhouse as rh\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            def get_pipeline():\n                model_id = \"facebook/bart-large\"\n                tokenizer = AutoTokenizer.from_pretrained(model_id)\n                model = AutoModelForCausalLM.from_pretrained(model_id)\n                return pipeline(\"feature-extraction\", model=model, tokenizer=tokenizer)\n            embeddings = SelfHostedEmbeddings(\n                model_load_fn=get_pipeline,\n                hardware=gpu", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted.html"}345{"id": "336388cb5cb1-1", "text": "model_load_fn=get_pipeline,\n                hardware=gpu\n                model_reqs=[\"./\", \"torch\", \"transformers\"],\n            )\n    Example passing in a pipeline path:\n        .. code-block:: python\n            from langchain.embeddings import SelfHostedHFEmbeddings\n            import runhouse as rh\n            from transformers import pipeline\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            pipeline = pipeline(model=\"bert-base-uncased\", task=\"feature-extraction\")\n            rh.blob(pickle.dumps(pipeline),\n                path=\"models/pipeline.pkl\").save().to(gpu, path=\"models\")\n            embeddings = SelfHostedHFEmbeddings.from_pipeline(\n                pipeline=\"models/pipeline.pkl\",\n                hardware=gpu,\n                model_reqs=[\"./\", \"torch\", \"transformers\"],\n            )\n    \"\"\"\n    inference_fn: Callable = _embed_documents\n    \"\"\"Inference function to extract the embeddings on the remote hardware.\"\"\"\n    inference_kwargs: Any = None\n    \"\"\"Any kwargs to pass to the model's inference function.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a HuggingFace transformer model.\n        Args:\n            texts: The list of texts to embed.s\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))\n        embeddings = self.client(self.pipeline_ref, texts)\n        if not isinstance(embeddings, list):\n            return embeddings.tolist()\n        return embeddings\n[docs]    def embed_query(self, text: str) -> List[float]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted.html"}346{"id": "336388cb5cb1-2", "text": "[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a HuggingFace transformer model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        text = text.replace(\"\\n\", \" \")\n        embeddings = self.client(self.pipeline_ref, text)\n        if not isinstance(embeddings, list):\n            return embeddings.tolist()\n        return embeddings\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted.html"}347{"id": "4206f436a0da-0", "text": "Source code for langchain.embeddings.mosaicml\n\"\"\"Wrapper around MosaicML APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional, Tuple\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\n[docs]class MosaicMLInstructorEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around MosaicML's embedding inference service.\n    To use, you should have the\n    environment variable ``MOSAICML_API_TOKEN`` set with your API token, or pass\n    it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.llms import MosaicMLInstructorEmbeddings\n            endpoint_url = (\n                \"https://models.hosted-on.mosaicml.hosting/instructor-large/v1/predict\"\n            )\n            mosaic_llm = MosaicMLInstructorEmbeddings(\n                endpoint_url=endpoint_url,\n                mosaicml_api_token=\"my-api-key\"\n            )\n    \"\"\"\n    endpoint_url: str = (\n        \"https://models.hosted-on.mosaicml.hosting/instructor-large/v1/predict\"\n    )\n    \"\"\"Endpoint URL to use.\"\"\"\n    embed_instruction: str = \"Represent the document for retrieval: \"\n    \"\"\"Instruction used to embed documents.\"\"\"\n    query_instruction: str = (\n        \"Represent the question for retrieving supporting documents: \"\n    )\n    \"\"\"Instruction used to embed the query.\"\"\"\n    retry_sleep: float = 1.0\n    \"\"\"How long to try sleeping for if a rate limit is encountered\"\"\"\n    mosaicml_api_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/mosaicml.html"}348{"id": "4206f436a0da-1", "text": "\"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        mosaicml_api_token = get_from_dict_or_env(\n            values, \"mosaicml_api_token\", \"MOSAICML_API_TOKEN\"\n        )\n        values[\"mosaicml_api_token\"] = mosaicml_api_token\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\"endpoint_url\": self.endpoint_url}\n    def _embed(\n        self, input: List[Tuple[str, str]], is_retry: bool = False\n    ) -> List[List[float]]:\n        payload = {\"input_strings\": input}\n        # HTTP headers for authorization\n        headers = {\n            \"Authorization\": f\"{self.mosaicml_api_token}\",\n            \"Content-Type\": \"application/json\",\n        }\n        # send request\n        try:\n            response = requests.post(self.endpoint_url, headers=headers, json=payload)\n        except requests.exceptions.RequestException as e:\n            raise ValueError(f\"Error raised by inference endpoint: {e}\")\n        try:\n            parsed_response = response.json()\n            if \"error\" in parsed_response:\n                # if we get rate limited, try sleeping for 1 second\n                if (\n                    not is_retry\n                    and \"rate limit exceeded\" in parsed_response[\"error\"].lower()\n                ):\n                    import time\n                    time.sleep(self.retry_sleep)\n                    return self._embed(input, is_retry=True)\n                raise ValueError(\n                    f\"Error raised by inference API: {parsed_response['error']}\"\n                )", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/mosaicml.html"}349{"id": "4206f436a0da-2", "text": "f\"Error raised by inference API: {parsed_response['error']}\"\n                )\n            if \"data\" not in parsed_response:\n                raise ValueError(\n                    f\"Error raised by inference API, no key data: {parsed_response}\"\n                )\n            embeddings = parsed_response[\"data\"]\n        except requests.exceptions.JSONDecodeError as e:\n            raise ValueError(\n                f\"Error raised by inference API: {e}.\\nResponse: {response.text}\"\n            )\n        return embeddings\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Embed documents using a MosaicML deployed instructor embedding model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        instruction_pairs = [(self.embed_instruction, text) for text in texts]\n        embeddings = self._embed(instruction_pairs)\n        return embeddings\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Embed a query using a MosaicML deployed instructor embedding model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        instruction_pair = (self.query_instruction, text)\n        embedding = self._embed([instruction_pair])[0]\n        return embedding\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/mosaicml.html"}350{"id": "713676e90011-0", "text": "Source code for langchain.embeddings.sagemaker_endpoint\n\"\"\"Wrapper around Sagemaker InvokeEndpoint API.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.llms.sagemaker_endpoint import ContentHandlerBase\nclass EmbeddingsContentHandler(ContentHandlerBase[List[str], List[List[float]]]):\n    \"\"\"Content handler for LLM class.\"\"\"\n[docs]class SagemakerEndpointEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around custom Sagemaker Inference Endpoints.\n    To use, you must supply the endpoint name from your deployed\n    Sagemaker model & the region where it is deployed.\n    To authenticate, the AWS client uses the following methods to\n    automatically load credentials:\n    https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html\n    If a specific credential profile should be used, you must pass\n    the name of the profile from the ~/.aws/credentials file that is to be used.\n    Make sure the credentials / roles used have the required policies to\n    access the Sagemaker endpoint.\n    See: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html\n    \"\"\"\n    \"\"\"\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import SagemakerEndpointEmbeddings\n            endpoint_name = (\n                \"my-endpoint-name\"\n            )\n            region_name = (\n                \"us-west-2\"\n            )\n            credentials_profile_name = (\n                \"default\"\n            )\n            se = SagemakerEndpointEmbeddings(\n                endpoint_name=endpoint_name,\n                region_name=region_name,\n                credentials_profile_name=credentials_profile_name\n            )\n    \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/sagemaker_endpoint.html"}351{"id": "713676e90011-1", "text": "credentials_profile_name=credentials_profile_name\n            )\n    \"\"\"\n    client: Any  #: :meta private:\n    endpoint_name: str = \"\"\n    \"\"\"The name of the endpoint from the deployed Sagemaker model.\n    Must be unique within an AWS Region.\"\"\"\n    region_name: str = \"\"\n    \"\"\"The aws region where the Sagemaker model is deployed, eg. `us-west-2`.\"\"\"\n    credentials_profile_name: Optional[str] = None\n    \"\"\"The name of the profile in the ~/.aws/credentials or ~/.aws/config files, which\n    has either access keys or role information specified.\n    If not specified, the default credential profile or, if on an EC2 instance,\n    credentials from IMDS will be used.\n    See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html\n    \"\"\"\n    content_handler: EmbeddingsContentHandler\n    \"\"\"The content handler class that provides an input and\n    output transform functions to handle formats between LLM\n    and the endpoint.\n    \"\"\"\n    \"\"\"\n     Example:\n        .. code-block:: python\n        from langchain.embeddings.sagemaker_endpoint import EmbeddingsContentHandler\n        class ContentHandler(EmbeddingsContentHandler):\n                content_type = \"application/json\"\n                accepts = \"application/json\"\n                def transform_input(self, prompts: List[str], model_kwargs: Dict) -> bytes:\n                    input_str = json.dumps({prompts: prompts, **model_kwargs})\n                    return input_str.encode('utf-8')\n                def transform_output(self, output: bytes) -> List[List[float]]:\n                    response_json = json.loads(output.read().decode(\"utf-8\"))\n                    return response_json[\"vectors\"]\n    \"\"\"  # noqa: E501\n    model_kwargs: Optional[Dict] = None", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/sagemaker_endpoint.html"}352{"id": "713676e90011-2", "text": "\"\"\"  # noqa: E501\n    model_kwargs: Optional[Dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    endpoint_kwargs: Optional[Dict] = None\n    \"\"\"Optional attributes passed to the invoke_endpoint\n    function. See `boto3`_. docs for more info.\n    .. _boto3: <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>\n    \"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that AWS credentials to and python package exists in environment.\"\"\"\n        try:\n            import boto3\n            try:\n                if values[\"credentials_profile_name\"] is not None:\n                    session = boto3.Session(\n                        profile_name=values[\"credentials_profile_name\"]\n                    )\n                else:\n                    # use default credentials\n                    session = boto3.Session()\n                values[\"client\"] = session.client(\n                    \"sagemaker-runtime\", region_name=values[\"region_name\"]\n                )\n            except Exception as e:\n                raise ValueError(\n                    \"Could not load credentials to authenticate with AWS client. \"\n                    \"Please check that credentials in the specified \"\n                    \"profile name are valid.\"\n                ) from e\n        except ImportError:\n            raise ValueError(\n                \"Could not import boto3 python package. \"\n                \"Please install it with `pip install boto3`.\"\n            )\n        return values\n    def _embedding_func(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Call out to SageMaker Inference embedding endpoint.\"\"\"\n        # replace newlines, which can negatively affect performance.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/sagemaker_endpoint.html"}353{"id": "713676e90011-3", "text": "# replace newlines, which can negatively affect performance.\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))\n        _model_kwargs = self.model_kwargs or {}\n        _endpoint_kwargs = self.endpoint_kwargs or {}\n        body = self.content_handler.transform_input(texts, _model_kwargs)\n        content_type = self.content_handler.content_type\n        accepts = self.content_handler.accepts\n        # send request\n        try:\n            response = self.client.invoke_endpoint(\n                EndpointName=self.endpoint_name,\n                Body=body,\n                ContentType=content_type,\n                Accept=accepts,\n                **_endpoint_kwargs,\n            )\n        except Exception as e:\n            raise ValueError(f\"Error raised by inference endpoint: {e}\")\n        return self.content_handler.transform_output(response[\"Body\"])\n[docs]    def embed_documents(\n        self, texts: List[str], chunk_size: int = 64\n    ) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a SageMaker Inference Endpoint.\n        Args:\n            texts: The list of texts to embed.\n            chunk_size: The chunk size defines how many input texts will\n                be grouped together as request. If None, will use the\n                chunk size specified by the class.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        results = []\n        _chunk_size = len(texts) if chunk_size > len(texts) else chunk_size\n        for i in range(0, len(texts), _chunk_size):\n            response = self._embedding_func(texts[i : i + _chunk_size])\n            results.extend(response)\n        return results\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a SageMaker inference endpoint.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/sagemaker_endpoint.html"}354{"id": "713676e90011-4", "text": "\"\"\"Compute query embeddings using a SageMaker inference endpoint.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        return self._embedding_func([text])[0]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/sagemaker_endpoint.html"}355{"id": "37c22bae8f69-0", "text": "Source code for langchain.embeddings.tensorflow_hub\n\"\"\"Wrapper around TensorflowHub embedding models.\"\"\"\nfrom typing import Any, List\nfrom pydantic import BaseModel, Extra\nfrom langchain.embeddings.base import Embeddings\nDEFAULT_MODEL_URL = \"https://tfhub.dev/google/universal-sentence-encoder-multilingual/3\"\n[docs]class TensorflowHubEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around tensorflow_hub embedding models.\n    To use, you should have the ``tensorflow_text`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import TensorflowHubEmbeddings\n            url = \"https://tfhub.dev/google/universal-sentence-encoder-multilingual/3\"\n            tf = TensorflowHubEmbeddings(model_url=url)\n    \"\"\"\n    embed: Any  #: :meta private:\n    model_url: str = DEFAULT_MODEL_URL\n    \"\"\"Model name to use.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the tensorflow_hub and tensorflow_text.\"\"\"\n        super().__init__(**kwargs)\n        try:\n            import tensorflow_hub\n        except ImportError:\n            raise ImportError(\n                \"Could not import tensorflow-hub python package. \"\n                \"Please install it with `pip install tensorflow-hub``.\"\n            )\n        try:\n            import tensorflow_text  # noqa\n        except ImportError:\n            raise ImportError(\n                \"Could not import tensorflow_text python package. \"\n                \"Please install it with `pip install tensorflow_text``.\"\n            )\n        self.embed = tensorflow_hub.load(self.model_url)\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/tensorflow_hub.html"}356{"id": "37c22bae8f69-1", "text": "[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a TensorflowHub embedding model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))\n        embeddings = self.embed(texts).numpy()\n        return embeddings.tolist()\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a TensorflowHub embedding model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        text = text.replace(\"\\n\", \" \")\n        embedding = self.embed([text]).numpy()[0]\n        return embedding.tolist()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/tensorflow_hub.html"}357{"id": "5a13324b0a36-0", "text": "Source code for langchain.embeddings.self_hosted_hugging_face\n\"\"\"Wrapper around HuggingFace embedding models for self-hosted remote hardware.\"\"\"\nimport importlib\nimport logging\nfrom typing import Any, Callable, List, Optional\nfrom langchain.embeddings.self_hosted import SelfHostedEmbeddings\nDEFAULT_MODEL_NAME = \"sentence-transformers/all-mpnet-base-v2\"\nDEFAULT_INSTRUCT_MODEL = \"hkunlp/instructor-large\"\nDEFAULT_EMBED_INSTRUCTION = \"Represent the document for retrieval: \"\nDEFAULT_QUERY_INSTRUCTION = (\n    \"Represent the question for retrieving supporting documents: \"\n)\nlogger = logging.getLogger(__name__)\ndef _embed_documents(client: Any, *args: Any, **kwargs: Any) -> List[List[float]]:\n    \"\"\"Inference function to send to the remote hardware.\n    Accepts a sentence_transformer model_id and\n    returns a list of embeddings for each document in the batch.\n    \"\"\"\n    return client.encode(*args, **kwargs)\ndef load_embedding_model(model_id: str, instruct: bool = False, device: int = 0) -> Any:\n    \"\"\"Load the embedding model.\"\"\"\n    if not instruct:\n        import sentence_transformers\n        client = sentence_transformers.SentenceTransformer(model_id)\n    else:\n        from InstructorEmbedding import INSTRUCTOR\n        client = INSTRUCTOR(model_id)\n    if importlib.util.find_spec(\"torch\") is not None:\n        import torch\n        cuda_device_count = torch.cuda.device_count()\n        if device < -1 or (device >= cuda_device_count):\n            raise ValueError(\n                f\"Got device=={device}, \"\n                f\"device is required to be within [-1, {cuda_device_count})\"\n            )\n        if device < 0 and cuda_device_count > 0:\n            logger.warning(", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted_hugging_face.html"}358{"id": "5a13324b0a36-1", "text": "if device < 0 and cuda_device_count > 0:\n            logger.warning(\n                \"Device has %d GPUs available. \"\n                \"Provide device={deviceId} to `from_model_id` to use available\"\n                \"GPUs for execution. deviceId is -1 for CPU and \"\n                \"can be a positive integer associated with CUDA device id.\",\n                cuda_device_count,\n            )\n        client = client.to(device)\n    return client\n[docs]class SelfHostedHuggingFaceEmbeddings(SelfHostedEmbeddings):\n    \"\"\"Runs sentence_transformers embedding models on self-hosted remote hardware.\n    Supported hardware includes auto-launched instances on AWS, GCP, Azure,\n    and Lambda, as well as servers specified\n    by IP address and SSH credentials (such as on-prem, or another cloud\n    like Paperspace, Coreweave, etc.).\n    To use, you should have the ``runhouse`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import SelfHostedHuggingFaceEmbeddings\n            import runhouse as rh\n            model_name = \"sentence-transformers/all-mpnet-base-v2\"\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            hf = SelfHostedHuggingFaceEmbeddings(model_name=model_name, hardware=gpu)\n    \"\"\"\n    client: Any  #: :meta private:\n    model_id: str = DEFAULT_MODEL_NAME\n    \"\"\"Model name to use.\"\"\"\n    model_reqs: List[str] = [\"./\", \"sentence_transformers\", \"torch\"]\n    \"\"\"Requirements to install on hardware to inference the model.\"\"\"\n    hardware: Any\n    \"\"\"Remote hardware to send the inference function to.\"\"\"\n    model_load_fn: Callable = load_embedding_model", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted_hugging_face.html"}359{"id": "5a13324b0a36-2", "text": "model_load_fn: Callable = load_embedding_model\n    \"\"\"Function to load the model remotely on the server.\"\"\"\n    load_fn_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model load function.\"\"\"\n    inference_fn: Callable = _embed_documents\n    \"\"\"Inference function to extract the embeddings.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the remote inference function.\"\"\"\n        load_fn_kwargs = kwargs.pop(\"load_fn_kwargs\", {})\n        load_fn_kwargs[\"model_id\"] = load_fn_kwargs.get(\"model_id\", DEFAULT_MODEL_NAME)\n        load_fn_kwargs[\"instruct\"] = load_fn_kwargs.get(\"instruct\", False)\n        load_fn_kwargs[\"device\"] = load_fn_kwargs.get(\"device\", 0)\n        super().__init__(load_fn_kwargs=load_fn_kwargs, **kwargs)\n[docs]class SelfHostedHuggingFaceInstructEmbeddings(SelfHostedHuggingFaceEmbeddings):\n    \"\"\"Runs InstructorEmbedding embedding models on self-hosted remote hardware.\n    Supported hardware includes auto-launched instances on AWS, GCP, Azure,\n    and Lambda, as well as servers specified\n    by IP address and SSH credentials (such as on-prem, or another\n    cloud like Paperspace, Coreweave, etc.).\n    To use, you should have the ``runhouse`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import SelfHostedHuggingFaceInstructEmbeddings\n            import runhouse as rh\n            model_name = \"hkunlp/instructor-large\"\n            gpu = rh.cluster(name='rh-a10x', instance_type='A100:1')\n            hf = SelfHostedHuggingFaceInstructEmbeddings(\n                model_name=model_name, hardware=gpu)\n    \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted_hugging_face.html"}360{"id": "5a13324b0a36-3", "text": "model_name=model_name, hardware=gpu)\n    \"\"\"\n    model_id: str = DEFAULT_INSTRUCT_MODEL\n    \"\"\"Model name to use.\"\"\"\n    embed_instruction: str = DEFAULT_EMBED_INSTRUCTION\n    \"\"\"Instruction to use for embedding documents.\"\"\"\n    query_instruction: str = DEFAULT_QUERY_INSTRUCTION\n    \"\"\"Instruction to use for embedding query.\"\"\"\n    model_reqs: List[str] = [\"./\", \"InstructorEmbedding\", \"torch\"]\n    \"\"\"Requirements to install on hardware to inference the model.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the remote inference function.\"\"\"\n        load_fn_kwargs = kwargs.pop(\"load_fn_kwargs\", {})\n        load_fn_kwargs[\"model_id\"] = load_fn_kwargs.get(\n            \"model_id\", DEFAULT_INSTRUCT_MODEL\n        )\n        load_fn_kwargs[\"instruct\"] = load_fn_kwargs.get(\"instruct\", True)\n        load_fn_kwargs[\"device\"] = load_fn_kwargs.get(\"device\", 0)\n        super().__init__(load_fn_kwargs=load_fn_kwargs, **kwargs)\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a HuggingFace instruct model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        instruction_pairs = []\n        for text in texts:\n            instruction_pairs.append([self.embed_instruction, text])\n        embeddings = self.client(self.pipeline_ref, instruction_pairs)\n        return embeddings.tolist()\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a HuggingFace instruct model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted_hugging_face.html"}361{"id": "5a13324b0a36-4", "text": "text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        instruction_pair = [self.query_instruction, text]\n        embedding = self.client(self.pipeline_ref, [instruction_pair])[0]\n        return embedding.tolist()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/self_hosted_hugging_face.html"}362{"id": "6955fd0c8f16-0", "text": "Source code for langchain.embeddings.cohere\n\"\"\"Wrapper around Cohere embedding models.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\n[docs]class CohereEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around Cohere embedding models.\n    To use, you should have the ``cohere`` python package installed, and the\n    environment variable ``COHERE_API_KEY`` set with your API key or pass it\n    as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import CohereEmbeddings\n            cohere = CohereEmbeddings(\n                model=\"embed-english-light-v2.0\", cohere_api_key=\"my-api-key\"\n            )\n    \"\"\"\n    client: Any  #: :meta private:\n    model: str = \"embed-english-v2.0\"\n    \"\"\"Model name to use.\"\"\"\n    truncate: Optional[str] = None\n    \"\"\"Truncate embeddings that are too long from start or end (\"NONE\"|\"START\"|\"END\")\"\"\"\n    cohere_api_key: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        cohere_api_key = get_from_dict_or_env(\n            values, \"cohere_api_key\", \"COHERE_API_KEY\"\n        )\n        try:\n            import cohere\n            values[\"client\"] = cohere.Client(cohere_api_key)\n        except ImportError:\n            raise ImportError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/cohere.html"}363{"id": "6955fd0c8f16-1", "text": "except ImportError:\n            raise ImportError(\n                \"Could not import cohere python package. \"\n                \"Please install it with `pip install cohere`.\"\n            )\n        return values\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Call out to Cohere's embedding endpoint.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        embeddings = self.client.embed(\n            model=self.model, texts=texts, truncate=self.truncate\n        ).embeddings\n        return [list(map(float, e)) for e in embeddings]\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Call out to Cohere's embedding endpoint.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        embedding = self.client.embed(\n            model=self.model, texts=[text], truncate=self.truncate\n        ).embeddings[0]\n        return list(map(float, embedding))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/cohere.html"}364{"id": "a385a8b570b1-0", "text": "Source code for langchain.embeddings.aleph_alpha\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, root_validator\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\n[docs]class AlephAlphaAsymmetricSemanticEmbedding(BaseModel, Embeddings):\n    \"\"\"\n    Wrapper for Aleph Alpha's Asymmetric Embeddings\n    AA provides you with an endpoint to embed a document and a query.\n    The models were optimized to make the embeddings of documents and\n    the query for a document as similar as possible.\n    To learn more, check out: https://docs.aleph-alpha.com/docs/tasks/semantic_embed/\n    Example:\n        .. code-block:: python\n            from aleph_alpha import AlephAlphaAsymmetricSemanticEmbedding\n            embeddings = AlephAlphaSymmetricSemanticEmbedding()\n            document = \"This is a content of the document\"\n            query = \"What is the content of the document?\"\n            doc_result = embeddings.embed_documents([document])\n            query_result = embeddings.embed_query(query)\n    \"\"\"\n    client: Any  #: :meta private:\n    model: Optional[str] = \"luminous-base\"\n    \"\"\"Model name to use.\"\"\"\n    hosting: Optional[str] = \"https://api.aleph-alpha.com\"\n    \"\"\"Optional parameter that specifies which datacenters may process the request.\"\"\"\n    normalize: Optional[bool] = True\n    \"\"\"Should returned embeddings be normalized\"\"\"\n    compress_to_size: Optional[int] = 128\n    \"\"\"Should the returned embeddings come back as an original 5120-dim vector, \n    or should it be compressed to 128-dim.\"\"\"\n    contextual_control_threshold: Optional[int] = None\n    \"\"\"Attention control parameters only apply to those tokens that have", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/aleph_alpha.html"}365{"id": "a385a8b570b1-1", "text": "\"\"\"Attention control parameters only apply to those tokens that have \n    explicitly been set in the request.\"\"\"\n    control_log_additive: Optional[bool] = True\n    \"\"\"Apply controls on prompt items by adding the log(control_factor) \n    to attention scores.\"\"\"\n    aleph_alpha_api_key: Optional[str] = None\n    \"\"\"API key for Aleph Alpha API.\"\"\"\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        aleph_alpha_api_key = get_from_dict_or_env(\n            values, \"aleph_alpha_api_key\", \"ALEPH_ALPHA_API_KEY\"\n        )\n        try:\n            from aleph_alpha_client import Client\n        except ImportError:\n            raise ValueError(\n                \"Could not import aleph_alpha_client python package. \"\n                \"Please install it with `pip install aleph_alpha_client`.\"\n            )\n        values[\"client\"] = Client(token=aleph_alpha_api_key)\n        return values\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Call out to Aleph Alpha's asymmetric Document endpoint.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        try:\n            from aleph_alpha_client import (\n                Prompt,\n                SemanticEmbeddingRequest,\n                SemanticRepresentation,\n            )\n        except ImportError:\n            raise ValueError(\n                \"Could not import aleph_alpha_client python package. \"\n                \"Please install it with `pip install aleph_alpha_client`.\"\n            )\n        document_embeddings = []\n        for text in texts:\n            document_params = {\n                \"prompt\": Prompt.from_text(text),", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/aleph_alpha.html"}366{"id": "a385a8b570b1-2", "text": "document_params = {\n                \"prompt\": Prompt.from_text(text),\n                \"representation\": SemanticRepresentation.Document,\n                \"compress_to_size\": self.compress_to_size,\n                \"normalize\": self.normalize,\n                \"contextual_control_threshold\": self.contextual_control_threshold,\n                \"control_log_additive\": self.control_log_additive,\n            }\n            document_request = SemanticEmbeddingRequest(**document_params)\n            document_response = self.client.semantic_embed(\n                request=document_request, model=self.model\n            )\n            document_embeddings.append(document_response.embedding)\n        return document_embeddings\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Call out to Aleph Alpha's asymmetric, query embedding endpoint\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        try:\n            from aleph_alpha_client import (\n                Prompt,\n                SemanticEmbeddingRequest,\n                SemanticRepresentation,\n            )\n        except ImportError:\n            raise ValueError(\n                \"Could not import aleph_alpha_client python package. \"\n                \"Please install it with `pip install aleph_alpha_client`.\"\n            )\n        symmetric_params = {\n            \"prompt\": Prompt.from_text(text),\n            \"representation\": SemanticRepresentation.Query,\n            \"compress_to_size\": self.compress_to_size,\n            \"normalize\": self.normalize,\n            \"contextual_control_threshold\": self.contextual_control_threshold,\n            \"control_log_additive\": self.control_log_additive,\n        }\n        symmetric_request = SemanticEmbeddingRequest(**symmetric_params)\n        symmetric_response = self.client.semantic_embed(\n            request=symmetric_request, model=self.model\n        )\n        return symmetric_response.embedding", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/aleph_alpha.html"}367{"id": "a385a8b570b1-3", "text": "request=symmetric_request, model=self.model\n        )\n        return symmetric_response.embedding\n[docs]class AlephAlphaSymmetricSemanticEmbedding(AlephAlphaAsymmetricSemanticEmbedding):\n    \"\"\"The symmetric version of the Aleph Alpha's semantic embeddings.\n    The main difference is that here, both the documents and\n    queries are embedded with a SemanticRepresentation.Symmetric\n    Example:\n        .. code-block:: python\n            from aleph_alpha import AlephAlphaSymmetricSemanticEmbedding\n            embeddings = AlephAlphaAsymmetricSemanticEmbedding()\n            text = \"This is a test text\"\n            doc_result = embeddings.embed_documents([text])\n            query_result = embeddings.embed_query(text)\n    \"\"\"\n    def _embed(self, text: str) -> List[float]:\n        try:\n            from aleph_alpha_client import (\n                Prompt,\n                SemanticEmbeddingRequest,\n                SemanticRepresentation,\n            )\n        except ImportError:\n            raise ValueError(\n                \"Could not import aleph_alpha_client python package. \"\n                \"Please install it with `pip install aleph_alpha_client`.\"\n            )\n        query_params = {\n            \"prompt\": Prompt.from_text(text),\n            \"representation\": SemanticRepresentation.Symmetric,\n            \"compress_to_size\": self.compress_to_size,\n            \"normalize\": self.normalize,\n            \"contextual_control_threshold\": self.contextual_control_threshold,\n            \"control_log_additive\": self.control_log_additive,\n        }\n        query_request = SemanticEmbeddingRequest(**query_params)\n        query_response = self.client.semantic_embed(\n            request=query_request, model=self.model\n        )\n        return query_response.embedding\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Call out to Aleph Alpha's Document endpoint.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/aleph_alpha.html"}368{"id": "a385a8b570b1-4", "text": "\"\"\"Call out to Aleph Alpha's Document endpoint.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        document_embeddings = []\n        for text in texts:\n            document_embeddings.append(self._embed(text))\n        return document_embeddings\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Call out to Aleph Alpha's asymmetric, query embedding endpoint\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        return self._embed(text)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/aleph_alpha.html"}369{"id": "86df1b58410e-0", "text": "Source code for langchain.embeddings.huggingface\n\"\"\"Wrapper around HuggingFace embedding models.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, Field\nfrom langchain.embeddings.base import Embeddings\nDEFAULT_MODEL_NAME = \"sentence-transformers/all-mpnet-base-v2\"\nDEFAULT_INSTRUCT_MODEL = \"hkunlp/instructor-large\"\nDEFAULT_EMBED_INSTRUCTION = \"Represent the document for retrieval: \"\nDEFAULT_QUERY_INSTRUCTION = (\n    \"Represent the question for retrieving supporting documents: \"\n)\n[docs]class HuggingFaceEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around sentence_transformers embedding models.\n    To use, you should have the ``sentence_transformers`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import HuggingFaceEmbeddings\n            model_name = \"sentence-transformers/all-mpnet-base-v2\"\n            model_kwargs = {'device': 'cpu'}\n            hf = HuggingFaceEmbeddings(model_name=model_name, model_kwargs=model_kwargs)\n    \"\"\"\n    client: Any  #: :meta private:\n    model_name: str = DEFAULT_MODEL_NAME\n    \"\"\"Model name to use.\"\"\"\n    cache_folder: Optional[str] = None\n    \"\"\"Path to store models. \n    Can be also set by SENTENCE_TRANSFORMERS_HOME environment variable.\"\"\"\n    model_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    encode_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Key word arguments to pass when calling the `encode` method of the model.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the sentence_transformer.\"\"\"\n        super().__init__(**kwargs)\n        try:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface.html"}370{"id": "86df1b58410e-1", "text": "super().__init__(**kwargs)\n        try:\n            import sentence_transformers\n        except ImportError as exc:\n            raise ImportError(\n                \"Could not import sentence_transformers python package. \"\n                \"Please install it with `pip install sentence_transformers`.\"\n            ) from exc\n        self.client = sentence_transformers.SentenceTransformer(\n            self.model_name, cache_folder=self.cache_folder, **self.model_kwargs\n        )\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a HuggingFace transformer model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))\n        embeddings = self.client.encode(texts, **self.encode_kwargs)\n        return embeddings.tolist()\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a HuggingFace transformer model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        text = text.replace(\"\\n\", \" \")\n        embedding = self.client.encode(text, **self.encode_kwargs)\n        return embedding.tolist()\n[docs]class HuggingFaceInstructEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around sentence_transformers embedding models.\n    To use, you should have the ``sentence_transformers``\n    and ``InstructorEmbedding`` python packages installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import HuggingFaceInstructEmbeddings", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface.html"}371{"id": "86df1b58410e-2", "text": "from langchain.embeddings import HuggingFaceInstructEmbeddings\n            model_name = \"hkunlp/instructor-large\"\n            model_kwargs = {'device': 'cpu'}\n            hf = HuggingFaceInstructEmbeddings(\n                model_name=model_name, model_kwargs=model_kwargs\n            )\n    \"\"\"\n    client: Any  #: :meta private:\n    model_name: str = DEFAULT_INSTRUCT_MODEL\n    \"\"\"Model name to use.\"\"\"\n    cache_folder: Optional[str] = None\n    \"\"\"Path to store models. \n    Can be also set by SENTENCE_TRANSFORMERS_HOME environment variable.\"\"\"\n    model_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    embed_instruction: str = DEFAULT_EMBED_INSTRUCTION\n    \"\"\"Instruction to use for embedding documents.\"\"\"\n    query_instruction: str = DEFAULT_QUERY_INSTRUCTION\n    \"\"\"Instruction to use for embedding query.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the sentence_transformer.\"\"\"\n        super().__init__(**kwargs)\n        try:\n            from InstructorEmbedding import INSTRUCTOR\n            self.client = INSTRUCTOR(\n                self.model_name, cache_folder=self.cache_folder, **self.model_kwargs\n            )\n        except ImportError as e:\n            raise ValueError(\"Dependencies for InstructorEmbedding not found.\") from e\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a HuggingFace instruct model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface.html"}372{"id": "86df1b58410e-3", "text": "Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        instruction_pairs = [[self.embed_instruction, text] for text in texts]\n        embeddings = self.client.encode(instruction_pairs)\n        return embeddings.tolist()\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a HuggingFace instruct model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        instruction_pair = [self.query_instruction, text]\n        embedding = self.client.encode([instruction_pair])[0]\n        return embedding.tolist()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/huggingface.html"}373{"id": "7deb32ea1505-0", "text": "Source code for langchain.embeddings.fake\nfrom typing import List\nimport numpy as np\nfrom pydantic import BaseModel\nfrom langchain.embeddings.base import Embeddings\n[docs]class FakeEmbeddings(Embeddings, BaseModel):\n    size: int\n    def _get_embedding(self) -> List[float]:\n        return list(np.random.normal(size=self.size))\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        return [self._get_embedding() for _ in texts]\n[docs]    def embed_query(self, text: str) -> List[float]:\n        return self._get_embedding()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/fake.html"}374{"id": "f1479d184b63-0", "text": "Source code for langchain.embeddings.llamacpp\n\"\"\"Wrapper around llama.cpp embedding models.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, Field, root_validator\nfrom langchain.embeddings.base import Embeddings\n[docs]class LlamaCppEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around llama.cpp embedding models.\n    To use, you should have the llama-cpp-python library installed, and provide the\n    path to the Llama model as a named parameter to the constructor.\n    Check out: https://github.com/abetlen/llama-cpp-python\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import LlamaCppEmbeddings\n            llama = LlamaCppEmbeddings(model_path=\"/path/to/model.bin\")\n    \"\"\"\n    client: Any  #: :meta private:\n    model_path: str\n    n_ctx: int = Field(512, alias=\"n_ctx\")\n    \"\"\"Token context window.\"\"\"\n    n_parts: int = Field(-1, alias=\"n_parts\")\n    \"\"\"Number of parts to split the model into. \n    If -1, the number of parts is automatically determined.\"\"\"\n    seed: int = Field(-1, alias=\"seed\")\n    \"\"\"Seed. If -1, a random seed is used.\"\"\"\n    f16_kv: bool = Field(False, alias=\"f16_kv\")\n    \"\"\"Use half-precision for key/value cache.\"\"\"\n    logits_all: bool = Field(False, alias=\"logits_all\")\n    \"\"\"Return logits for all tokens, not just the last token.\"\"\"\n    vocab_only: bool = Field(False, alias=\"vocab_only\")\n    \"\"\"Only load the vocabulary, no weights.\"\"\"\n    use_mlock: bool = Field(False, alias=\"use_mlock\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/llamacpp.html"}375{"id": "f1479d184b63-1", "text": "use_mlock: bool = Field(False, alias=\"use_mlock\")\n    \"\"\"Force system to keep model in RAM.\"\"\"\n    n_threads: Optional[int] = Field(None, alias=\"n_threads\")\n    \"\"\"Number of threads to use. If None, the number \n    of threads is automatically determined.\"\"\"\n    n_batch: Optional[int] = Field(8, alias=\"n_batch\")\n    \"\"\"Number of tokens to process in parallel.\n    Should be a number between 1 and n_ctx.\"\"\"\n    n_gpu_layers: Optional[int] = Field(None, alias=\"n_gpu_layers\")\n    \"\"\"Number of layers to be loaded into gpu memory. Default None.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that llama-cpp-python library is installed.\"\"\"\n        model_path = values[\"model_path\"]\n        model_param_names = [\n            \"n_ctx\",\n            \"n_parts\",\n            \"seed\",\n            \"f16_kv\",\n            \"logits_all\",\n            \"vocab_only\",\n            \"use_mlock\",\n            \"n_threads\",\n            \"n_batch\",\n        ]\n        model_params = {k: values[k] for k in model_param_names}\n        # For backwards compatibility, only include if non-null.\n        if values[\"n_gpu_layers\"] is not None:\n            model_params[\"n_gpu_layers\"] = values[\"n_gpu_layers\"]\n        try:\n            from llama_cpp import Llama\n            values[\"client\"] = Llama(model_path, embedding=True, **model_params)\n        except ImportError:\n            raise ModuleNotFoundError(\n                \"Could not import llama-cpp-python library. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/llamacpp.html"}376{"id": "f1479d184b63-2", "text": "raise ModuleNotFoundError(\n                \"Could not import llama-cpp-python library. \"\n                \"Please install the llama-cpp-python library to \"\n                \"use this embedding model: pip install llama-cpp-python\"\n            )\n        except Exception as e:\n            raise ValueError(\n                f\"Could not load Llama model from path: {model_path}. \"\n                f\"Received error {e}\"\n            )\n        return values\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Embed a list of documents using the Llama model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        embeddings = [self.client.embed(text) for text in texts]\n        return [list(map(float, e)) for e in embeddings]\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Embed a query using the Llama model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        embedding = self.client.embed(text)\n        return list(map(float, embedding))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/llamacpp.html"}377{"id": "e0c365fc8a03-0", "text": "Source code for langchain.embeddings.minimax\n\"\"\"Wrapper around MiniMax APIs.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, Callable, Dict, List, Optional\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom tenacity import (\n    before_sleep_log,\n    retry,\n    stop_after_attempt,\n    wait_exponential,\n)\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\ndef _create_retry_decorator() -> Callable[[Any], Any]:\n    \"\"\"Returns a tenacity retry decorator.\"\"\"\n    multiplier = 1\n    min_seconds = 1\n    max_seconds = 4\n    max_retries = 6\n    return retry(\n        reraise=True,\n        stop=stop_after_attempt(max_retries),\n        wait=wait_exponential(multiplier=multiplier, min=min_seconds, max=max_seconds),\n        before_sleep=before_sleep_log(logger, logging.WARNING),\n    )\ndef embed_with_retry(embeddings: MiniMaxEmbeddings, *args: Any, **kwargs: Any) -> Any:\n    \"\"\"Use tenacity to retry the completion call.\"\"\"\n    retry_decorator = _create_retry_decorator()\n    @retry_decorator\n    def _embed_with_retry(*args: Any, **kwargs: Any) -> Any:\n        return embeddings.embed(*args, **kwargs)\n    return _embed_with_retry(*args, **kwargs)\n[docs]class MiniMaxEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around MiniMax's embedding inference service.\n    To use, you should have the environment variable ``MINIMAX_GROUP_ID`` and\n    ``MINIMAX_API_KEY`` set with your API token, or pass it as a named parameter to\n    the constructor.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/minimax.html"}378{"id": "e0c365fc8a03-1", "text": "the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import MiniMaxEmbeddings\n            embeddings = MiniMaxEmbeddings()\n            query_text = \"This is a test query.\"\n            query_result = embeddings.embed_query(query_text)\n            document_text = \"This is a test document.\"\n            document_result = embeddings.embed_documents([document_text])\n    \"\"\"\n    endpoint_url: str = \"https://api.minimax.chat/v1/embeddings\"\n    \"\"\"Endpoint URL to use.\"\"\"\n    model: str = \"embo-01\"\n    \"\"\"Embeddings model name to use.\"\"\"\n    embed_type_db: str = \"db\"\n    \"\"\"For embed_documents\"\"\"\n    embed_type_query: str = \"query\"\n    \"\"\"For embed_query\"\"\"\n    minimax_group_id: Optional[str] = None\n    \"\"\"Group ID for MiniMax API.\"\"\"\n    minimax_api_key: Optional[str] = None\n    \"\"\"API Key for MiniMax API.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that group id and api key exists in environment.\"\"\"\n        minimax_group_id = get_from_dict_or_env(\n            values, \"minimax_group_id\", \"MINIMAX_GROUP_ID\"\n        )\n        minimax_api_key = get_from_dict_or_env(\n            values, \"minimax_api_key\", \"MINIMAX_API_KEY\"\n        )\n        values[\"minimax_group_id\"] = minimax_group_id\n        values[\"minimax_api_key\"] = minimax_api_key\n        return values\n    def embed(\n        self,\n        texts: List[str],\n        embed_type: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/minimax.html"}379{"id": "e0c365fc8a03-2", "text": "self,\n        texts: List[str],\n        embed_type: str,\n    ) -> List[List[float]]:\n        payload = {\n            \"model\": self.model,\n            \"type\": embed_type,\n            \"texts\": texts,\n        }\n        # HTTP headers for authorization\n        headers = {\n            \"Authorization\": f\"Bearer {self.minimax_api_key}\",\n            \"Content-Type\": \"application/json\",\n        }\n        params = {\n            \"GroupId\": self.minimax_group_id,\n        }\n        # send request\n        response = requests.post(\n            self.endpoint_url, params=params, headers=headers, json=payload\n        )\n        parsed_response = response.json()\n        # check for errors\n        if parsed_response[\"base_resp\"][\"status_code\"] != 0:\n            raise ValueError(\n                f\"MiniMax API returned an error: {parsed_response['base_resp']}\"\n            )\n        embeddings = parsed_response[\"vectors\"]\n        return embeddings\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Embed documents using a MiniMax embedding endpoint.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        embeddings = embed_with_retry(self, texts=texts, embed_type=self.embed_type_db)\n        return embeddings\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Embed a query using a MiniMax embedding endpoint.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        embeddings = embed_with_retry(\n            self, texts=[text], embed_type=self.embed_type_query\n        )\n        return embeddings[0]", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/minimax.html"}380{"id": "e0c365fc8a03-3", "text": ")\n        return embeddings[0]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/minimax.html"}381{"id": "8b74e2dddf1a-0", "text": "Source code for langchain.embeddings.openai\n\"\"\"Wrapper around OpenAI embedding models.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom typing import (\n    Any,\n    Callable,\n    Dict,\n    List,\n    Literal,\n    Optional,\n    Sequence,\n    Set,\n    Tuple,\n    Union,\n)\nimport numpy as np\nfrom pydantic import BaseModel, Extra, root_validator\nfrom tenacity import (\n    before_sleep_log,\n    retry,\n    retry_if_exception_type,\n    stop_after_attempt,\n    wait_exponential,\n)\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\ndef _create_retry_decorator(embeddings: OpenAIEmbeddings) -> Callable[[Any], Any]:\n    import openai\n    min_seconds = 4\n    max_seconds = 10\n    # Wait 2^x * 1 second between each retry starting with\n    # 4 seconds, then up to 10 seconds, then 10 seconds afterwards\n    return retry(\n        reraise=True,\n        stop=stop_after_attempt(embeddings.max_retries),\n        wait=wait_exponential(multiplier=1, min=min_seconds, max=max_seconds),\n        retry=(\n            retry_if_exception_type(openai.error.Timeout)\n            | retry_if_exception_type(openai.error.APIError)\n            | retry_if_exception_type(openai.error.APIConnectionError)\n            | retry_if_exception_type(openai.error.RateLimitError)\n            | retry_if_exception_type(openai.error.ServiceUnavailableError)\n        ),\n        before_sleep=before_sleep_log(logger, logging.WARNING),\n    )\ndef embed_with_retry(embeddings: OpenAIEmbeddings, **kwargs: Any) -> Any:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}382{"id": "8b74e2dddf1a-1", "text": "def embed_with_retry(embeddings: OpenAIEmbeddings, **kwargs: Any) -> Any:\n    \"\"\"Use tenacity to retry the embedding call.\"\"\"\n    retry_decorator = _create_retry_decorator(embeddings)\n    @retry_decorator\n    def _embed_with_retry(**kwargs: Any) -> Any:\n        return embeddings.client.create(**kwargs)\n    return _embed_with_retry(**kwargs)\n[docs]class OpenAIEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around OpenAI embedding models.\n    To use, you should have the ``openai`` python package installed, and the\n    environment variable ``OPENAI_API_KEY`` set with your API key or pass it\n    as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import OpenAIEmbeddings\n            openai = OpenAIEmbeddings(openai_api_key=\"my-api-key\")\n    In order to use the library with Microsoft Azure endpoints, you need to set\n    the OPENAI_API_TYPE, OPENAI_API_BASE, OPENAI_API_KEY and OPENAI_API_VERSION.\n    The OPENAI_API_TYPE must be set to 'azure' and the others correspond to\n    the properties of your endpoint.\n    In addition, the deployment name must be passed as the model parameter.\n    Example:\n        .. code-block:: python\n            import os\n            os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n            os.environ[\"OPENAI_API_BASE\"] = \"https://<your-endpoint.openai.azure.com/\"\n            os.environ[\"OPENAI_API_KEY\"] = \"your AzureOpenAI key\"\n            os.environ[\"OPENAI_API_VERSION\"] = \"2023-03-15-preview\"\n            os.environ[\"OPENAI_PROXY\"] = \"http://your-corporate-proxy:8080\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}383{"id": "8b74e2dddf1a-2", "text": "from langchain.embeddings.openai import OpenAIEmbeddings\n            embeddings = OpenAIEmbeddings(\n                deployment=\"your-embeddings-deployment-name\",\n                model=\"your-embeddings-model-name\",\n                api_base=\"https://your-endpoint.openai.azure.com/\",\n                api_type=\"azure\",\n            )\n            text = \"This is a test query.\"\n            query_result = embeddings.embed_query(text)\n    \"\"\"\n    client: Any  #: :meta private:\n    model: str = \"text-embedding-ada-002\"\n    deployment: str = model  # to support Azure OpenAI Service custom deployment names\n    openai_api_version: Optional[str] = None\n    # to support Azure OpenAI Service custom endpoints\n    openai_api_base: Optional[str] = None\n    # to support Azure OpenAI Service custom endpoints\n    openai_api_type: Optional[str] = None\n    # to support explicit proxy for OpenAI\n    openai_proxy: Optional[str] = None\n    embedding_ctx_length: int = 8191\n    openai_api_key: Optional[str] = None\n    openai_organization: Optional[str] = None\n    allowed_special: Union[Literal[\"all\"], Set[str]] = set()\n    disallowed_special: Union[Literal[\"all\"], Set[str], Sequence[str]] = \"all\"\n    chunk_size: int = 1000\n    \"\"\"Maximum number of texts to embed in each batch\"\"\"\n    max_retries: int = 6\n    \"\"\"Maximum number of retries to make when generating.\"\"\"\n    request_timeout: Optional[Union[float, Tuple[float, float]]] = None\n    \"\"\"Timeout in seconds for the OpenAPI request.\"\"\"\n    headers: Any = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}384{"id": "8b74e2dddf1a-3", "text": "\"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        openai_api_key = get_from_dict_or_env(\n            values, \"openai_api_key\", \"OPENAI_API_KEY\"\n        )\n        openai_api_base = get_from_dict_or_env(\n            values,\n            \"openai_api_base\",\n            \"OPENAI_API_BASE\",\n            default=\"\",\n        )\n        openai_api_type = get_from_dict_or_env(\n            values,\n            \"openai_api_type\",\n            \"OPENAI_API_TYPE\",\n            default=\"\",\n        )\n        openai_proxy = get_from_dict_or_env(\n            values,\n            \"openai_proxy\",\n            \"OPENAI_PROXY\",\n            default=\"\",\n        )\n        if openai_api_type in (\"azure\", \"azure_ad\", \"azuread\"):\n            default_api_version = \"2022-12-01\"\n        else:\n            default_api_version = \"\"\n        openai_api_version = get_from_dict_or_env(\n            values,\n            \"openai_api_version\",\n            \"OPENAI_API_VERSION\",\n            default=default_api_version,\n        )\n        openai_organization = get_from_dict_or_env(\n            values,\n            \"openai_organization\",\n            \"OPENAI_ORGANIZATION\",\n            default=\"\",\n        )\n        try:\n            import openai\n            openai.api_key = openai_api_key\n            if openai_organization:\n                openai.organization = openai_organization\n            if openai_api_base:\n                openai.api_base = openai_api_base\n            if openai_api_type:\n                openai.api_version = openai_api_version", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}385{"id": "8b74e2dddf1a-4", "text": "if openai_api_type:\n                openai.api_version = openai_api_version\n            if openai_api_type:\n                openai.api_type = openai_api_type\n            if openai_proxy:\n                openai.proxy = {\"http\": openai_proxy, \"https\": openai_proxy}  # type: ignore[assignment]  # noqa: E501\n            values[\"client\"] = openai.Embedding\n        except ImportError:\n            raise ImportError(\n                \"Could not import openai python package. \"\n                \"Please install it with `pip install openai`.\"\n            )\n        return values\n    # please refer to\n    # https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\n    def _get_len_safe_embeddings(\n        self, texts: List[str], *, engine: str, chunk_size: Optional[int] = None\n    ) -> List[List[float]]:\n        embeddings: List[List[float]] = [[] for _ in range(len(texts))]\n        try:\n            import tiktoken\n        except ImportError:\n            raise ImportError(\n                \"Could not import tiktoken python package. \"\n                \"This is needed in order to for OpenAIEmbeddings. \"\n                \"Please install it with `pip install tiktoken`.\"\n            )\n        tokens = []\n        indices = []\n        encoding = tiktoken.model.encoding_for_model(self.model)\n        for i, text in enumerate(texts):\n            if self.model.endswith(\"001\"):\n                # See: https://github.com/openai/openai-python/issues/418#issuecomment-1525939500\n                # replace newlines, which can negatively affect performance.\n                text = text.replace(\"\\n\", \" \")\n            token = encoding.encode(\n                text,\n                allowed_special=self.allowed_special,", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}386{"id": "8b74e2dddf1a-5", "text": "token = encoding.encode(\n                text,\n                allowed_special=self.allowed_special,\n                disallowed_special=self.disallowed_special,\n            )\n            for j in range(0, len(token), self.embedding_ctx_length):\n                tokens += [token[j : j + self.embedding_ctx_length]]\n                indices += [i]\n        batched_embeddings = []\n        _chunk_size = chunk_size or self.chunk_size\n        for i in range(0, len(tokens), _chunk_size):\n            response = embed_with_retry(\n                self,\n                input=tokens[i : i + _chunk_size],\n                engine=self.deployment,\n                request_timeout=self.request_timeout,\n                headers=self.headers,\n            )\n            batched_embeddings += [r[\"embedding\"] for r in response[\"data\"]]\n        results: List[List[List[float]]] = [[] for _ in range(len(texts))]\n        num_tokens_in_batch: List[List[int]] = [[] for _ in range(len(texts))]\n        for i in range(len(indices)):\n            results[indices[i]].append(batched_embeddings[i])\n            num_tokens_in_batch[indices[i]].append(len(tokens[i]))\n        for i in range(len(texts)):\n            _result = results[i]\n            if len(_result) == 0:\n                average = embed_with_retry(\n                    self,\n                    input=\"\",\n                    engine=self.deployment,\n                    request_timeout=self.request_timeout,\n                    headers=self.headers,\n                )[\"data\"][0][\"embedding\"]\n            else:\n                average = np.average(_result, axis=0, weights=num_tokens_in_batch[i])\n            embeddings[i] = (average / np.linalg.norm(average)).tolist()\n        return embeddings\n    def _embedding_func(self, text: str, *, engine: str) -> List[float]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}387{"id": "8b74e2dddf1a-6", "text": "def _embedding_func(self, text: str, *, engine: str) -> List[float]:\n        \"\"\"Call out to OpenAI's embedding endpoint.\"\"\"\n        # handle large input text\n        if len(text) > self.embedding_ctx_length:\n            return self._get_len_safe_embeddings([text], engine=engine)[0]\n        else:\n            if self.model.endswith(\"001\"):\n                # See: https://github.com/openai/openai-python/issues/418#issuecomment-1525939500\n                # replace newlines, which can negatively affect performance.\n                text = text.replace(\"\\n\", \" \")\n            return embed_with_retry(\n                self,\n                input=[text],\n                engine=engine,\n                request_timeout=self.request_timeout,\n                headers=self.headers,\n            )[\"data\"][0][\"embedding\"]\n[docs]    def embed_documents(\n        self, texts: List[str], chunk_size: Optional[int] = 0\n    ) -> List[List[float]]:\n        \"\"\"Call out to OpenAI's embedding endpoint for embedding search docs.\n        Args:\n            texts: The list of texts to embed.\n            chunk_size: The chunk size of embeddings. If None, will use the chunk size\n                specified by the class.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        # NOTE: to keep things simple, we assume the list may contain texts longer\n        #       than the maximum context and use length-safe embedding function.\n        return self._get_len_safe_embeddings(texts, engine=self.deployment)\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Call out to OpenAI's embedding endpoint for embedding query text.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embedding for the text.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}388{"id": "8b74e2dddf1a-7", "text": "text: The text to embed.\n        Returns:\n            Embedding for the text.\n        \"\"\"\n        embedding = self._embedding_func(text, engine=self.deployment)\n        return embedding\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/openai.html"}389{"id": "9ca92af7b7a5-0", "text": "Source code for langchain.embeddings.elasticsearch\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, List, Optional\nfrom langchain.utils import get_from_env\nif TYPE_CHECKING:\n    from elasticsearch.client import MlClient\nfrom langchain.embeddings.base import Embeddings\n[docs]class ElasticsearchEmbeddings(Embeddings):\n    \"\"\"\n    Wrapper around Elasticsearch embedding models.\n    This class provides an interface to generate embeddings using a model deployed\n    in an Elasticsearch cluster. It requires an Elasticsearch connection object\n    and the model_id of the model deployed in the cluster.\n    In Elasticsearch you need to have an embedding model loaded and deployed.\n    - https://www.elastic.co/guide/en/elasticsearch/reference/current/infer-trained-model.html\n    - https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-deploy-models.html\n    \"\"\"  # noqa: E501\n    def __init__(\n        self,\n        client: MlClient,\n        model_id: str,\n        *,\n        input_field: str = \"text_field\",\n    ):\n        \"\"\"\n        Initialize the ElasticsearchEmbeddings instance.\n        Args:\n            client (MlClient): An Elasticsearch ML client object.\n            model_id (str): The model_id of the model deployed in the Elasticsearch\n                cluster.\n            input_field (str): The name of the key for the input text field in the\n                document. Defaults to 'text_field'.\n        \"\"\"\n        self.client = client\n        self.model_id = model_id\n        self.input_field = input_field\n[docs]    @classmethod\n    def from_credentials(\n        cls,\n        model_id: str,\n        *,\n        es_cloud_id: Optional[str] = None,\n        es_user: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/elasticsearch.html"}390{"id": "9ca92af7b7a5-1", "text": "es_user: Optional[str] = None,\n        es_password: Optional[str] = None,\n        input_field: str = \"text_field\",\n    ) -> ElasticsearchEmbeddings:\n        \"\"\"Instantiate embeddings from Elasticsearch credentials.\n        Args:\n            model_id (str): The model_id of the model deployed in the Elasticsearch\n                cluster.\n            input_field (str): The name of the key for the input text field in the\n                document. Defaults to 'text_field'.\n            es_cloud_id: (str, optional): The Elasticsearch cloud ID to connect to.\n            es_user: (str, optional): Elasticsearch username.\n            es_password: (str, optional): Elasticsearch password.\n        Example Usage:\n            from langchain.embeddings import ElasticsearchEmbeddings\n            # Define the model ID and input field name (if different from default)\n            model_id = \"your_model_id\"\n            # Optional, only if different from 'text_field'\n            input_field = \"your_input_field\"\n            # Credentials can be passed in two ways. Either set the env vars\n            # ES_CLOUD_ID, ES_USER, ES_PASSWORD and they will be automatically pulled\n            # in, or pass them in directly as kwargs.\n            embeddings = ElasticsearchEmbeddings.from_credentials(\n                model_id,\n                input_field=input_field,\n                # es_cloud_id=\"foo\",\n                # es_user=\"bar\",\n                # es_password=\"baz\",\n            )\n            documents = [\n                \"This is an example document.\",\n                \"Another example document to generate embeddings for.\",\n            ]\n            embeddings_generator.embed_documents(documents)\n        \"\"\"\n        try:\n            from elasticsearch import Elasticsearch\n            from elasticsearch.client import MlClient\n        except ImportError:\n            raise ImportError(\n                \"elasticsearch package not found, please install with 'pip install \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/elasticsearch.html"}391{"id": "9ca92af7b7a5-2", "text": "raise ImportError(\n                \"elasticsearch package not found, please install with 'pip install \"\n                \"elasticsearch'\"\n            )\n        es_cloud_id = es_cloud_id or get_from_env(\"es_cloud_id\", \"ES_CLOUD_ID\")\n        es_user = es_user or get_from_env(\"es_user\", \"ES_USER\")\n        es_password = es_password or get_from_env(\"es_password\", \"ES_PASSWORD\")\n        # Connect to Elasticsearch\n        es_connection = Elasticsearch(\n            cloud_id=es_cloud_id, basic_auth=(es_user, es_password)\n        )\n        client = MlClient(es_connection)\n        return cls(client, model_id, input_field=input_field)\n    def _embedding_func(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"\n        Generate embeddings for the given texts using the Elasticsearch model.\n        Args:\n            texts (List[str]): A list of text strings to generate embeddings for.\n        Returns:\n            List[List[float]]: A list of embeddings, one for each text in the input\n                list.\n        \"\"\"\n        response = self.client.infer_trained_model(\n            model_id=self.model_id, docs=[{self.input_field: text} for text in texts]\n        )\n        embeddings = [doc[\"predicted_value\"] for doc in response[\"inference_results\"]]\n        return embeddings\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"\n        Generate embeddings for a list of documents.\n        Args:\n            texts (List[str]): A list of document text strings to generate embeddings\n                for.\n        Returns:\n            List[List[float]]: A list of embeddings, one for each document in the input\n                list.\n        \"\"\"\n        return self._embedding_func(texts)", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/elasticsearch.html"}392{"id": "9ca92af7b7a5-3", "text": "list.\n        \"\"\"\n        return self._embedding_func(texts)\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"\n        Generate an embedding for a single query text.\n        Args:\n            text (str): The query text to generate an embedding for.\n        Returns:\n            List[float]: The embedding for the input query text.\n        \"\"\"\n        return self._embedding_func([text])[0]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/elasticsearch.html"}393{"id": "378597d4bc83-0", "text": "Source code for langchain.embeddings.modelscope_hub\n\"\"\"Wrapper around ModelScopeHub embedding models.\"\"\"\nfrom typing import Any, List\nfrom pydantic import BaseModel, Extra\nfrom langchain.embeddings.base import Embeddings\n[docs]class ModelScopeEmbeddings(BaseModel, Embeddings):\n    \"\"\"Wrapper around modelscope_hub embedding models.\n    To use, you should have the ``modelscope`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embeddings import ModelScopeEmbeddings\n            model_id = \"damo/nlp_corom_sentence-embedding_english-base\"\n            embed = ModelScopeEmbeddings(model_id=model_id)\n    \"\"\"\n    embed: Any\n    model_id: str = \"damo/nlp_corom_sentence-embedding_english-base\"\n    \"\"\"Model name to use.\"\"\"\n    def __init__(self, **kwargs: Any):\n        \"\"\"Initialize the modelscope\"\"\"\n        super().__init__(**kwargs)\n        try:\n            from modelscope.pipelines import pipeline\n            from modelscope.utils.constant import Tasks\n            self.embed = pipeline(Tasks.sentence_embedding, model=self.model_id)\n        except ImportError as e:\n            raise ImportError(\n                \"Could not import some python packages.\"\n                \"Please install it with `pip install modelscope`.\"\n            ) from e\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        \"\"\"Compute doc embeddings using a modelscope embedding model.\n        Args:\n            texts: The list of texts to embed.\n        Returns:\n            List of embeddings, one for each text.\n        \"\"\"\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/modelscope_hub.html"}394{"id": "378597d4bc83-1", "text": "\"\"\"\n        texts = list(map(lambda x: x.replace(\"\\n\", \" \"), texts))\n        inputs = {\"source_sentence\": texts}\n        embeddings = self.embed(input=inputs)[\"text_embedding\"]\n        return embeddings.tolist()\n[docs]    def embed_query(self, text: str) -> List[float]:\n        \"\"\"Compute query embeddings using a modelscope embedding model.\n        Args:\n            text: The text to embed.\n        Returns:\n            Embeddings for the text.\n        \"\"\"\n        text = text.replace(\"\\n\", \" \")\n        inputs = {\"source_sentence\": [text]}\n        embedding = self.embed(input=inputs)[\"text_embedding\"][0]\n        return embedding.tolist()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/embeddings/modelscope_hub.html"}395{"id": "7866ee0443e6-0", "text": "Source code for langchain.memory.vectorstore\n\"\"\"Class for a VectorStore-backed memory object.\"\"\"\nfrom typing import Any, Dict, List, Optional, Union\nfrom pydantic import Field\nfrom langchain.memory.chat_memory import BaseMemory\nfrom langchain.memory.utils import get_prompt_input_key\nfrom langchain.schema import Document\nfrom langchain.vectorstores.base import VectorStoreRetriever\n[docs]class VectorStoreRetrieverMemory(BaseMemory):\n    \"\"\"Class for a VectorStore-backed memory object.\"\"\"\n    retriever: VectorStoreRetriever = Field(exclude=True)\n    \"\"\"VectorStoreRetriever object to connect to.\"\"\"\n    memory_key: str = \"history\"  #: :meta private:\n    \"\"\"Key name to locate the memories in the result of load_memory_variables.\"\"\"\n    input_key: Optional[str] = None\n    \"\"\"Key name to index the inputs to load_memory_variables.\"\"\"\n    return_docs: bool = False\n    \"\"\"Whether or not to return the result of querying the database directly.\"\"\"\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"The list of keys emitted from the load_memory_variables method.\"\"\"\n        return [self.memory_key]\n    def _get_prompt_input_key(self, inputs: Dict[str, Any]) -> str:\n        \"\"\"Get the input key for the prompt.\"\"\"\n        if self.input_key is None:\n            return get_prompt_input_key(inputs, self.memory_variables)\n        return self.input_key\n[docs]    def load_memory_variables(\n        self, inputs: Dict[str, Any]\n    ) -> Dict[str, Union[List[Document], str]]:\n        \"\"\"Return history buffer.\"\"\"\n        input_key = self._get_prompt_input_key(inputs)\n        query = inputs[input_key]\n        docs = self.retriever.get_relevant_documents(query)", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/vectorstore.html"}396{"id": "7866ee0443e6-1", "text": "docs = self.retriever.get_relevant_documents(query)\n        result: Union[List[Document], str]\n        if not self.return_docs:\n            result = \"\\n\".join([doc.page_content for doc in docs])\n        else:\n            result = docs\n        return {self.memory_key: result}\n    def _form_documents(\n        self, inputs: Dict[str, Any], outputs: Dict[str, str]\n    ) -> List[Document]:\n        \"\"\"Format context from this conversation to buffer.\"\"\"\n        # Each document should only include the current turn, not the chat history\n        filtered_inputs = {k: v for k, v in inputs.items() if k != self.memory_key}\n        texts = [\n            f\"{k}: {v}\"\n            for k, v in list(filtered_inputs.items()) + list(outputs.items())\n        ]\n        page_content = \"\\n\".join(texts)\n        return [Document(page_content=page_content)]\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        documents = self._form_documents(inputs, outputs)\n        self.retriever.add_documents(documents)\n[docs]    def clear(self) -> None:\n        \"\"\"Nothing to clear.\"\"\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/vectorstore.html"}397{"id": "624fcf64fdb7-0", "text": "Source code for langchain.memory.buffer_window\nfrom typing import Any, Dict, List\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.schema import BaseMessage, get_buffer_string\n[docs]class ConversationBufferWindowMemory(BaseChatMemory):\n    \"\"\"Buffer for storing conversation memory.\"\"\"\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    memory_key: str = \"history\"  #: :meta private:\n    k: int = 5\n    @property\n    def buffer(self) -> List[BaseMessage]:\n        \"\"\"String buffer of memory.\"\"\"\n        return self.chat_memory.messages\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        \"\"\"Return history buffer.\"\"\"\n        buffer: Any = self.buffer[-self.k * 2 :] if self.k > 0 else []\n        if not self.return_messages:\n            buffer = get_buffer_string(\n                buffer,\n                human_prefix=self.human_prefix,\n                ai_prefix=self.ai_prefix,\n            )\n        return {self.memory_key: buffer}\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/buffer_window.html"}398{"id": "d08213db8fff-0", "text": "Source code for langchain.memory.token_buffer\nfrom typing import Any, Dict, List\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.schema import BaseMessage, get_buffer_string\n[docs]class ConversationTokenBufferMemory(BaseChatMemory):\n    \"\"\"Buffer for storing conversation memory.\"\"\"\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    llm: BaseLanguageModel\n    memory_key: str = \"history\"\n    max_token_limit: int = 2000\n    @property\n    def buffer(self) -> List[BaseMessage]:\n        \"\"\"String buffer of memory.\"\"\"\n        return self.chat_memory.messages\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        buffer: Any = self.buffer\n        if self.return_messages:\n            final_buffer: Any = buffer\n        else:\n            final_buffer = get_buffer_string(\n                buffer,\n                human_prefix=self.human_prefix,\n                ai_prefix=self.ai_prefix,\n            )\n        return {self.memory_key: final_buffer}\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer. Pruned.\"\"\"\n        super().save_context(inputs, outputs)\n        # Prune buffer if it exceeds max token limit\n        buffer = self.chat_memory.messages\n        curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)\n        if curr_buffer_length > self.max_token_limit:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/token_buffer.html"}399{"id": "d08213db8fff-1", "text": "if curr_buffer_length > self.max_token_limit:\n            pruned_memory = []\n            while curr_buffer_length > self.max_token_limit:\n                pruned_memory.append(buffer.pop(0))\n                curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/token_buffer.html"}400{"id": "2f45db8bfcd1-0", "text": "Source code for langchain.memory.summary_buffer\nfrom typing import Any, Dict, List\nfrom pydantic import root_validator\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.memory.summary import SummarizerMixin\nfrom langchain.schema import BaseMessage, get_buffer_string\n[docs]class ConversationSummaryBufferMemory(BaseChatMemory, SummarizerMixin):\n    \"\"\"Buffer with summarizer for storing conversation memory.\"\"\"\n    max_token_limit: int = 2000\n    moving_summary_buffer: str = \"\"\n    memory_key: str = \"history\"\n    @property\n    def buffer(self) -> List[BaseMessage]:\n        return self.chat_memory.messages\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        buffer = self.buffer\n        if self.moving_summary_buffer != \"\":\n            first_messages: List[BaseMessage] = [\n                self.summary_message_cls(content=self.moving_summary_buffer)\n            ]\n            buffer = first_messages + buffer\n        if self.return_messages:\n            final_buffer: Any = buffer\n        else:\n            final_buffer = get_buffer_string(\n                buffer, human_prefix=self.human_prefix, ai_prefix=self.ai_prefix\n            )\n        return {self.memory_key: final_buffer}\n    @root_validator()\n    def validate_prompt_input_variables(cls, values: Dict) -> Dict:\n        \"\"\"Validate that prompt input variables are consistent.\"\"\"\n        prompt_variables = values[\"prompt\"].input_variables\n        expected_keys = {\"summary\", \"new_lines\"}\n        if expected_keys != set(prompt_variables):\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/summary_buffer.html"}401{"id": "2f45db8bfcd1-1", "text": "if expected_keys != set(prompt_variables):\n            raise ValueError(\n                \"Got unexpected prompt input variables. The prompt expects \"\n                f\"{prompt_variables}, but it should have {expected_keys}.\"\n            )\n        return values\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        super().save_context(inputs, outputs)\n        self.prune()\n[docs]    def prune(self) -> None:\n        \"\"\"Prune buffer if it exceeds max token limit\"\"\"\n        buffer = self.chat_memory.messages\n        curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)\n        if curr_buffer_length > self.max_token_limit:\n            pruned_memory = []\n            while curr_buffer_length > self.max_token_limit:\n                pruned_memory.append(buffer.pop(0))\n                curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)\n            self.moving_summary_buffer = self.predict_new_summary(\n                pruned_memory, self.moving_summary_buffer\n            )\n[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        super().clear()\n        self.moving_summary_buffer = \"\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/summary_buffer.html"}402{"id": "e05d80b285e8-0", "text": "Source code for langchain.memory.kg\nfrom typing import Any, Dict, List, Type, Union\nfrom pydantic import Field\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.llm import LLMChain\nfrom langchain.graphs import NetworkxEntityGraph\nfrom langchain.graphs.networkx_graph import KnowledgeTriple, get_entities, parse_triples\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.memory.prompt import (\n    ENTITY_EXTRACTION_PROMPT,\n    KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT,\n)\nfrom langchain.memory.utils import get_prompt_input_key\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.schema import (\n    BaseMessage,\n    SystemMessage,\n    get_buffer_string,\n)\n[docs]class ConversationKGMemory(BaseChatMemory):\n    \"\"\"Knowledge graph memory for storing conversation memory.\n    Integrates with external knowledge graph to store and retrieve\n    information about knowledge triples in the conversation.\n    \"\"\"\n    k: int = 2\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    kg: NetworkxEntityGraph = Field(default_factory=NetworkxEntityGraph)\n    knowledge_extraction_prompt: BasePromptTemplate = KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT\n    entity_extraction_prompt: BasePromptTemplate = ENTITY_EXTRACTION_PROMPT\n    llm: BaseLanguageModel\n    summary_message_cls: Type[BaseMessage] = SystemMessage\n    \"\"\"Number of previous utterances to include in the context.\"\"\"\n    memory_key: str = \"history\"  #: :meta private:\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        entities = self._get_current_entities(inputs)\n        summary_strings = []", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/kg.html"}403{"id": "e05d80b285e8-1", "text": "entities = self._get_current_entities(inputs)\n        summary_strings = []\n        for entity in entities:\n            knowledge = self.kg.get_entity_knowledge(entity)\n            if knowledge:\n                summary = f\"On {entity}: {'. '.join(knowledge)}.\"\n                summary_strings.append(summary)\n        context: Union[str, List]\n        if not summary_strings:\n            context = [] if self.return_messages else \"\"\n        elif self.return_messages:\n            context = [\n                self.summary_message_cls(content=text) for text in summary_strings\n            ]\n        else:\n            context = \"\\n\".join(summary_strings)\n        return {self.memory_key: context}\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n    def _get_prompt_input_key(self, inputs: Dict[str, Any]) -> str:\n        \"\"\"Get the input key for the prompt.\"\"\"\n        if self.input_key is None:\n            return get_prompt_input_key(inputs, self.memory_variables)\n        return self.input_key\n    def _get_prompt_output_key(self, outputs: Dict[str, Any]) -> str:\n        \"\"\"Get the output key for the prompt.\"\"\"\n        if self.output_key is None:\n            if len(outputs) != 1:\n                raise ValueError(f\"One output key expected, got {outputs.keys()}\")\n            return list(outputs.keys())[0]\n        return self.output_key\n[docs]    def get_current_entities(self, input_string: str) -> List[str]:\n        chain = LLMChain(llm=self.llm, prompt=self.entity_extraction_prompt)\n        buffer_string = get_buffer_string(\n            self.chat_memory.messages[-self.k * 2 :],\n            human_prefix=self.human_prefix,", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/kg.html"}404{"id": "e05d80b285e8-2", "text": "human_prefix=self.human_prefix,\n            ai_prefix=self.ai_prefix,\n        )\n        output = chain.predict(\n            history=buffer_string,\n            input=input_string,\n        )\n        return get_entities(output)\n    def _get_current_entities(self, inputs: Dict[str, Any]) -> List[str]:\n        \"\"\"Get the current entities in the conversation.\"\"\"\n        prompt_input_key = self._get_prompt_input_key(inputs)\n        return self.get_current_entities(inputs[prompt_input_key])\n[docs]    def get_knowledge_triplets(self, input_string: str) -> List[KnowledgeTriple]:\n        chain = LLMChain(llm=self.llm, prompt=self.knowledge_extraction_prompt)\n        buffer_string = get_buffer_string(\n            self.chat_memory.messages[-self.k * 2 :],\n            human_prefix=self.human_prefix,\n            ai_prefix=self.ai_prefix,\n        )\n        output = chain.predict(\n            history=buffer_string,\n            input=input_string,\n            verbose=True,\n        )\n        knowledge = parse_triples(output)\n        return knowledge\n    def _get_and_update_kg(self, inputs: Dict[str, Any]) -> None:\n        \"\"\"Get and update knowledge graph from the conversation history.\"\"\"\n        prompt_input_key = self._get_prompt_input_key(inputs)\n        knowledge = self.get_knowledge_triplets(inputs[prompt_input_key])\n        for triple in knowledge:\n            self.kg.add_triple(triple)\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        super().save_context(inputs, outputs)\n        self._get_and_update_kg(inputs)\n[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        super().clear()", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/kg.html"}405{"id": "e05d80b285e8-3", "text": "\"\"\"Clear memory contents.\"\"\"\n        super().clear()\n        self.kg.clear()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/kg.html"}406{"id": "5ec395642e44-0", "text": "Source code for langchain.memory.buffer\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import root_validator\nfrom langchain.memory.chat_memory import BaseChatMemory, BaseMemory\nfrom langchain.memory.utils import get_prompt_input_key\nfrom langchain.schema import get_buffer_string\n[docs]class ConversationBufferMemory(BaseChatMemory):\n    \"\"\"Buffer for storing conversation memory.\"\"\"\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    memory_key: str = \"history\"  #: :meta private:\n    @property\n    def buffer(self) -> Any:\n        \"\"\"String buffer of memory.\"\"\"\n        if self.return_messages:\n            return self.chat_memory.messages\n        else:\n            return get_buffer_string(\n                self.chat_memory.messages,\n                human_prefix=self.human_prefix,\n                ai_prefix=self.ai_prefix,\n            )\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        return {self.memory_key: self.buffer}\n[docs]class ConversationStringBufferMemory(BaseMemory):\n    \"\"\"Buffer for storing conversation memory.\"\"\"\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    \"\"\"Prefix to use for AI generated responses.\"\"\"\n    buffer: str = \"\"\n    output_key: Optional[str] = None\n    input_key: Optional[str] = None\n    memory_key: str = \"history\"  #: :meta private:\n    @root_validator()\n    def validate_chains(cls, values: Dict) -> Dict:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/buffer.html"}407{"id": "5ec395642e44-1", "text": "@root_validator()\n    def validate_chains(cls, values: Dict) -> Dict:\n        \"\"\"Validate that return messages is not True.\"\"\"\n        if values.get(\"return_messages\", False):\n            raise ValueError(\n                \"return_messages must be False for ConversationStringBufferMemory\"\n            )\n        return values\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        \"\"\"Return history buffer.\"\"\"\n        return {self.memory_key: self.buffer}\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        if self.input_key is None:\n            prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)\n        else:\n            prompt_input_key = self.input_key\n        if self.output_key is None:\n            if len(outputs) != 1:\n                raise ValueError(f\"One output key expected, got {outputs.keys()}\")\n            output_key = list(outputs.keys())[0]\n        else:\n            output_key = self.output_key\n        human = f\"{self.human_prefix}: \" + inputs[prompt_input_key]\n        ai = f\"{self.ai_prefix}: \" + outputs[output_key]\n        self.buffer += \"\\n\" + \"\\n\".join([human, ai])\n[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        self.buffer = \"\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/buffer.html"}408{"id": "682b29ee7dff-0", "text": "Source code for langchain.memory.entity\nimport logging\nfrom abc import ABC, abstractmethod\nfrom itertools import islice\nfrom typing import Any, Dict, Iterable, List, Optional\nfrom pydantic import Field\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.llm import LLMChain\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.memory.prompt import (\n    ENTITY_EXTRACTION_PROMPT,\n    ENTITY_SUMMARIZATION_PROMPT,\n)\nfrom langchain.memory.utils import get_prompt_input_key\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.schema import BaseMessage, get_buffer_string\nlogger = logging.getLogger(__name__)\nclass BaseEntityStore(ABC):\n    @abstractmethod\n    def get(self, key: str, default: Optional[str] = None) -> Optional[str]:\n        \"\"\"Get entity value from store.\"\"\"\n        pass\n    @abstractmethod\n    def set(self, key: str, value: Optional[str]) -> None:\n        \"\"\"Set entity value in store.\"\"\"\n        pass\n    @abstractmethod\n    def delete(self, key: str) -> None:\n        \"\"\"Delete entity value from store.\"\"\"\n        pass\n    @abstractmethod\n    def exists(self, key: str) -> bool:\n        \"\"\"Check if entity exists in store.\"\"\"\n        pass\n    @abstractmethod\n    def clear(self) -> None:\n        \"\"\"Delete all entities from store.\"\"\"\n        pass\n[docs]class InMemoryEntityStore(BaseEntityStore):\n    \"\"\"Basic in-memory entity store.\"\"\"\n    store: Dict[str, Optional[str]] = {}\n[docs]    def get(self, key: str, default: Optional[str] = None) -> Optional[str]:\n        return self.store.get(key, default)\n[docs]    def set(self, key: str, value: Optional[str]) -> None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}409{"id": "682b29ee7dff-1", "text": "[docs]    def set(self, key: str, value: Optional[str]) -> None:\n        self.store[key] = value\n[docs]    def delete(self, key: str) -> None:\n        del self.store[key]\n[docs]    def exists(self, key: str) -> bool:\n        return key in self.store\n[docs]    def clear(self) -> None:\n        return self.store.clear()\n[docs]class RedisEntityStore(BaseEntityStore):\n    \"\"\"Redis-backed Entity store. Entities get a TTL of 1 day by default, and\n    that TTL is extended by 3 days every time the entity is read back.\n    \"\"\"\n    redis_client: Any\n    session_id: str = \"default\"\n    key_prefix: str = \"memory_store\"\n    ttl: Optional[int] = 60 * 60 * 24\n    recall_ttl: Optional[int] = 60 * 60 * 24 * 3\n    def __init__(\n        self,\n        session_id: str = \"default\",\n        url: str = \"redis://localhost:6379/0\",\n        key_prefix: str = \"memory_store\",\n        ttl: Optional[int] = 60 * 60 * 24,\n        recall_ttl: Optional[int] = 60 * 60 * 24 * 3,\n        *args: Any,\n        **kwargs: Any,\n    ):\n        try:\n            import redis\n        except ImportError:\n            raise ImportError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        super().__init__(*args, **kwargs)\n        try:\n            self.redis_client = redis.Redis.from_url(url=url, decode_responses=True)\n        except redis.exceptions.ConnectionError as error:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}410{"id": "682b29ee7dff-2", "text": "except redis.exceptions.ConnectionError as error:\n            logger.error(error)\n        self.session_id = session_id\n        self.key_prefix = key_prefix\n        self.ttl = ttl\n        self.recall_ttl = recall_ttl or ttl\n    @property\n    def full_key_prefix(self) -> str:\n        return f\"{self.key_prefix}:{self.session_id}\"\n[docs]    def get(self, key: str, default: Optional[str] = None) -> Optional[str]:\n        res = (\n            self.redis_client.getex(f\"{self.full_key_prefix}:{key}\", ex=self.recall_ttl)\n            or default\n            or \"\"\n        )\n        logger.debug(f\"REDIS MEM get '{self.full_key_prefix}:{key}': '{res}'\")\n        return res\n[docs]    def set(self, key: str, value: Optional[str]) -> None:\n        if not value:\n            return self.delete(key)\n        self.redis_client.set(f\"{self.full_key_prefix}:{key}\", value, ex=self.ttl)\n        logger.debug(\n            f\"REDIS MEM set '{self.full_key_prefix}:{key}': '{value}' EX {self.ttl}\"\n        )\n[docs]    def delete(self, key: str) -> None:\n        self.redis_client.delete(f\"{self.full_key_prefix}:{key}\")\n[docs]    def exists(self, key: str) -> bool:\n        return self.redis_client.exists(f\"{self.full_key_prefix}:{key}\") == 1\n[docs]    def clear(self) -> None:\n        # iterate a list in batches of size batch_size\n        def batched(iterable: Iterable[Any], batch_size: int) -> Iterable[Any]:\n            iterator = iter(iterable)\n            while batch := list(islice(iterator, batch_size)):\n                yield batch\n        for keybatch in batched(", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}411{"id": "682b29ee7dff-3", "text": "yield batch\n        for keybatch in batched(\n            self.redis_client.scan_iter(f\"{self.full_key_prefix}:*\"), 500\n        ):\n            self.redis_client.delete(*keybatch)\n[docs]class ConversationEntityMemory(BaseChatMemory):\n    \"\"\"Entity extractor & summarizer to memory.\"\"\"\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    llm: BaseLanguageModel\n    entity_extraction_prompt: BasePromptTemplate = ENTITY_EXTRACTION_PROMPT\n    entity_summarization_prompt: BasePromptTemplate = ENTITY_SUMMARIZATION_PROMPT\n    entity_cache: List[str] = []\n    k: int = 3\n    chat_history_key: str = \"history\"\n    entity_store: BaseEntityStore = Field(default_factory=InMemoryEntityStore)\n    @property\n    def buffer(self) -> List[BaseMessage]:\n        return self.chat_memory.messages\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [\"entities\", self.chat_history_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        chain = LLMChain(llm=self.llm, prompt=self.entity_extraction_prompt)\n        if self.input_key is None:\n            prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)\n        else:\n            prompt_input_key = self.input_key\n        buffer_string = get_buffer_string(\n            self.buffer[-self.k * 2 :],\n            human_prefix=self.human_prefix,\n            ai_prefix=self.ai_prefix,\n        )\n        output = chain.predict(\n            history=buffer_string,\n            input=inputs[prompt_input_key],\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}412{"id": "682b29ee7dff-4", "text": "history=buffer_string,\n            input=inputs[prompt_input_key],\n        )\n        if output.strip() == \"NONE\":\n            entities = []\n        else:\n            entities = [w.strip() for w in output.split(\",\")]\n        entity_summaries = {}\n        for entity in entities:\n            entity_summaries[entity] = self.entity_store.get(entity, \"\")\n        self.entity_cache = entities\n        if self.return_messages:\n            buffer: Any = self.buffer[-self.k * 2 :]\n        else:\n            buffer = buffer_string\n        return {\n            self.chat_history_key: buffer,\n            \"entities\": entity_summaries,\n        }\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        super().save_context(inputs, outputs)\n        if self.input_key is None:\n            prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)\n        else:\n            prompt_input_key = self.input_key\n        buffer_string = get_buffer_string(\n            self.buffer[-self.k * 2 :],\n            human_prefix=self.human_prefix,\n            ai_prefix=self.ai_prefix,\n        )\n        input_data = inputs[prompt_input_key]\n        chain = LLMChain(llm=self.llm, prompt=self.entity_summarization_prompt)\n        for entity in self.entity_cache:\n            existing_summary = self.entity_store.get(entity, \"\")\n            output = chain.predict(\n                summary=existing_summary,\n                entity=entity,\n                history=buffer_string,\n                input=input_data,\n            )\n            self.entity_store.set(entity, output.strip())\n[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        self.chat_memory.clear()", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}413{"id": "682b29ee7dff-5", "text": "\"\"\"Clear memory contents.\"\"\"\n        self.chat_memory.clear()\n        self.entity_cache.clear()\n        self.entity_store.clear()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/entity.html"}414{"id": "e90f19e560be-0", "text": "Source code for langchain.memory.combined\nimport warnings\nfrom typing import Any, Dict, List, Set\nfrom pydantic import validator\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.schema import BaseMemory\n[docs]class CombinedMemory(BaseMemory):\n    \"\"\"Class for combining multiple memories' data together.\"\"\"\n    memories: List[BaseMemory]\n    \"\"\"For tracking all the memories that should be accessed.\"\"\"\n    @validator(\"memories\")\n    def check_repeated_memory_variable(\n        cls, value: List[BaseMemory]\n    ) -> List[BaseMemory]:\n        all_variables: Set[str] = set()\n        for val in value:\n            overlap = all_variables.intersection(val.memory_variables)\n            if overlap:\n                raise ValueError(\n                    f\"The same variables {overlap} are found in multiple\"\n                    \"memory object, which is not allowed by CombinedMemory.\"\n                )\n            all_variables |= set(val.memory_variables)\n        return value\n    @validator(\"memories\")\n    def check_input_key(cls, value: List[BaseMemory]) -> List[BaseMemory]:\n        \"\"\"Check that if memories are of type BaseChatMemory that input keys exist.\"\"\"\n        for val in value:\n            if isinstance(val, BaseChatMemory):\n                if val.input_key is None:\n                    warnings.warn(\n                        \"When using CombinedMemory, \"\n                        \"input keys should be so the input is known. \"\n                        f\" Was not set on {val}\"\n                    )\n        return value\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"All the memory variables that this instance provides.\"\"\"\n        \"\"\"Collected from the all the linked memories.\"\"\"\n        memory_variables = []\n        for memory in self.memories:\n            memory_variables.extend(memory.memory_variables)", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/combined.html"}415{"id": "e90f19e560be-1", "text": "for memory in self.memories:\n            memory_variables.extend(memory.memory_variables)\n        return memory_variables\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        \"\"\"Load all vars from sub-memories.\"\"\"\n        memory_data: Dict[str, Any] = {}\n        # Collect vars from all sub-memories\n        for memory in self.memories:\n            data = memory.load_memory_variables(inputs)\n            memory_data = {\n                **memory_data,\n                **data,\n            }\n        return memory_data\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this session for every memory.\"\"\"\n        # Save context for all sub-memories\n        for memory in self.memories:\n            memory.save_context(inputs, outputs)\n[docs]    def clear(self) -> None:\n        \"\"\"Clear context from this session for every memory.\"\"\"\n        for memory in self.memories:\n            memory.clear()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/combined.html"}416{"id": "0304a934f8bf-0", "text": "Source code for langchain.memory.summary\nfrom __future__ import annotations\nfrom typing import Any, Dict, List, Type\nfrom pydantic import BaseModel, root_validator\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.llm import LLMChain\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.memory.prompt import SUMMARY_PROMPT\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.schema import (\n    BaseChatMessageHistory,\n    BaseMessage,\n    SystemMessage,\n    get_buffer_string,\n)\nclass SummarizerMixin(BaseModel):\n    human_prefix: str = \"Human\"\n    ai_prefix: str = \"AI\"\n    llm: BaseLanguageModel\n    prompt: BasePromptTemplate = SUMMARY_PROMPT\n    summary_message_cls: Type[BaseMessage] = SystemMessage\n    def predict_new_summary(\n        self, messages: List[BaseMessage], existing_summary: str\n    ) -> str:\n        new_lines = get_buffer_string(\n            messages,\n            human_prefix=self.human_prefix,\n            ai_prefix=self.ai_prefix,\n        )\n        chain = LLMChain(llm=self.llm, prompt=self.prompt)\n        return chain.predict(summary=existing_summary, new_lines=new_lines)\n[docs]class ConversationSummaryMemory(BaseChatMemory, SummarizerMixin):\n    \"\"\"Conversation summarizer to memory.\"\"\"\n    buffer: str = \"\"\n    memory_key: str = \"history\"  #: :meta private:\n[docs]    @classmethod\n    def from_messages(\n        cls,\n        llm: BaseLanguageModel,\n        chat_memory: BaseChatMessageHistory,\n        *,\n        summarize_step: int = 2,\n        **kwargs: Any,\n    ) -> ConversationSummaryMemory:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/summary.html"}417{"id": "0304a934f8bf-1", "text": "**kwargs: Any,\n    ) -> ConversationSummaryMemory:\n        obj = cls(llm=llm, chat_memory=chat_memory, **kwargs)\n        for i in range(0, len(obj.chat_memory.messages), summarize_step):\n            obj.buffer = obj.predict_new_summary(\n                obj.chat_memory.messages[i : i + summarize_step], obj.buffer\n            )\n        return obj\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Will always return list of memory variables.\n        :meta private:\n        \"\"\"\n        return [self.memory_key]\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Return history buffer.\"\"\"\n        if self.return_messages:\n            buffer: Any = [self.summary_message_cls(content=self.buffer)]\n        else:\n            buffer = self.buffer\n        return {self.memory_key: buffer}\n    @root_validator()\n    def validate_prompt_input_variables(cls, values: Dict) -> Dict:\n        \"\"\"Validate that prompt input variables are consistent.\"\"\"\n        prompt_variables = values[\"prompt\"].input_variables\n        expected_keys = {\"summary\", \"new_lines\"}\n        if expected_keys != set(prompt_variables):\n            raise ValueError(\n                \"Got unexpected prompt input variables. The prompt expects \"\n                f\"{prompt_variables}, but it should have {expected_keys}.\"\n            )\n        return values\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Save context from this conversation to buffer.\"\"\"\n        super().save_context(inputs, outputs)\n        self.buffer = self.predict_new_summary(\n            self.chat_memory.messages[-2:], self.buffer\n        )\n[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/summary.html"}418{"id": "0304a934f8bf-2", "text": "[docs]    def clear(self) -> None:\n        \"\"\"Clear memory contents.\"\"\"\n        super().clear()\n        self.buffer = \"\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/summary.html"}419{"id": "da7368793a5b-0", "text": "Source code for langchain.memory.simple\nfrom typing import Any, Dict, List\nfrom langchain.schema import BaseMemory\n[docs]class SimpleMemory(BaseMemory):\n    \"\"\"Simple memory for storing context or other bits of information that shouldn't\n    ever change between prompts.\n    \"\"\"\n    memories: Dict[str, Any] = dict()\n    @property\n    def memory_variables(self) -> List[str]:\n        return list(self.memories.keys())\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        return self.memories\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Nothing should be saved or changed, my memory is set in stone.\"\"\"\n        pass\n[docs]    def clear(self) -> None:\n        \"\"\"Nothing to clear, got a memory like a vault.\"\"\"\n        pass\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/simple.html"}420{"id": "936c47534dd1-0", "text": "Source code for langchain.memory.readonly\nfrom typing import Any, Dict, List\nfrom langchain.schema import BaseMemory\n[docs]class ReadOnlySharedMemory(BaseMemory):\n    \"\"\"A memory wrapper that is read-only and cannot be changed.\"\"\"\n    memory: BaseMemory\n    @property\n    def memory_variables(self) -> List[str]:\n        \"\"\"Return memory variables.\"\"\"\n        return self.memory.memory_variables\n[docs]    def load_memory_variables(self, inputs: Dict[str, Any]) -> Dict[str, str]:\n        \"\"\"Load memory variables from memory.\"\"\"\n        return self.memory.load_memory_variables(inputs)\n[docs]    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:\n        \"\"\"Nothing should be saved or changed\"\"\"\n        pass\n[docs]    def clear(self) -> None:\n        \"\"\"Nothing to clear, got a memory like a vault.\"\"\"\n        pass\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/readonly.html"}421{"id": "92323cff4319-0", "text": "Source code for langchain.memory.chat_message_histories.dynamodb\nimport logging\nfrom typing import List\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n    messages_to_dict,\n)\nlogger = logging.getLogger(__name__)\n[docs]class DynamoDBChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"Chat message history that stores history in AWS DynamoDB.\n    This class expects that a DynamoDB table with name `table_name`\n    and a partition Key of `SessionId` is present.\n    Args:\n        table_name: name of the DynamoDB table\n        session_id: arbitrary key that is used to store the messages\n            of a single chat session.\n    \"\"\"\n    def __init__(self, table_name: str, session_id: str):\n        import boto3\n        client = boto3.resource(\"dynamodb\")\n        self.table = client.Table(table_name)\n        self.session_id = session_id\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from DynamoDB\"\"\"\n        from botocore.exceptions import ClientError\n        try:\n            response = self.table.get_item(Key={\"SessionId\": self.session_id})\n        except ClientError as error:\n            if error.response[\"Error\"][\"Code\"] == \"ResourceNotFoundException\":\n                logger.warning(\"No record found with session id: %s\", self.session_id)\n            else:\n                logger.error(error)\n        if response and \"Item\" in response:\n            items = response[\"Item\"][\"History\"]\n        else:\n            items = []\n        messages = messages_from_dict(items)\n        return messages", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/dynamodb.html"}422{"id": "92323cff4319-1", "text": "items = []\n        messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in DynamoDB\"\"\"\n        from botocore.exceptions import ClientError\n        messages = messages_to_dict(self.messages)\n        _message = _message_to_dict(message)\n        messages.append(_message)\n        try:\n            self.table.put_item(\n                Item={\"SessionId\": self.session_id, \"History\": messages}\n            )\n        except ClientError as err:\n            logger.error(err)\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from DynamoDB\"\"\"\n        from botocore.exceptions import ClientError\n        try:\n            self.table.delete_item(Key={\"SessionId\": self.session_id})\n        except ClientError as err:\n            logger.error(err)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/dynamodb.html"}423{"id": "e3becfd7d5aa-0", "text": "Source code for langchain.memory.chat_message_histories.momento\nfrom __future__ import annotations\nimport json\nfrom datetime import timedelta\nfrom typing import TYPE_CHECKING, Any, Optional\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n)\nfrom langchain.utils import get_from_env\nif TYPE_CHECKING:\n    import momento\ndef _ensure_cache_exists(cache_client: momento.CacheClient, cache_name: str) -> None:\n    \"\"\"Create cache if it doesn't exist.\n    Raises:\n        SdkException: Momento service or network error\n        Exception: Unexpected response\n    \"\"\"\n    from momento.responses import CreateCache\n    create_cache_response = cache_client.create_cache(cache_name)\n    if isinstance(create_cache_response, CreateCache.Success) or isinstance(\n        create_cache_response, CreateCache.CacheAlreadyExists\n    ):\n        return None\n    elif isinstance(create_cache_response, CreateCache.Error):\n        raise create_cache_response.inner_exception\n    else:\n        raise Exception(f\"Unexpected response cache creation: {create_cache_response}\")\n[docs]class MomentoChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"Chat message history cache that uses Momento as a backend.\n    See https://gomomento.com/\"\"\"\n    def __init__(\n        self,\n        session_id: str,\n        cache_client: momento.CacheClient,\n        cache_name: str,\n        *,\n        key_prefix: str = \"message_store:\",\n        ttl: Optional[timedelta] = None,\n        ensure_cache_exists: bool = True,\n    ):\n        \"\"\"Instantiate a chat message history cache that uses Momento as a backend.\n        Note: to instantiate the cache client passed to MomentoChatMessageHistory,", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/momento.html"}424{"id": "e3becfd7d5aa-1", "text": "Note: to instantiate the cache client passed to MomentoChatMessageHistory,\n        you must have a Momento account at https://gomomento.com/.\n        Args:\n            session_id (str): The session ID to use for this chat session.\n            cache_client (CacheClient): The Momento cache client.\n            cache_name (str): The name of the cache to use to store the messages.\n            key_prefix (str, optional): The prefix to apply to the cache key.\n                Defaults to \"message_store:\".\n            ttl (Optional[timedelta], optional): The TTL to use for the messages.\n                Defaults to None, ie the default TTL of the cache will be used.\n            ensure_cache_exists (bool, optional): Create the cache if it doesn't exist.\n                Defaults to True.\n        Raises:\n            ImportError: Momento python package is not installed.\n            TypeError: cache_client is not of type momento.CacheClientObject\n        \"\"\"\n        try:\n            from momento import CacheClient\n            from momento.requests import CollectionTtl\n        except ImportError:\n            raise ImportError(\n                \"Could not import momento python package. \"\n                \"Please install it with `pip install momento`.\"\n            )\n        if not isinstance(cache_client, CacheClient):\n            raise TypeError(\"cache_client must be a momento.CacheClient object.\")\n        if ensure_cache_exists:\n            _ensure_cache_exists(cache_client, cache_name)\n        self.key = key_prefix + session_id\n        self.cache_client = cache_client\n        self.cache_name = cache_name\n        if ttl is not None:\n            self.ttl = CollectionTtl.of(ttl)\n        else:\n            self.ttl = CollectionTtl.from_cache_ttl()\n[docs]    @classmethod\n    def from_client_params(\n        cls,\n        session_id: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/momento.html"}425{"id": "e3becfd7d5aa-2", "text": "def from_client_params(\n        cls,\n        session_id: str,\n        cache_name: str,\n        ttl: timedelta,\n        *,\n        configuration: Optional[momento.config.Configuration] = None,\n        auth_token: Optional[str] = None,\n        **kwargs: Any,\n    ) -> MomentoChatMessageHistory:\n        \"\"\"Construct cache from CacheClient parameters.\"\"\"\n        try:\n            from momento import CacheClient, Configurations, CredentialProvider\n        except ImportError:\n            raise ImportError(\n                \"Could not import momento python package. \"\n                \"Please install it with `pip install momento`.\"\n            )\n        if configuration is None:\n            configuration = Configurations.Laptop.v1()\n        auth_token = auth_token or get_from_env(\"auth_token\", \"MOMENTO_AUTH_TOKEN\")\n        credentials = CredentialProvider.from_string(auth_token)\n        cache_client = CacheClient(configuration, credentials, default_ttl=ttl)\n        return cls(session_id, cache_client, cache_name, ttl=ttl, **kwargs)\n    @property\n    def messages(self) -> list[BaseMessage]:  # type: ignore[override]\n        \"\"\"Retrieve the messages from Momento.\n        Raises:\n            SdkException: Momento service or network error\n            Exception: Unexpected response\n        Returns:\n            list[BaseMessage]: List of cached messages\n        \"\"\"\n        from momento.responses import CacheListFetch\n        fetch_response = self.cache_client.list_fetch(self.cache_name, self.key)\n        if isinstance(fetch_response, CacheListFetch.Hit):\n            items = [json.loads(m) for m in fetch_response.value_list_string]\n            return messages_from_dict(items)\n        elif isinstance(fetch_response, CacheListFetch.Miss):\n            return []\n        elif isinstance(fetch_response, CacheListFetch.Error):", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/momento.html"}426{"id": "e3becfd7d5aa-3", "text": "return []\n        elif isinstance(fetch_response, CacheListFetch.Error):\n            raise fetch_response.inner_exception\n        else:\n            raise Exception(f\"Unexpected response: {fetch_response}\")\n[docs]    def add_user_message(self, message: str) -> None:\n        \"\"\"Store a user message in the cache.\n        Args:\n            message (str): The message to store.\n        \"\"\"\n        self.__add_message(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        \"\"\"Store an AI message in the cache.\n        Args:\n            message (str): The message to store.\n        \"\"\"\n        self.__add_message(AIMessage(content=message))\n    def __add_message(self, message: BaseMessage) -> None:\n        \"\"\"Store a message in the cache.\n        Args:\n            message (BaseMessage): The message object to store.\n        Raises:\n            SdkException: Momento service or network error.\n            Exception: Unexpected response.\n        \"\"\"\n        from momento.responses import CacheListPushBack\n        item = json.dumps(_message_to_dict(message))\n        push_response = self.cache_client.list_push_back(\n            self.cache_name, self.key, item, ttl=self.ttl\n        )\n        if isinstance(push_response, CacheListPushBack.Success):\n            return None\n        elif isinstance(push_response, CacheListPushBack.Error):\n            raise push_response.inner_exception\n        else:\n            raise Exception(f\"Unexpected response: {push_response}\")\n[docs]    def clear(self) -> None:\n        \"\"\"Remove the session's messages from the cache.\n        Raises:\n            SdkException: Momento service or network error.\n            Exception: Unexpected response.\n        \"\"\"\n        from momento.responses import CacheDelete", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/momento.html"}427{"id": "e3becfd7d5aa-4", "text": "Exception: Unexpected response.\n        \"\"\"\n        from momento.responses import CacheDelete\n        delete_response = self.cache_client.delete(self.cache_name, self.key)\n        if isinstance(delete_response, CacheDelete.Success):\n            return None\n        elif isinstance(delete_response, CacheDelete.Error):\n            raise delete_response.inner_exception\n        else:\n            raise Exception(f\"Unexpected response: {delete_response}\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/momento.html"}428{"id": "82dfbfcfbf53-0", "text": "Source code for langchain.memory.chat_message_histories.redis\nimport json\nimport logging\nfrom typing import List, Optional\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n)\nlogger = logging.getLogger(__name__)\n[docs]class RedisChatMessageHistory(BaseChatMessageHistory):\n    def __init__(\n        self,\n        session_id: str,\n        url: str = \"redis://localhost:6379/0\",\n        key_prefix: str = \"message_store:\",\n        ttl: Optional[int] = None,\n    ):\n        try:\n            import redis\n        except ImportError:\n            raise ImportError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        try:\n            self.redis_client = redis.Redis.from_url(url=url)\n        except redis.exceptions.ConnectionError as error:\n            logger.error(error)\n        self.session_id = session_id\n        self.key_prefix = key_prefix\n        self.ttl = ttl\n    @property\n    def key(self) -> str:\n        \"\"\"Construct the record key to use\"\"\"\n        return self.key_prefix + self.session_id\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from Redis\"\"\"\n        _items = self.redis_client.lrange(self.key, 0, -1)\n        items = [json.loads(m.decode(\"utf-8\")) for m in _items[::-1]]\n        messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/redis.html"}429{"id": "82dfbfcfbf53-1", "text": "self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in Redis\"\"\"\n        self.redis_client.lpush(self.key, json.dumps(_message_to_dict(message)))\n        if self.ttl:\n            self.redis_client.expire(self.key, self.ttl)\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from Redis\"\"\"\n        self.redis_client.delete(self.key)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/redis.html"}430{"id": "733952677054-0", "text": "Source code for langchain.memory.chat_message_histories.postgres\nimport json\nimport logging\nfrom typing import List\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n)\nlogger = logging.getLogger(__name__)\nDEFAULT_CONNECTION_STRING = \"postgresql://postgres:mypassword@localhost/chat_history\"\n[docs]class PostgresChatMessageHistory(BaseChatMessageHistory):\n    def __init__(\n        self,\n        session_id: str,\n        connection_string: str = DEFAULT_CONNECTION_STRING,\n        table_name: str = \"message_store\",\n    ):\n        import psycopg\n        from psycopg.rows import dict_row\n        try:\n            self.connection = psycopg.connect(connection_string)\n            self.cursor = self.connection.cursor(row_factory=dict_row)\n        except psycopg.OperationalError as error:\n            logger.error(error)\n        self.session_id = session_id\n        self.table_name = table_name\n        self._create_table_if_not_exists()\n    def _create_table_if_not_exists(self) -> None:\n        create_table_query = f\"\"\"CREATE TABLE IF NOT EXISTS {self.table_name} (\n            id SERIAL PRIMARY KEY,\n            session_id TEXT NOT NULL,\n            message JSONB NOT NULL\n        );\"\"\"\n        self.cursor.execute(create_table_query)\n        self.connection.commit()\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from PostgreSQL\"\"\"\n        query = f\"SELECT message FROM {self.table_name} WHERE session_id = %s;\"\n        self.cursor.execute(query, (self.session_id,))\n        items = [record[\"message\"] for record in self.cursor.fetchall()]\n        messages = messages_from_dict(items)\n        return messages", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/postgres.html"}431{"id": "733952677054-1", "text": "messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in PostgreSQL\"\"\"\n        from psycopg import sql\n        query = sql.SQL(\"INSERT INTO {} (session_id, message) VALUES (%s, %s);\").format(\n            sql.Identifier(self.table_name)\n        )\n        self.cursor.execute(\n            query, (self.session_id, json.dumps(_message_to_dict(message)))\n        )\n        self.connection.commit()\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from PostgreSQL\"\"\"\n        query = f\"DELETE FROM {self.table_name} WHERE session_id = %s;\"\n        self.cursor.execute(query, (self.session_id,))\n        self.connection.commit()\n    def __del__(self) -> None:\n        if self.cursor:\n            self.cursor.close()\n        if self.connection:\n            self.connection.close()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/postgres.html"}432{"id": "73cb746ad493-0", "text": "Source code for langchain.memory.chat_message_histories.cassandra\nimport json\nimport logging\nfrom typing import List\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n)\nlogger = logging.getLogger(__name__)\nDEFAULT_KEYSPACE_NAME = \"chat_history\"\nDEFAULT_TABLE_NAME = \"message_store\"\nDEFAULT_USERNAME = \"cassandra\"\nDEFAULT_PASSWORD = \"cassandra\"\nDEFAULT_PORT = 9042\n[docs]class CassandraChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"Chat message history that stores history in Cassandra.\n    Args:\n        contact_points: list of ips to connect to Cassandra cluster\n        session_id: arbitrary key that is used to store the messages\n            of a single chat session.\n        port: port to connect to Cassandra cluster\n        username: username to connect to Cassandra cluster\n        password: password to connect to Cassandra cluster\n        keyspace_name: name of the keyspace to use\n        table_name: name of the table to use\n    \"\"\"\n    def __init__(\n        self,\n        contact_points: List[str],\n        session_id: str,\n        port: int = DEFAULT_PORT,\n        username: str = DEFAULT_USERNAME,\n        password: str = DEFAULT_PASSWORD,\n        keyspace_name: str = DEFAULT_KEYSPACE_NAME,\n        table_name: str = DEFAULT_TABLE_NAME,\n    ):\n        self.contact_points = contact_points\n        self.session_id = session_id\n        self.port = port\n        self.username = username\n        self.password = password\n        self.keyspace_name = keyspace_name\n        self.table_name = table_name\n        try:\n            from cassandra import (\n                AuthenticationFailed,\n                OperationTimedOut,", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cassandra.html"}433{"id": "73cb746ad493-1", "text": "from cassandra import (\n                AuthenticationFailed,\n                OperationTimedOut,\n                UnresolvableContactPoints,\n            )\n            from cassandra.cluster import Cluster, PlainTextAuthProvider\n        except ImportError:\n            raise ValueError(\n                \"Could not import cassandra-driver python package. \"\n                \"Please install it with `pip install cassandra-driver`.\"\n            )\n        self.cluster: Cluster = Cluster(\n            contact_points,\n            port=port,\n            auth_provider=PlainTextAuthProvider(\n                username=self.username, password=self.password\n            ),\n        )\n        try:\n            self.session = self.cluster.connect()\n        except (\n            AuthenticationFailed,\n            UnresolvableContactPoints,\n            OperationTimedOut,\n        ) as error:\n            logger.error(\n                \"Unable to establish connection with \\\n                cassandra chat message history database\"\n            )\n            raise error\n        self._prepare_cassandra()\n    def _prepare_cassandra(self) -> None:\n        \"\"\"Create the keyspace and table if they don't exist yet\"\"\"\n        from cassandra import OperationTimedOut, Unavailable\n        try:\n            self.session.execute(\n                f\"\"\"CREATE KEYSPACE IF NOT EXISTS \n                {self.keyspace_name} WITH REPLICATION = \n                {{ 'class' : 'SimpleStrategy', 'replication_factor' : 1 }};\"\"\"\n            )\n        except (OperationTimedOut, Unavailable) as error:\n            logger.error(\n                f\"Unable to create cassandra \\\n                chat message history keyspace: {self.keyspace_name}.\"\n            )\n            raise error\n        self.session.set_keyspace(self.keyspace_name)\n        try:\n            self.session.execute(\n                f\"\"\"CREATE TABLE IF NOT EXISTS", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cassandra.html"}434{"id": "73cb746ad493-2", "text": "try:\n            self.session.execute(\n                f\"\"\"CREATE TABLE IF NOT EXISTS \n                {self.table_name} (id UUID, session_id varchar, \n                history text,  PRIMARY KEY ((session_id), id) );\"\"\"\n            )\n        except (OperationTimedOut, Unavailable) as error:\n            logger.error(\n                f\"Unable to create cassandra \\\n                chat message history table: {self.table_name}\"\n            )\n            raise error\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from Cassandra\"\"\"\n        from cassandra import ReadFailure, ReadTimeout, Unavailable\n        try:\n            rows = self.session.execute(\n                f\"\"\"SELECT * FROM {self.table_name}\n                WHERE session_id = '{self.session_id}' ;\"\"\"\n            )\n        except (Unavailable, ReadTimeout, ReadFailure) as error:\n            logger.error(\"Unable to Retreive chat history messages from cassadra\")\n            raise error\n        if rows:\n            items = [json.loads(row.history) for row in rows]\n        else:\n            items = []\n        messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in Cassandra\"\"\"\n        import uuid\n        from cassandra import Unavailable, WriteFailure, WriteTimeout\n        try:\n            self.session.execute(\n                \"\"\"INSERT INTO message_store", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cassandra.html"}435{"id": "73cb746ad493-3", "text": "try:\n            self.session.execute(\n                \"\"\"INSERT INTO message_store\n                (id, session_id, history) VALUES (%s, %s, %s);\"\"\",\n                (uuid.uuid4(), self.session_id, json.dumps(_message_to_dict(message))),\n            )\n        except (Unavailable, WriteTimeout, WriteFailure) as error:\n            logger.error(\"Unable to write chat history messages to cassandra\")\n            raise error\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from Cassandra\"\"\"\n        from cassandra import OperationTimedOut, Unavailable\n        try:\n            self.session.execute(\n                f\"DELETE FROM {self.table_name} WHERE session_id = '{self.session_id}';\"\n            )\n        except (Unavailable, OperationTimedOut) as error:\n            logger.error(\"Unable to clear chat history messages from cassandra\")\n            raise error\n    def __del__(self) -> None:\n        if self.session:\n            self.session.shutdown()\n        if self.cluster:\n            self.cluster.shutdown()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cassandra.html"}436{"id": "cf79fb00636e-0", "text": "Source code for langchain.memory.chat_message_histories.file\nimport json\nimport logging\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    messages_from_dict,\n    messages_to_dict,\n)\nlogger = logging.getLogger(__name__)\n[docs]class FileChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"\n    Chat message history that stores history in a local file.\n    Args:\n        file_path: path of the local file to store the messages.\n    \"\"\"\n    def __init__(self, file_path: str):\n        self.file_path = Path(file_path)\n        if not self.file_path.exists():\n            self.file_path.touch()\n            self.file_path.write_text(json.dumps([]))\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from the local file\"\"\"\n        items = json.loads(self.file_path.read_text())\n        messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in the local file\"\"\"\n        messages = messages_to_dict(self.messages)\n        messages.append(messages_to_dict([message])[0])\n        self.file_path.write_text(json.dumps(messages))\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from the local file\"\"\"\n        self.file_path.write_text(json.dumps([]))\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/file.html"}437{"id": "cf79fb00636e-1", "text": "self.file_path.write_text(json.dumps([]))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/file.html"}438{"id": "66f9661c6dff-0", "text": "Source code for langchain.memory.chat_message_histories.cosmos_db\n\"\"\"Azure CosmosDB Memory History.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom types import TracebackType\nfrom typing import TYPE_CHECKING, Any, List, Optional, Type\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    messages_from_dict,\n    messages_to_dict,\n)\nlogger = logging.getLogger(__name__)\nif TYPE_CHECKING:\n    from azure.cosmos import ContainerProxy\n[docs]class CosmosDBChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"Chat history backed by Azure CosmosDB.\"\"\"\n    def __init__(\n        self,\n        cosmos_endpoint: str,\n        cosmos_database: str,\n        cosmos_container: str,\n        session_id: str,\n        user_id: str,\n        credential: Any = None,\n        connection_string: Optional[str] = None,\n        ttl: Optional[int] = None,\n        cosmos_client_kwargs: Optional[dict] = None,\n    ):\n        \"\"\"\n        Initializes a new instance of the CosmosDBChatMessageHistory class.\n        Make sure to call prepare_cosmos or use the context manager to make\n        sure your database is ready.\n        Either a credential or a connection string must be provided.\n        :param cosmos_endpoint: The connection endpoint for the Azure Cosmos DB account.\n        :param cosmos_database: The name of the database to use.\n        :param cosmos_container: The name of the container to use.\n        :param session_id: The session ID to use, can be overwritten while loading.\n        :param user_id: The user ID to use, can be overwritten while loading.\n        :param credential: The credential to use to authenticate to Azure Cosmos DB.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html"}439{"id": "66f9661c6dff-1", "text": ":param credential: The credential to use to authenticate to Azure Cosmos DB.\n        :param connection_string: The connection string to use to authenticate.\n        :param ttl: The time to live (in seconds) to use for documents in the container.\n        :param cosmos_client_kwargs: Additional kwargs to pass to the CosmosClient.\n        \"\"\"\n        self.cosmos_endpoint = cosmos_endpoint\n        self.cosmos_database = cosmos_database\n        self.cosmos_container = cosmos_container\n        self.credential = credential\n        self.conn_string = connection_string\n        self.session_id = session_id\n        self.user_id = user_id\n        self.ttl = ttl\n        self.messages: List[BaseMessage] = []\n        try:\n            from azure.cosmos import (  # pylint: disable=import-outside-toplevel # noqa: E501\n                CosmosClient,\n            )\n        except ImportError as exc:\n            raise ImportError(\n                \"You must install the azure-cosmos package to use the CosmosDBChatMessageHistory.\"  # noqa: E501\n            ) from exc\n        if self.credential:\n            self._client = CosmosClient(\n                url=self.cosmos_endpoint,\n                credential=self.credential,\n                **cosmos_client_kwargs or {},\n            )\n        elif self.conn_string:\n            self._client = CosmosClient.from_connection_string(\n                conn_str=self.conn_string,\n                **cosmos_client_kwargs or {},\n            )\n        else:\n            raise ValueError(\"Either a connection string or a credential must be set.\")\n        self._container: Optional[ContainerProxy] = None\n[docs]    def prepare_cosmos(self) -> None:\n        \"\"\"Prepare the CosmosDB client.\n        Use this function or the context manager to make sure your database is ready.\n        \"\"\"\n        try:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html"}440{"id": "66f9661c6dff-2", "text": "\"\"\"\n        try:\n            from azure.cosmos import (  # pylint: disable=import-outside-toplevel # noqa: E501\n                PartitionKey,\n            )\n        except ImportError as exc:\n            raise ImportError(\n                \"You must install the azure-cosmos package to use the CosmosDBChatMessageHistory.\"  # noqa: E501\n            ) from exc\n        database = self._client.create_database_if_not_exists(self.cosmos_database)\n        self._container = database.create_container_if_not_exists(\n            self.cosmos_container,\n            partition_key=PartitionKey(\"/user_id\"),\n            default_ttl=self.ttl,\n        )\n        self.load_messages()\n    def __enter__(self) -> \"CosmosDBChatMessageHistory\":\n        \"\"\"Context manager entry point.\"\"\"\n        self._client.__enter__()\n        self.prepare_cosmos()\n        return self\n    def __exit__(\n        self,\n        exc_type: Optional[Type[BaseException]],\n        exc_val: Optional[BaseException],\n        traceback: Optional[TracebackType],\n    ) -> None:\n        \"\"\"Context manager exit\"\"\"\n        self.upsert_messages()\n        self._client.__exit__(exc_type, exc_val, traceback)\n[docs]    def load_messages(self) -> None:\n        \"\"\"Retrieve the messages from Cosmos\"\"\"\n        if not self._container:\n            raise ValueError(\"Container not initialized\")\n        try:\n            from azure.cosmos.exceptions import (  # pylint: disable=import-outside-toplevel # noqa: E501\n                CosmosHttpResponseError,\n            )\n        except ImportError as exc:\n            raise ImportError(\n                \"You must install the azure-cosmos package to use the CosmosDBChatMessageHistory.\"  # noqa: E501\n            ) from exc\n        try:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html"}441{"id": "66f9661c6dff-3", "text": ") from exc\n        try:\n            item = self._container.read_item(\n                item=self.session_id, partition_key=self.user_id\n            )\n        except CosmosHttpResponseError:\n            logger.info(\"no session found\")\n            return\n        if \"messages\" in item and len(item[\"messages\"]) > 0:\n            self.messages = messages_from_dict(item[\"messages\"])\n[docs]    def add_user_message(self, message: str) -> None:\n        \"\"\"Add a user message to the memory.\"\"\"\n        self.upsert_messages(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        \"\"\"Add a AI message to the memory.\"\"\"\n        self.upsert_messages(AIMessage(content=message))\n[docs]    def upsert_messages(self, new_message: Optional[BaseMessage] = None) -> None:\n        \"\"\"Update the cosmosdb item.\"\"\"\n        if new_message:\n            self.messages.append(new_message)\n        if not self._container:\n            raise ValueError(\"Container not initialized\")\n        self._container.upsert_item(\n            body={\n                \"id\": self.session_id,\n                \"user_id\": self.user_id,\n                \"messages\": messages_to_dict(self.messages),\n            }\n        )\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from this memory and cosmos.\"\"\"\n        self.messages = []\n        if self._container:\n            self._container.delete_item(\n                item=self.session_id, partition_key=self.user_id\n            )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/cosmos_db.html"}442{"id": "22217ffbbfdd-0", "text": "Source code for langchain.memory.chat_message_histories.mongodb\nimport json\nimport logging\nfrom typing import List\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n    _message_to_dict,\n    messages_from_dict,\n)\nlogger = logging.getLogger(__name__)\nDEFAULT_DBNAME = \"chat_history\"\nDEFAULT_COLLECTION_NAME = \"message_store\"\n[docs]class MongoDBChatMessageHistory(BaseChatMessageHistory):\n    \"\"\"Chat message history that stores history in MongoDB.\n    Args:\n        connection_string: connection string to connect to MongoDB\n        session_id: arbitrary key that is used to store the messages\n            of a single chat session.\n        database_name: name of the database to use\n        collection_name: name of the collection to use\n    \"\"\"\n    def __init__(\n        self,\n        connection_string: str,\n        session_id: str,\n        database_name: str = DEFAULT_DBNAME,\n        collection_name: str = DEFAULT_COLLECTION_NAME,\n    ):\n        from pymongo import MongoClient, errors\n        self.connection_string = connection_string\n        self.session_id = session_id\n        self.database_name = database_name\n        self.collection_name = collection_name\n        try:\n            self.client: MongoClient = MongoClient(connection_string)\n        except errors.ConnectionFailure as error:\n            logger.error(error)\n        self.db = self.client[database_name]\n        self.collection = self.db[collection_name]\n    @property\n    def messages(self) -> List[BaseMessage]:  # type: ignore\n        \"\"\"Retrieve the messages from MongoDB\"\"\"\n        from pymongo import errors\n        try:\n            cursor = self.collection.find({\"SessionId\": self.session_id})\n        except errors.OperationFailure as error:\n            logger.error(error)\n        if cursor:", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/mongodb.html"}443{"id": "22217ffbbfdd-1", "text": "except errors.OperationFailure as error:\n            logger.error(error)\n        if cursor:\n            items = [json.loads(document[\"History\"]) for document in cursor]\n        else:\n            items = []\n        messages = messages_from_dict(items)\n        return messages\n[docs]    def add_user_message(self, message: str) -> None:\n        self.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.append(AIMessage(content=message))\n[docs]    def append(self, message: BaseMessage) -> None:\n        \"\"\"Append the message to the record in MongoDB\"\"\"\n        from pymongo import errors\n        try:\n            self.collection.insert_one(\n                {\n                    \"SessionId\": self.session_id,\n                    \"History\": json.dumps(_message_to_dict(message)),\n                }\n            )\n        except errors.WriteError as err:\n            logger.error(err)\n[docs]    def clear(self) -> None:\n        \"\"\"Clear session memory from MongoDB\"\"\"\n        from pymongo import errors\n        try:\n            self.collection.delete_many({\"SessionId\": self.session_id})\n        except errors.WriteError as err:\n            logger.error(err)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/mongodb.html"}444{"id": "b107c0e9571a-0", "text": "Source code for langchain.memory.chat_message_histories.in_memory\nfrom typing import List\nfrom pydantic import BaseModel\nfrom langchain.schema import (\n    AIMessage,\n    BaseChatMessageHistory,\n    BaseMessage,\n    HumanMessage,\n)\n[docs]class ChatMessageHistory(BaseChatMessageHistory, BaseModel):\n    messages: List[BaseMessage] = []\n[docs]    def add_user_message(self, message: str) -> None:\n        self.messages.append(HumanMessage(content=message))\n[docs]    def add_ai_message(self, message: str) -> None:\n        self.messages.append(AIMessage(content=message))\n[docs]    def clear(self) -> None:\n        self.messages = []\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/memory/chat_message_histories/in_memory.html"}445{"id": "88760a80a428-0", "text": "Source code for langchain.chat_models.azure_openai\n\"\"\"Azure OpenAI chat wrapper.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, Dict, Mapping\nfrom pydantic import root_validator\nfrom langchain.chat_models.openai import ChatOpenAI\nfrom langchain.schema import ChatResult\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class AzureChatOpenAI(ChatOpenAI):\n    \"\"\"Wrapper around Azure OpenAI Chat Completion API. To use this class you\n    must have a deployed model on Azure OpenAI. Use `deployment_name` in the\n    constructor to refer to the \"Model deployment name\" in the Azure portal.\n    In addition, you should have the ``openai`` python package installed, and the\n    following environment variables set or passed in constructor in lower case:\n    - ``OPENAI_API_TYPE`` (default: ``azure``)\n    - ``OPENAI_API_KEY``\n    - ``OPENAI_API_BASE``\n    - ``OPENAI_API_VERSION``\n    - ``OPENAI_PROXY``\n    For exmaple, if you have `gpt-35-turbo` deployed, with the deployment name\n    `35-turbo-dev`, the constructor should look like:\n    .. code-block:: python\n        AzureChatOpenAI(\n            deployment_name=\"35-turbo-dev\",\n            openai_api_version=\"2023-03-15-preview\",\n        )\n    Be aware the API version may change.\n    Any parameters that are valid to be passed to the openai.create call can be passed\n    in, even if not explicitly saved on this class.\n    \"\"\"\n    deployment_name: str = \"\"\n    openai_api_type: str = \"azure\"\n    openai_api_base: str = \"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/azure_openai.html"}446{"id": "88760a80a428-1", "text": "openai_api_base: str = \"\"\n    openai_api_version: str = \"\"\n    openai_api_key: str = \"\"\n    openai_organization: str = \"\"\n    openai_proxy: str = \"\"\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        openai_api_key = get_from_dict_or_env(\n            values,\n            \"openai_api_key\",\n            \"OPENAI_API_KEY\",\n        )\n        openai_api_base = get_from_dict_or_env(\n            values,\n            \"openai_api_base\",\n            \"OPENAI_API_BASE\",\n        )\n        openai_api_version = get_from_dict_or_env(\n            values,\n            \"openai_api_version\",\n            \"OPENAI_API_VERSION\",\n        )\n        openai_api_type = get_from_dict_or_env(\n            values,\n            \"openai_api_type\",\n            \"OPENAI_API_TYPE\",\n        )\n        openai_organization = get_from_dict_or_env(\n            values,\n            \"openai_organization\",\n            \"OPENAI_ORGANIZATION\",\n            default=\"\",\n        )\n        openai_proxy = get_from_dict_or_env(\n            values,\n            \"openai_proxy\",\n            \"OPENAI_PROXY\",\n            default=\"\",\n        )\n        try:\n            import openai\n            openai.api_type = openai_api_type\n            openai.api_base = openai_api_base\n            openai.api_version = openai_api_version\n            openai.api_key = openai_api_key\n            if openai_organization:\n                openai.organization = openai_organization\n            if openai_proxy:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/azure_openai.html"}447{"id": "88760a80a428-2", "text": "openai.organization = openai_organization\n            if openai_proxy:\n                openai.proxy = {\"http\": openai_proxy, \"https\": openai_proxy}  # type: ignore[assignment]  # noqa: E501\n        except ImportError:\n            raise ImportError(\n                \"Could not import openai python package. \"\n                \"Please install it with `pip install openai`.\"\n            )\n        try:\n            values[\"client\"] = openai.ChatCompletion\n        except AttributeError:\n            raise ValueError(\n                \"`openai` has no `ChatCompletion` attribute, this is likely \"\n                \"due to an old version of the openai package. Try upgrading it \"\n                \"with `pip install --upgrade openai`.\"\n            )\n        if values[\"n\"] < 1:\n            raise ValueError(\"n must be at least 1.\")\n        if values[\"n\"] > 1 and values[\"streaming\"]:\n            raise ValueError(\"n must be 1 when streaming.\")\n        return values\n    @property\n    def _default_params(self) -> Dict[str, Any]:\n        \"\"\"Get the default parameters for calling OpenAI API.\"\"\"\n        return {\n            **super()._default_params,\n            \"engine\": self.deployment_name,\n        }\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {**self._default_params}\n    @property\n    def _llm_type(self) -> str:\n        return \"azure-openai-chat\"\n    def _create_chat_result(self, response: Mapping[str, Any]) -> ChatResult:\n        for res in response[\"choices\"]:\n            if res.get(\"finish_reason\", None) == \"content_filter\":\n                raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/azure_openai.html"}448{"id": "88760a80a428-3", "text": "if res.get(\"finish_reason\", None) == \"content_filter\":\n                raise ValueError(\n                    \"Azure has not provided the response due to a content\"\n                    \" filter being triggered\"\n                )\n        return super()._create_chat_result(response)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/azure_openai.html"}449{"id": "a339475cd469-0", "text": "Source code for langchain.chat_models.google_palm\n\"\"\"Wrapper around Google's PaLM Chat API.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom typing import TYPE_CHECKING, Any, Callable, Dict, List, Mapping, Optional\nfrom pydantic import BaseModel, root_validator\nfrom tenacity import (\n    before_sleep_log,\n    retry,\n    retry_if_exception_type,\n    stop_after_attempt,\n    wait_exponential,\n)\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForLLMRun,\n    CallbackManagerForLLMRun,\n)\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    ChatGeneration,\n    ChatMessage,\n    ChatResult,\n    HumanMessage,\n    SystemMessage,\n)\nfrom langchain.utils import get_from_dict_or_env\nif TYPE_CHECKING:\n    import google.generativeai as genai\nlogger = logging.getLogger(__name__)\nclass ChatGooglePalmError(Exception):\n    pass\ndef _truncate_at_stop_tokens(\n    text: str,\n    stop: Optional[List[str]],\n) -> str:\n    \"\"\"Truncates text at the earliest stop token found.\"\"\"\n    if stop is None:\n        return text\n    for stop_token in stop:\n        stop_token_idx = text.find(stop_token)\n        if stop_token_idx != -1:\n            text = text[:stop_token_idx]\n    return text\ndef _response_to_result(\n    response: genai.types.ChatResponse,\n    stop: Optional[List[str]],\n) -> ChatResult:\n    \"\"\"Converts a PaLM API response into a LangChain ChatResult.\"\"\"\n    if not response.candidates:\n        raise ChatGooglePalmError(\"ChatResponse must have at least one candidate.\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}450{"id": "a339475cd469-1", "text": "raise ChatGooglePalmError(\"ChatResponse must have at least one candidate.\")\n    generations: List[ChatGeneration] = []\n    for candidate in response.candidates:\n        author = candidate.get(\"author\")\n        if author is None:\n            raise ChatGooglePalmError(f\"ChatResponse must have an author: {candidate}\")\n        content = _truncate_at_stop_tokens(candidate.get(\"content\", \"\"), stop)\n        if content is None:\n            raise ChatGooglePalmError(f\"ChatResponse must have a content: {candidate}\")\n        if author == \"ai\":\n            generations.append(\n                ChatGeneration(text=content, message=AIMessage(content=content))\n            )\n        elif author == \"human\":\n            generations.append(\n                ChatGeneration(\n                    text=content,\n                    message=HumanMessage(content=content),\n                )\n            )\n        else:\n            generations.append(\n                ChatGeneration(\n                    text=content,\n                    message=ChatMessage(role=author, content=content),\n                )\n            )\n    return ChatResult(generations=generations)\ndef _messages_to_prompt_dict(\n    input_messages: List[BaseMessage],\n) -> genai.types.MessagePromptDict:\n    \"\"\"Converts a list of LangChain messages into a PaLM API MessagePrompt structure.\"\"\"\n    import google.generativeai as genai\n    context: str = \"\"\n    examples: List[genai.types.MessageDict] = []\n    messages: List[genai.types.MessageDict] = []\n    remaining = list(enumerate(input_messages))\n    while remaining:\n        index, input_message = remaining.pop(0)\n        if isinstance(input_message, SystemMessage):\n            if index != 0:\n                raise ChatGooglePalmError(\"System message must be first input message.\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}451{"id": "a339475cd469-2", "text": "raise ChatGooglePalmError(\"System message must be first input message.\")\n            context = input_message.content\n        elif isinstance(input_message, HumanMessage) and input_message.example:\n            if messages:\n                raise ChatGooglePalmError(\n                    \"Message examples must come before other messages.\"\n                )\n            _, next_input_message = remaining.pop(0)\n            if isinstance(next_input_message, AIMessage) and next_input_message.example:\n                examples.extend(\n                    [\n                        genai.types.MessageDict(\n                            author=\"human\", content=input_message.content\n                        ),\n                        genai.types.MessageDict(\n                            author=\"ai\", content=next_input_message.content\n                        ),\n                    ]\n                )\n            else:\n                raise ChatGooglePalmError(\n                    \"Human example message must be immediately followed by an \"\n                    \" AI example response.\"\n                )\n        elif isinstance(input_message, AIMessage) and input_message.example:\n            raise ChatGooglePalmError(\n                \"AI example message must be immediately preceded by a Human \"\n                \"example message.\"\n            )\n        elif isinstance(input_message, AIMessage):\n            messages.append(\n                genai.types.MessageDict(author=\"ai\", content=input_message.content)\n            )\n        elif isinstance(input_message, HumanMessage):\n            messages.append(\n                genai.types.MessageDict(author=\"human\", content=input_message.content)\n            )\n        elif isinstance(input_message, ChatMessage):\n            messages.append(\n                genai.types.MessageDict(\n                    author=input_message.role, content=input_message.content\n                )\n            )\n        else:\n            raise ChatGooglePalmError(\n                \"Messages without an explicit role not supported by PaLM API.\"\n            )\n    return genai.types.MessagePromptDict(\n        context=context,\n        examples=examples,", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}452{"id": "a339475cd469-3", "text": "return genai.types.MessagePromptDict(\n        context=context,\n        examples=examples,\n        messages=messages,\n    )\ndef _create_retry_decorator() -> Callable[[Any], Any]:\n    \"\"\"Returns a tenacity retry decorator, preconfigured to handle PaLM exceptions\"\"\"\n    import google.api_core.exceptions\n    multiplier = 2\n    min_seconds = 1\n    max_seconds = 60\n    max_retries = 10\n    return retry(\n        reraise=True,\n        stop=stop_after_attempt(max_retries),\n        wait=wait_exponential(multiplier=multiplier, min=min_seconds, max=max_seconds),\n        retry=(\n            retry_if_exception_type(google.api_core.exceptions.ResourceExhausted)\n            | retry_if_exception_type(google.api_core.exceptions.ServiceUnavailable)\n            | retry_if_exception_type(google.api_core.exceptions.GoogleAPIError)\n        ),\n        before_sleep=before_sleep_log(logger, logging.WARNING),\n    )\ndef chat_with_retry(llm: ChatGooglePalm, **kwargs: Any) -> Any:\n    \"\"\"Use tenacity to retry the completion call.\"\"\"\n    retry_decorator = _create_retry_decorator()\n    @retry_decorator\n    def _chat_with_retry(**kwargs: Any) -> Any:\n        return llm.client.chat(**kwargs)\n    return _chat_with_retry(**kwargs)\nasync def achat_with_retry(llm: ChatGooglePalm, **kwargs: Any) -> Any:\n    \"\"\"Use tenacity to retry the async completion call.\"\"\"\n    retry_decorator = _create_retry_decorator()\n    @retry_decorator\n    async def _achat_with_retry(**kwargs: Any) -> Any:\n        # Use OpenAI's async api https://github.com/openai/openai-python#async-api\n        return await llm.client.chat_async(**kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}453{"id": "a339475cd469-4", "text": "return await llm.client.chat_async(**kwargs)\n    return await _achat_with_retry(**kwargs)\n[docs]class ChatGooglePalm(BaseChatModel, BaseModel):\n    \"\"\"Wrapper around Google's PaLM Chat API.\n    To use you must have the google.generativeai Python package installed and\n    either:\n        1. The ``GOOGLE_API_KEY``` environment varaible set with your API key, or\n        2. Pass your API key using the google_api_key kwarg to the ChatGoogle\n           constructor.\n    Example:\n        .. code-block:: python\n            from langchain.chat_models import ChatGooglePalm\n            chat = ChatGooglePalm()\n    \"\"\"\n    client: Any  #: :meta private:\n    model_name: str = \"models/chat-bison-001\"\n    \"\"\"Model name to use.\"\"\"\n    google_api_key: Optional[str] = None\n    temperature: Optional[float] = None\n    \"\"\"Run inference with this temperature. Must by in the closed\n       interval [0.0, 1.0].\"\"\"\n    top_p: Optional[float] = None\n    \"\"\"Decode using nucleus sampling: consider the smallest set of tokens whose\n       probability sum is at least top_p. Must be in the closed interval [0.0, 1.0].\"\"\"\n    top_k: Optional[int] = None\n    \"\"\"Decode using top-k sampling: consider the set of top_k most probable tokens.\n       Must be positive.\"\"\"\n    n: int = 1\n    \"\"\"Number of chat completions to generate for each prompt. Note that the API may\n       not return the full n completions if duplicates are generated.\"\"\"\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate api key, python package exists, temperature, top_p, and top_k.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}454{"id": "a339475cd469-5", "text": "\"\"\"Validate api key, python package exists, temperature, top_p, and top_k.\"\"\"\n        google_api_key = get_from_dict_or_env(\n            values, \"google_api_key\", \"GOOGLE_API_KEY\"\n        )\n        try:\n            import google.generativeai as genai\n            genai.configure(api_key=google_api_key)\n        except ImportError:\n            raise ChatGooglePalmError(\n                \"Could not import google.generativeai python package. \"\n                \"Please install it with `pip install google-generativeai`\"\n            )\n        values[\"client\"] = genai\n        if values[\"temperature\"] is not None and not 0 <= values[\"temperature\"] <= 1:\n            raise ValueError(\"temperature must be in the range [0.0, 1.0]\")\n        if values[\"top_p\"] is not None and not 0 <= values[\"top_p\"] <= 1:\n            raise ValueError(\"top_p must be in the range [0.0, 1.0]\")\n        if values[\"top_k\"] is not None and values[\"top_k\"] <= 0:\n            raise ValueError(\"top_k must be positive\")\n        return values\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        prompt = _messages_to_prompt_dict(messages)\n        response: genai.types.ChatResponse = chat_with_retry(\n            self,\n            model=self.model_name,\n            prompt=prompt,\n            temperature=self.temperature,\n            top_p=self.top_p,\n            top_k=self.top_k,\n            candidate_count=self.n,\n        )\n        return _response_to_result(response, stop)", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}455{"id": "a339475cd469-6", "text": "candidate_count=self.n,\n        )\n        return _response_to_result(response, stop)\n    async def _agenerate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        prompt = _messages_to_prompt_dict(messages)\n        response: genai.types.ChatResponse = await achat_with_retry(\n            self,\n            model=self.model_name,\n            prompt=prompt,\n            temperature=self.temperature,\n            top_p=self.top_p,\n            top_k=self.top_k,\n            candidate_count=self.n,\n        )\n        return _response_to_result(response, stop)\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            \"model_name\": self.model_name,\n            \"temperature\": self.temperature,\n            \"top_p\": self.top_p,\n            \"top_k\": self.top_k,\n            \"n\": self.n,\n        }\n    @property\n    def _llm_type(self) -> str:\n        return \"google-palm-chat\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/google_palm.html"}456{"id": "e5a2bcea1e5c-0", "text": "Source code for langchain.chat_models.promptlayer_openai\n\"\"\"PromptLayer wrapper.\"\"\"\nimport datetime\nfrom typing import Any, List, Mapping, Optional\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForLLMRun,\n    CallbackManagerForLLMRun,\n)\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.schema import BaseMessage, ChatResult\n[docs]class PromptLayerChatOpenAI(ChatOpenAI):\n    \"\"\"Wrapper around OpenAI Chat large language models and PromptLayer.\n    To use, you should have the ``openai`` and ``promptlayer`` python\n    package installed, and the environment variable ``OPENAI_API_KEY``\n    and ``PROMPTLAYER_API_KEY`` set with your openAI API key and\n    promptlayer key respectively.\n    All parameters that can be passed to the OpenAI LLM can also\n    be passed here. The PromptLayerChatOpenAI adds to optional\n    parameters:\n        ``pl_tags``: List of strings to tag the request with.\n        ``return_pl_id``: If True, the PromptLayer request ID will be\n            returned in the ``generation_info`` field of the\n            ``Generation`` object.\n    Example:\n        .. code-block:: python\n            from langchain.chat_models import PromptLayerChatOpenAI\n            openai = PromptLayerChatOpenAI(model_name=\"gpt-3.5-turbo\")\n    \"\"\"\n    pl_tags: Optional[List[str]]\n    return_pl_id: Optional[bool] = False\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> ChatResult:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/promptlayer_openai.html"}457{"id": "e5a2bcea1e5c-1", "text": ") -> ChatResult:\n        \"\"\"Call ChatOpenAI generate and then call PromptLayer API to log the request.\"\"\"\n        from promptlayer.utils import get_api_key, promptlayer_api_request\n        request_start_time = datetime.datetime.now().timestamp()\n        generated_responses = super()._generate(messages, stop, run_manager)\n        request_end_time = datetime.datetime.now().timestamp()\n        message_dicts, params = super()._create_message_dicts(messages, stop)\n        for i, generation in enumerate(generated_responses.generations):\n            response_dict, params = super()._create_message_dicts(\n                [generation.message], stop\n            )\n            pl_request_id = promptlayer_api_request(\n                \"langchain.PromptLayerChatOpenAI\",\n                \"langchain\",\n                message_dicts,\n                params,\n                self.pl_tags,\n                response_dict,\n                request_start_time,\n                request_end_time,\n                get_api_key(),\n                return_pl_id=self.return_pl_id,\n            )\n            if self.return_pl_id:\n                if generation.generation_info is None or not isinstance(\n                    generation.generation_info, dict\n                ):\n                    generation.generation_info = {}\n                generation.generation_info[\"pl_request_id\"] = pl_request_id\n        return generated_responses\n    async def _agenerate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        \"\"\"Call ChatOpenAI agenerate and then call PromptLayer to log.\"\"\"\n        from promptlayer.utils import get_api_key, promptlayer_api_request_async\n        request_start_time = datetime.datetime.now().timestamp()\n        generated_responses = await super()._agenerate(messages, stop, run_manager)", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/promptlayer_openai.html"}458{"id": "e5a2bcea1e5c-2", "text": "generated_responses = await super()._agenerate(messages, stop, run_manager)\n        request_end_time = datetime.datetime.now().timestamp()\n        message_dicts, params = super()._create_message_dicts(messages, stop)\n        for i, generation in enumerate(generated_responses.generations):\n            response_dict, params = super()._create_message_dicts(\n                [generation.message], stop\n            )\n            pl_request_id = await promptlayer_api_request_async(\n                \"langchain.PromptLayerChatOpenAI.async\",\n                \"langchain\",\n                message_dicts,\n                params,\n                self.pl_tags,\n                response_dict,\n                request_start_time,\n                request_end_time,\n                get_api_key(),\n                return_pl_id=self.return_pl_id,\n            )\n            if self.return_pl_id:\n                if generation.generation_info is None or not isinstance(\n                    generation.generation_info, dict\n                ):\n                    generation.generation_info = {}\n                generation.generation_info[\"pl_request_id\"] = pl_request_id\n        return generated_responses\n    @property\n    def _llm_type(self) -> str:\n        return \"promptlayer-openai-chat\"\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        return {\n            **super()._identifying_params,\n            \"pl_tags\": self.pl_tags,\n            \"return_pl_id\": self.return_pl_id,\n        }\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/promptlayer_openai.html"}459{"id": "9f7fe4919dd1-0", "text": "Source code for langchain.chat_models.vertexai\n\"\"\"Wrapper around Google VertexAI chat-based models.\"\"\"\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Optional\nfrom pydantic import root_validator\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForLLMRun,\n    CallbackManagerForLLMRun,\n)\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.llms.vertexai import _VertexAICommon\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    ChatGeneration,\n    ChatResult,\n    HumanMessage,\n    SystemMessage,\n)\nfrom langchain.utilities.vertexai import raise_vertex_import_error\n@dataclass\nclass _MessagePair:\n    \"\"\"InputOutputTextPair represents a pair of input and output texts.\"\"\"\n    question: HumanMessage\n    answer: AIMessage\n@dataclass\nclass _ChatHistory:\n    \"\"\"InputOutputTextPair represents a pair of input and output texts.\"\"\"\n    history: List[_MessagePair] = field(default_factory=list)\n    system_message: Optional[SystemMessage] = None\ndef _parse_chat_history(history: List[BaseMessage]) -> _ChatHistory:\n    \"\"\"Parse a sequence of messages into history.\n    A sequence should be either (SystemMessage, HumanMessage, AIMessage,\n    HumanMessage, AIMessage, ...) or (HumanMessage, AIMessage, HumanMessage,\n    AIMessage, ...).\n    Args:\n        history: The list of messages to re-create the history of the chat.\n    Returns:\n        A parsed chat history.\n    Raises:\n        ValueError: If a sequence of message is odd, or a human message is not followed\n            by a message from AI (e.g., Human, Human, AI or AI, AI, Human).\n    \"\"\"\n    if not history:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/vertexai.html"}460{"id": "9f7fe4919dd1-1", "text": "\"\"\"\n    if not history:\n        return _ChatHistory()\n    first_message = history[0]\n    system_message = first_message if isinstance(first_message, SystemMessage) else None\n    chat_history = _ChatHistory(system_message=system_message)\n    messages_left = history[1:] if system_message else history\n    if len(messages_left) % 2 != 0:\n        raise ValueError(\n            f\"Amount of messages in history should be even, got {len(messages_left)}!\"\n        )\n    for question, answer in zip(messages_left[::2], messages_left[1::2]):\n        if not isinstance(question, HumanMessage) or not isinstance(answer, AIMessage):\n            raise ValueError(\n                \"A human message should follow a bot one, \"\n                f\"got {question.type}, {answer.type}.\"\n            )\n        chat_history.history.append(_MessagePair(question=question, answer=answer))\n    return chat_history\n[docs]class ChatVertexAI(_VertexAICommon, BaseChatModel):\n    \"\"\"Wrapper around Vertex AI large language models.\"\"\"\n    model_name: str = \"chat-bison\"\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that the python package exists in environment.\"\"\"\n        cls._try_init_vertexai(values)\n        try:\n            from vertexai.preview.language_models import ChatModel\n        except ImportError:\n            raise_vertex_import_error()\n        values[\"client\"] = ChatModel.from_pretrained(values[\"model_name\"])\n        return values\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> ChatResult:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/vertexai.html"}461{"id": "9f7fe4919dd1-2", "text": ") -> ChatResult:\n        \"\"\"Generate next turn in the conversation.\n        Args:\n            messages: The history of the conversation as a list of messages.\n            stop: The list of stop words (optional).\n            run_manager: The Callbackmanager for LLM run, it's not used at the moment.\n        Returns:\n            The ChatResult that contains outputs generated by the model.\n        Raises:\n            ValueError: if the last message in the list is not from human.\n        \"\"\"\n        if not messages:\n            raise ValueError(\n                \"You should provide at least one message to start the chat!\"\n            )\n        question = messages[-1]\n        if not isinstance(question, HumanMessage):\n            raise ValueError(\n                f\"Last message in the list should be from human, got {question.type}.\"\n            )\n        history = _parse_chat_history(messages[:-1])\n        context = history.system_message.content if history.system_message else None\n        chat = self.client.start_chat(context=context, **self._default_params)\n        for pair in history.history:\n            chat._history.append((pair.question.content, pair.answer.content))\n        response = chat.send_message(question.content)\n        text = self._enforce_stop_words(response.text, stop)\n        return ChatResult(generations=[ChatGeneration(message=AIMessage(content=text))])\n    async def _agenerate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        raise NotImplementedError(\n            \"\"\"Vertex AI doesn't support async requests at the moment.\"\"\"\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/vertexai.html"}462{"id": "9f7fe4919dd1-3", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/vertexai.html"}463{"id": "83b0d9442ada-0", "text": "Source code for langchain.chat_models.anthropic\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import Extra\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForLLMRun,\n    CallbackManagerForLLMRun,\n)\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.llms.anthropic import _AnthropicCommon\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    ChatGeneration,\n    ChatMessage,\n    ChatResult,\n    HumanMessage,\n    SystemMessage,\n)\n[docs]class ChatAnthropic(BaseChatModel, _AnthropicCommon):\n    r\"\"\"Wrapper around Anthropic's large language model.\n    To use, you should have the ``anthropic`` python package installed, and the\n    environment variable ``ANTHROPIC_API_KEY`` set with your API key, or pass\n    it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            import anthropic\n            from langchain.llms import Anthropic\n            model = ChatAnthropic(model=\"<model_name>\", anthropic_api_key=\"my-api-key\")\n    \"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of chat model.\"\"\"\n        return \"anthropic-chat\"\n    def _convert_one_message_to_text(self, message: BaseMessage) -> str:\n        if isinstance(message, ChatMessage):\n            message_text = f\"\\n\\n{message.role.capitalize()}: {message.content}\"\n        elif isinstance(message, HumanMessage):\n            message_text = f\"{self.HUMAN_PROMPT} {message.content}\"\n        elif isinstance(message, AIMessage):", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/anthropic.html"}464{"id": "83b0d9442ada-1", "text": "elif isinstance(message, AIMessage):\n            message_text = f\"{self.AI_PROMPT} {message.content}\"\n        elif isinstance(message, SystemMessage):\n            message_text = f\"{self.HUMAN_PROMPT} <admin>{message.content}</admin>\"\n        else:\n            raise ValueError(f\"Got unknown type {message}\")\n        return message_text\n    def _convert_messages_to_text(self, messages: List[BaseMessage]) -> str:\n        \"\"\"Format a list of strings into a single string with necessary newlines.\n        Args:\n            messages (List[BaseMessage]): List of BaseMessage to combine.\n        Returns:\n            str: Combined string with necessary newlines.\n        \"\"\"\n        return \"\".join(\n            self._convert_one_message_to_text(message) for message in messages\n        )\n    def _convert_messages_to_prompt(self, messages: List[BaseMessage]) -> str:\n        \"\"\"Format a list of messages into a full prompt for the Anthropic model\n        Args:\n            messages (List[BaseMessage]): List of BaseMessage to combine.\n        Returns:\n            str: Combined string with necessary HUMAN_PROMPT and AI_PROMPT tags.\n        \"\"\"\n        if not self.AI_PROMPT:\n            raise NameError(\"Please ensure the anthropic package is loaded\")\n        if not isinstance(messages[-1], AIMessage):\n            messages.append(AIMessage(content=\"\"))\n        text = self._convert_messages_to_text(messages)\n        return (\n            text.rstrip()\n        )  # trim off the trailing ' ' that might come from the \"Assistant: \"\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> ChatResult:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/anthropic.html"}465{"id": "83b0d9442ada-2", "text": ") -> ChatResult:\n        prompt = self._convert_messages_to_prompt(messages)\n        params: Dict[str, Any] = {\"prompt\": prompt, **self._default_params}\n        if stop:\n            params[\"stop_sequences\"] = stop\n        if self.streaming:\n            completion = \"\"\n            stream_resp = self.client.completion_stream(**params)\n            for data in stream_resp:\n                delta = data[\"completion\"][len(completion) :]\n                completion = data[\"completion\"]\n                if run_manager:\n                    run_manager.on_llm_new_token(\n                        delta,\n                    )\n        else:\n            response = self.client.completion(**params)\n            completion = response[\"completion\"]\n        message = AIMessage(content=completion)\n        return ChatResult(generations=[ChatGeneration(message=message)])\n    async def _agenerate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        prompt = self._convert_messages_to_prompt(messages)\n        params: Dict[str, Any] = {\"prompt\": prompt, **self._default_params}\n        if stop:\n            params[\"stop_sequences\"] = stop\n        if self.streaming:\n            completion = \"\"\n            stream_resp = await self.client.acompletion_stream(**params)\n            async for data in stream_resp:\n                delta = data[\"completion\"][len(completion) :]\n                completion = data[\"completion\"]\n                if run_manager:\n                    await run_manager.on_llm_new_token(\n                        delta,\n                    )\n        else:\n            response = await self.client.acompletion(**params)\n            completion = response[\"completion\"]\n        message = AIMessage(content=completion)", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/anthropic.html"}466{"id": "83b0d9442ada-3", "text": "completion = response[\"completion\"]\n        message = AIMessage(content=completion)\n        return ChatResult(generations=[ChatGeneration(message=message)])\n[docs]    def get_num_tokens(self, text: str) -> int:\n        \"\"\"Calculate number of tokens.\"\"\"\n        if not self.count_tokens:\n            raise NameError(\"Please ensure the anthropic package is loaded\")\n        return self.count_tokens(text)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/anthropic.html"}467{"id": "90176d6fca88-0", "text": "Source code for langchain.chat_models.openai\n\"\"\"OpenAI chat wrapper.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport sys\nfrom typing import (\n    TYPE_CHECKING,\n    Any,\n    Callable,\n    Dict,\n    List,\n    Mapping,\n    Optional,\n    Tuple,\n    Union,\n)\nfrom pydantic import Extra, Field, root_validator\nfrom tenacity import (\n    before_sleep_log,\n    retry,\n    retry_if_exception_type,\n    stop_after_attempt,\n    wait_exponential,\n)\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForLLMRun,\n    CallbackManagerForLLMRun,\n)\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    ChatGeneration,\n    ChatMessage,\n    ChatResult,\n    HumanMessage,\n    SystemMessage,\n)\nfrom langchain.utils import get_from_dict_or_env\nif TYPE_CHECKING:\n    import tiktoken\nlogger = logging.getLogger(__name__)\ndef _import_tiktoken() -> Any:\n    try:\n        import tiktoken\n    except ImportError:\n        raise ValueError(\n            \"Could not import tiktoken python package. \"\n            \"This is needed in order to calculate get_token_ids. \"\n            \"Please install it with `pip install tiktoken`.\"\n        )\n    return tiktoken\ndef _create_retry_decorator(llm: ChatOpenAI) -> Callable[[Any], Any]:\n    import openai\n    min_seconds = 1\n    max_seconds = 60\n    # Wait 2^x * 1 second between each retry starting with\n    # 4 seconds, then up to 10 seconds, then 10 seconds afterwards\n    return retry(", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}468{"id": "90176d6fca88-1", "text": "return retry(\n        reraise=True,\n        stop=stop_after_attempt(llm.max_retries),\n        wait=wait_exponential(multiplier=1, min=min_seconds, max=max_seconds),\n        retry=(\n            retry_if_exception_type(openai.error.Timeout)\n            | retry_if_exception_type(openai.error.APIError)\n            | retry_if_exception_type(openai.error.APIConnectionError)\n            | retry_if_exception_type(openai.error.RateLimitError)\n            | retry_if_exception_type(openai.error.ServiceUnavailableError)\n        ),\n        before_sleep=before_sleep_log(logger, logging.WARNING),\n    )\nasync def acompletion_with_retry(llm: ChatOpenAI, **kwargs: Any) -> Any:\n    \"\"\"Use tenacity to retry the async completion call.\"\"\"\n    retry_decorator = _create_retry_decorator(llm)\n    @retry_decorator\n    async def _completion_with_retry(**kwargs: Any) -> Any:\n        # Use OpenAI's async api https://github.com/openai/openai-python#async-api\n        return await llm.client.acreate(**kwargs)\n    return await _completion_with_retry(**kwargs)\ndef _convert_dict_to_message(_dict: dict) -> BaseMessage:\n    role = _dict[\"role\"]\n    if role == \"user\":\n        return HumanMessage(content=_dict[\"content\"])\n    elif role == \"assistant\":\n        return AIMessage(content=_dict[\"content\"])\n    elif role == \"system\":\n        return SystemMessage(content=_dict[\"content\"])\n    else:\n        return ChatMessage(content=_dict[\"content\"], role=role)\ndef _convert_message_to_dict(message: BaseMessage) -> dict:\n    if isinstance(message, ChatMessage):\n        message_dict = {\"role\": message.role, \"content\": message.content}\n    elif isinstance(message, HumanMessage):", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}469{"id": "90176d6fca88-2", "text": "elif isinstance(message, HumanMessage):\n        message_dict = {\"role\": \"user\", \"content\": message.content}\n    elif isinstance(message, AIMessage):\n        message_dict = {\"role\": \"assistant\", \"content\": message.content}\n    elif isinstance(message, SystemMessage):\n        message_dict = {\"role\": \"system\", \"content\": message.content}\n    else:\n        raise ValueError(f\"Got unknown type {message}\")\n    if \"name\" in message.additional_kwargs:\n        message_dict[\"name\"] = message.additional_kwargs[\"name\"]\n    return message_dict\n[docs]class ChatOpenAI(BaseChatModel):\n    \"\"\"Wrapper around OpenAI Chat large language models.\n    To use, you should have the ``openai`` python package installed, and the\n    environment variable ``OPENAI_API_KEY`` set with your API key.\n    Any parameters that are valid to be passed to the openai.create call can be passed\n    in, even if not explicitly saved on this class.\n    Example:\n        .. code-block:: python\n            from langchain.chat_models import ChatOpenAI\n            openai = ChatOpenAI(model_name=\"gpt-3.5-turbo\")\n    \"\"\"\n    client: Any  #: :meta private:\n    model_name: str = Field(default=\"gpt-3.5-turbo\", alias=\"model\")\n    \"\"\"Model name to use.\"\"\"\n    temperature: float = 0.7\n    \"\"\"What sampling temperature to use.\"\"\"\n    model_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Holds any model parameters valid for `create` call not explicitly specified.\"\"\"\n    openai_api_key: Optional[str] = None\n    \"\"\"Base URL path for API requests, \n    leave blank if not using a proxy or service emulator.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}470{"id": "90176d6fca88-3", "text": "leave blank if not using a proxy or service emulator.\"\"\"\n    openai_api_base: Optional[str] = None\n    openai_organization: Optional[str] = None\n    # to support explicit proxy for OpenAI\n    openai_proxy: Optional[str] = None\n    request_timeout: Optional[Union[float, Tuple[float, float]]] = None\n    \"\"\"Timeout for requests to OpenAI completion API. Default is 600 seconds.\"\"\"\n    max_retries: int = 6\n    \"\"\"Maximum number of retries to make when generating.\"\"\"\n    streaming: bool = False\n    \"\"\"Whether to stream the results or not.\"\"\"\n    n: int = 1\n    \"\"\"Number of chat completions to generate for each prompt.\"\"\"\n    max_tokens: Optional[int] = None\n    \"\"\"Maximum number of tokens to generate.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.ignore\n        allow_population_by_field_name = True\n    @root_validator(pre=True)\n    def build_extra(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Build extra kwargs from additional params that were passed in.\"\"\"\n        all_required_field_names = cls.all_required_field_names()\n        extra = values.get(\"model_kwargs\", {})\n        for field_name in list(values):\n            if field_name in extra:\n                raise ValueError(f\"Found {field_name} supplied twice.\")\n            if field_name not in all_required_field_names:\n                logger.warning(\n                    f\"\"\"WARNING! {field_name} is not default parameter.\n                    {field_name} was transferred to model_kwargs.\n                    Please confirm that {field_name} is what you intended.\"\"\"\n                )\n                extra[field_name] = values.pop(field_name)\n        invalid_model_kwargs = all_required_field_names.intersection(extra.keys())\n        if invalid_model_kwargs:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}471{"id": "90176d6fca88-4", "text": "invalid_model_kwargs = all_required_field_names.intersection(extra.keys())\n        if invalid_model_kwargs:\n            raise ValueError(\n                f\"Parameters {invalid_model_kwargs} should be specified explicitly. \"\n                f\"Instead they were passed in as part of `model_kwargs` parameter.\"\n            )\n        values[\"model_kwargs\"] = extra\n        return values\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        openai_api_key = get_from_dict_or_env(\n            values, \"openai_api_key\", \"OPENAI_API_KEY\"\n        )\n        openai_organization = get_from_dict_or_env(\n            values,\n            \"openai_organization\",\n            \"OPENAI_ORGANIZATION\",\n            default=\"\",\n        )\n        openai_api_base = get_from_dict_or_env(\n            values,\n            \"openai_api_base\",\n            \"OPENAI_API_BASE\",\n            default=\"\",\n        )\n        openai_proxy = get_from_dict_or_env(\n            values,\n            \"openai_proxy\",\n            \"OPENAI_PROXY\",\n            default=\"\",\n        )\n        try:\n            import openai\n        except ImportError:\n            raise ValueError(\n                \"Could not import openai python package. \"\n                \"Please install it with `pip install openai`.\"\n            )\n        openai.api_key = openai_api_key\n        if openai_organization:\n            openai.organization = openai_organization\n        if openai_api_base:\n            openai.api_base = openai_api_base\n        if openai_proxy:\n            openai.proxy = {\"http\": openai_proxy, \"https\": openai_proxy}  # type: ignore[assignment]  # noqa: E501\n        try:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}472{"id": "90176d6fca88-5", "text": "try:\n            values[\"client\"] = openai.ChatCompletion\n        except AttributeError:\n            raise ValueError(\n                \"`openai` has no `ChatCompletion` attribute, this is likely \"\n                \"due to an old version of the openai package. Try upgrading it \"\n                \"with `pip install --upgrade openai`.\"\n            )\n        if values[\"n\"] < 1:\n            raise ValueError(\"n must be at least 1.\")\n        if values[\"n\"] > 1 and values[\"streaming\"]:\n            raise ValueError(\"n must be 1 when streaming.\")\n        return values\n    @property\n    def _default_params(self) -> Dict[str, Any]:\n        \"\"\"Get the default parameters for calling OpenAI API.\"\"\"\n        return {\n            \"model\": self.model_name,\n            \"request_timeout\": self.request_timeout,\n            \"max_tokens\": self.max_tokens,\n            \"stream\": self.streaming,\n            \"n\": self.n,\n            \"temperature\": self.temperature,\n            **self.model_kwargs,\n        }\n    def _create_retry_decorator(self) -> Callable[[Any], Any]:\n        import openai\n        min_seconds = 1\n        max_seconds = 60\n        # Wait 2^x * 1 second between each retry starting with\n        # 4 seconds, then up to 10 seconds, then 10 seconds afterwards\n        return retry(\n            reraise=True,\n            stop=stop_after_attempt(self.max_retries),\n            wait=wait_exponential(multiplier=1, min=min_seconds, max=max_seconds),\n            retry=(\n                retry_if_exception_type(openai.error.Timeout)\n                | retry_if_exception_type(openai.error.APIError)\n                | retry_if_exception_type(openai.error.APIConnectionError)", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}473{"id": "90176d6fca88-6", "text": "| retry_if_exception_type(openai.error.APIConnectionError)\n                | retry_if_exception_type(openai.error.RateLimitError)\n                | retry_if_exception_type(openai.error.ServiceUnavailableError)\n            ),\n            before_sleep=before_sleep_log(logger, logging.WARNING),\n        )\n[docs]    def completion_with_retry(self, **kwargs: Any) -> Any:\n        \"\"\"Use tenacity to retry the completion call.\"\"\"\n        retry_decorator = self._create_retry_decorator()\n        @retry_decorator\n        def _completion_with_retry(**kwargs: Any) -> Any:\n            return self.client.create(**kwargs)\n        return _completion_with_retry(**kwargs)\n    def _combine_llm_outputs(self, llm_outputs: List[Optional[dict]]) -> dict:\n        overall_token_usage: dict = {}\n        for output in llm_outputs:\n            if output is None:\n                # Happens in streaming\n                continue\n            token_usage = output[\"token_usage\"]\n            for k, v in token_usage.items():\n                if k in overall_token_usage:\n                    overall_token_usage[k] += v\n                else:\n                    overall_token_usage[k] = v\n        return {\"token_usage\": overall_token_usage, \"model_name\": self.model_name}\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        message_dicts, params = self._create_message_dicts(messages, stop)\n        if self.streaming:\n            inner_completion = \"\"\n            role = \"assistant\"\n            params[\"stream\"] = True\n            for stream_resp in self.completion_with_retry(\n                messages=message_dicts, **params\n            ):", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}474{"id": "90176d6fca88-7", "text": "messages=message_dicts, **params\n            ):\n                role = stream_resp[\"choices\"][0][\"delta\"].get(\"role\", role)\n                token = stream_resp[\"choices\"][0][\"delta\"].get(\"content\", \"\")\n                inner_completion += token\n                if run_manager:\n                    run_manager.on_llm_new_token(token)\n            message = _convert_dict_to_message(\n                {\"content\": inner_completion, \"role\": role}\n            )\n            return ChatResult(generations=[ChatGeneration(message=message)])\n        response = self.completion_with_retry(messages=message_dicts, **params)\n        return self._create_chat_result(response)\n    def _create_message_dicts(\n        self, messages: List[BaseMessage], stop: Optional[List[str]]\n    ) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:\n        params: Dict[str, Any] = {**{\"model\": self.model_name}, **self._default_params}\n        if stop is not None:\n            if \"stop\" in params:\n                raise ValueError(\"`stop` found in both the input and default params.\")\n            params[\"stop\"] = stop\n        message_dicts = [_convert_message_to_dict(m) for m in messages]\n        return message_dicts, params\n    def _create_chat_result(self, response: Mapping[str, Any]) -> ChatResult:\n        generations = []\n        for res in response[\"choices\"]:\n            message = _convert_dict_to_message(res[\"message\"])\n            gen = ChatGeneration(message=message)\n            generations.append(gen)\n        llm_output = {\"token_usage\": response[\"usage\"], \"model_name\": self.model_name}\n        return ChatResult(generations=generations, llm_output=llm_output)\n    async def _agenerate(\n        self,\n        messages: List[BaseMessage],", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}475{"id": "90176d6fca88-8", "text": "async def _agenerate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,\n    ) -> ChatResult:\n        message_dicts, params = self._create_message_dicts(messages, stop)\n        if self.streaming:\n            inner_completion = \"\"\n            role = \"assistant\"\n            params[\"stream\"] = True\n            async for stream_resp in await acompletion_with_retry(\n                self, messages=message_dicts, **params\n            ):\n                role = stream_resp[\"choices\"][0][\"delta\"].get(\"role\", role)\n                token = stream_resp[\"choices\"][0][\"delta\"].get(\"content\", \"\")\n                inner_completion += token\n                if run_manager:\n                    await run_manager.on_llm_new_token(token)\n            message = _convert_dict_to_message(\n                {\"content\": inner_completion, \"role\": role}\n            )\n            return ChatResult(generations=[ChatGeneration(message=message)])\n        else:\n            response = await acompletion_with_retry(\n                self, messages=message_dicts, **params\n            )\n            return self._create_chat_result(response)\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {**{\"model_name\": self.model_name}, **self._default_params}\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of chat model.\"\"\"\n        return \"openai-chat\"\n    def _get_encoding_model(self) -> Tuple[str, tiktoken.Encoding]:\n        tiktoken_ = _import_tiktoken()\n        model = self.model_name\n        if model == \"gpt-3.5-turbo\":", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}476{"id": "90176d6fca88-9", "text": "if model == \"gpt-3.5-turbo\":\n            # gpt-3.5-turbo may change over time.\n            # Returning num tokens assuming gpt-3.5-turbo-0301.\n            model = \"gpt-3.5-turbo-0301\"\n        elif model == \"gpt-4\":\n            # gpt-4 may change over time.\n            # Returning num tokens assuming gpt-4-0314.\n            model = \"gpt-4-0314\"\n        # Returns the number of tokens used by a list of messages.\n        try:\n            encoding = tiktoken_.encoding_for_model(model)\n        except KeyError:\n            logger.warning(\"Warning: model not found. Using cl100k_base encoding.\")\n            model = \"cl100k_base\"\n            encoding = tiktoken_.get_encoding(model)\n        return model, encoding\n[docs]    def get_token_ids(self, text: str) -> List[int]:\n        \"\"\"Get the tokens present in the text with tiktoken package.\"\"\"\n        # tiktoken NOT supported for Python 3.7 or below\n        if sys.version_info[1] <= 7:\n            return super().get_token_ids(text)\n        _, encoding_model = self._get_encoding_model()\n        return encoding_model.encode(text)\n[docs]    def get_num_tokens_from_messages(self, messages: List[BaseMessage]) -> int:\n        \"\"\"Calculate num tokens for gpt-3.5-turbo and gpt-4 with tiktoken package.\n        Official documentation: https://github.com/openai/openai-cookbook/blob/\n        main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb\"\"\"\n        if sys.version_info[1] <= 7:", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}477{"id": "90176d6fca88-10", "text": "if sys.version_info[1] <= 7:\n            return super().get_num_tokens_from_messages(messages)\n        model, encoding = self._get_encoding_model()\n        if model == \"gpt-3.5-turbo-0301\":\n            # every message follows <im_start>{role/name}\\n{content}<im_end>\\n\n            tokens_per_message = 4\n            # if there's a name, the role is omitted\n            tokens_per_name = -1\n        elif model == \"gpt-4-0314\":\n            tokens_per_message = 3\n            tokens_per_name = 1\n        else:\n            raise NotImplementedError(\n                f\"get_num_tokens_from_messages() is not presently implemented \"\n                f\"for model {model}.\"\n                \"See https://github.com/openai/openai-python/blob/main/chatml.md for \"\n                \"information on how messages are converted to tokens.\"\n            )\n        num_tokens = 0\n        messages_dict = [_convert_message_to_dict(m) for m in messages]\n        for message in messages_dict:\n            num_tokens += tokens_per_message\n            for key, value in message.items():\n                num_tokens += len(encoding.encode(value))\n                if key == \"name\":\n                    num_tokens += tokens_per_name\n        # every reply is primed with <im_start>assistant\n        num_tokens += 3\n        return num_tokens\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/chat_models/openai.html"}478{"id": "08640ac78ea8-0", "text": "Source code for langchain.agents.initialize\n\"\"\"Load agent.\"\"\"\nfrom typing import Any, Optional, Sequence\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.loading import AGENT_TO_CLASS, load_agent\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.tools.base import BaseTool\n[docs]def initialize_agent(\n    tools: Sequence[BaseTool],\n    llm: BaseLanguageModel,\n    agent: Optional[AgentType] = None,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    agent_path: Optional[str] = None,\n    agent_kwargs: Optional[dict] = None,\n    **kwargs: Any,\n) -> AgentExecutor:\n    \"\"\"Load an agent executor given tools and LLM.\n    Args:\n        tools: List of tools this agent has access to.\n        llm: Language model to use as the agent.\n        agent: Agent type to use. If None and agent_path is also None, will default to\n            AgentType.ZERO_SHOT_REACT_DESCRIPTION.\n        callback_manager: CallbackManager to use. Global callback manager is used if\n            not provided. Defaults to None.\n        agent_path: Path to serialized agent to use.\n        agent_kwargs: Additional key word arguments to pass to the underlying agent\n        **kwargs: Additional key word arguments passed to the agent executor\n    Returns:\n        An agent executor\n    \"\"\"\n    if agent is None and agent_path is None:\n        agent = AgentType.ZERO_SHOT_REACT_DESCRIPTION\n    if agent is not None and agent_path is not None:\n        raise ValueError(\n            \"Both `agent` and `agent_path` are specified, \"\n            \"but at most only one should be.\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/initialize.html"}479{"id": "08640ac78ea8-1", "text": "\"but at most only one should be.\"\n        )\n    if agent is not None:\n        if agent not in AGENT_TO_CLASS:\n            raise ValueError(\n                f\"Got unknown agent type: {agent}. \"\n                f\"Valid types are: {AGENT_TO_CLASS.keys()}.\"\n            )\n        agent_cls = AGENT_TO_CLASS[agent]\n        agent_kwargs = agent_kwargs or {}\n        agent_obj = agent_cls.from_llm_and_tools(\n            llm, tools, callback_manager=callback_manager, **agent_kwargs\n        )\n    elif agent_path is not None:\n        agent_obj = load_agent(\n            agent_path, llm=llm, tools=tools, callback_manager=callback_manager\n        )\n    else:\n        raise ValueError(\n            \"Somehow both `agent` and `agent_path` are None, \"\n            \"this should never happen.\"\n        )\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent_obj,\n        tools=tools,\n        callback_manager=callback_manager,\n        **kwargs,\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/initialize.html"}480{"id": "df388dc9008a-0", "text": "Source code for langchain.agents.agent\n\"\"\"Chain that takes in an input and produces an action and action input.\"\"\"\nfrom __future__ import annotations\nimport asyncio\nimport json\nimport logging\nimport time\nfrom abc import abstractmethod\nfrom pathlib import Path\nfrom typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union\nimport yaml\nfrom pydantic import BaseModel, root_validator\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.tools import InvalidTool\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.callbacks.manager import (\n    AsyncCallbackManagerForChainRun,\n    AsyncCallbackManagerForToolRun,\n    CallbackManagerForChainRun,\n    CallbackManagerForToolRun,\n    Callbacks,\n)\nfrom langchain.chains.base import Chain\nfrom langchain.chains.llm import LLMChain\nfrom langchain.input import get_color_mapping\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.prompts.few_shot import FewShotPromptTemplate\nfrom langchain.prompts.prompt import PromptTemplate\nfrom langchain.schema import (\n    AgentAction,\n    AgentFinish,\n    BaseMessage,\n    BaseOutputParser,\n    OutputParserException,\n)\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.asyncio import asyncio_timeout\nlogger = logging.getLogger(__name__)\n[docs]class BaseSingleActionAgent(BaseModel):\n    \"\"\"Base Agent class.\"\"\"\n    @property\n    def return_values(self) -> List[str]:\n        \"\"\"Return values of the agent.\"\"\"\n        return [\"output\"]\n[docs]    def get_allowed_tools(self) -> Optional[List[str]]:\n        return None\n[docs]    @abstractmethod\n    def plan(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}481{"id": "df388dc9008a-1", "text": "return None\n[docs]    @abstractmethod\n    def plan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n[docs]    @abstractmethod\n    async def aplan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n    @property\n    @abstractmethod\n    def input_keys(self) -> List[str]:\n        \"\"\"Return the input keys.\n        :meta private:\n        \"\"\"\n[docs]    def return_stopped_response(\n        self,\n        early_stopping_method: str,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        **kwargs: Any,\n    ) -> AgentFinish:\n        \"\"\"Return response when agent has been stopped due to max iterations.\"\"\"\n        if early_stopping_method == \"force\":\n            # `force` just returns a constant string\n            return AgentFinish(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}482{"id": "df388dc9008a-2", "text": "# `force` just returns a constant string\n            return AgentFinish(\n                {\"output\": \"Agent stopped due to iteration limit or time limit.\"}, \"\"\n            )\n        else:\n            raise ValueError(\n                f\"Got unsupported early_stopping_method `{early_stopping_method}`\"\n            )\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        **kwargs: Any,\n    ) -> BaseSingleActionAgent:\n        raise NotImplementedError\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        raise NotImplementedError\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return dictionary representation of agent.\"\"\"\n        _dict = super().dict()\n        _type = self._agent_type\n        if isinstance(_type, AgentType):\n            _dict[\"_type\"] = str(_type.value)\n        else:\n            _dict[\"_type\"] = _type\n        return _dict\n[docs]    def save(self, file_path: Union[Path, str]) -> None:\n        \"\"\"Save the agent.\n        Args:\n            file_path: Path to file to save the agent to.\n        Example:\n        .. code-block:: python\n            # If working with agent executor\n            agent.agent.save(file_path=\"path/agent.yaml\")\n        \"\"\"\n        # Convert file to Path object.\n        if isinstance(file_path, str):\n            save_path = Path(file_path)\n        else:\n            save_path = file_path\n        directory_path = save_path.parent\n        directory_path.mkdir(parents=True, exist_ok=True)\n        # Fetch dictionary to save", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}483{"id": "df388dc9008a-3", "text": "directory_path.mkdir(parents=True, exist_ok=True)\n        # Fetch dictionary to save\n        agent_dict = self.dict()\n        if save_path.suffix == \".json\":\n            with open(file_path, \"w\") as f:\n                json.dump(agent_dict, f, indent=4)\n        elif save_path.suffix == \".yaml\":\n            with open(file_path, \"w\") as f:\n                yaml.dump(agent_dict, f, default_flow_style=False)\n        else:\n            raise ValueError(f\"{save_path} must be json or yaml\")\n[docs]    def tool_run_logging_kwargs(self) -> Dict:\n        return {}\n[docs]class BaseMultiActionAgent(BaseModel):\n    \"\"\"Base Agent class.\"\"\"\n    @property\n    def return_values(self) -> List[str]:\n        \"\"\"Return values of the agent.\"\"\"\n        return [\"output\"]\n[docs]    def get_allowed_tools(self) -> Optional[List[str]]:\n        return None\n[docs]    @abstractmethod\n    def plan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[List[AgentAction], AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Actions specifying what tool to use.\n        \"\"\"\n[docs]    @abstractmethod\n    async def aplan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[List[AgentAction], AgentFinish]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}484{"id": "df388dc9008a-4", "text": "**kwargs: Any,\n    ) -> Union[List[AgentAction], AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Actions specifying what tool to use.\n        \"\"\"\n    @property\n    @abstractmethod\n    def input_keys(self) -> List[str]:\n        \"\"\"Return the input keys.\n        :meta private:\n        \"\"\"\n[docs]    def return_stopped_response(\n        self,\n        early_stopping_method: str,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        **kwargs: Any,\n    ) -> AgentFinish:\n        \"\"\"Return response when agent has been stopped due to max iterations.\"\"\"\n        if early_stopping_method == \"force\":\n            # `force` just returns a constant string\n            return AgentFinish({\"output\": \"Agent stopped due to max iterations.\"}, \"\")\n        else:\n            raise ValueError(\n                f\"Got unsupported early_stopping_method `{early_stopping_method}`\"\n            )\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        raise NotImplementedError\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return dictionary representation of agent.\"\"\"\n        _dict = super().dict()\n        _dict[\"_type\"] = str(self._agent_type)\n        return _dict\n[docs]    def save(self, file_path: Union[Path, str]) -> None:\n        \"\"\"Save the agent.\n        Args:\n            file_path: Path to file to save the agent to.\n        Example:\n        .. code-block:: python", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}485{"id": "df388dc9008a-5", "text": "Example:\n        .. code-block:: python\n            # If working with agent executor\n            agent.agent.save(file_path=\"path/agent.yaml\")\n        \"\"\"\n        # Convert file to Path object.\n        if isinstance(file_path, str):\n            save_path = Path(file_path)\n        else:\n            save_path = file_path\n        directory_path = save_path.parent\n        directory_path.mkdir(parents=True, exist_ok=True)\n        # Fetch dictionary to save\n        agent_dict = self.dict()\n        if save_path.suffix == \".json\":\n            with open(file_path, \"w\") as f:\n                json.dump(agent_dict, f, indent=4)\n        elif save_path.suffix == \".yaml\":\n            with open(file_path, \"w\") as f:\n                yaml.dump(agent_dict, f, default_flow_style=False)\n        else:\n            raise ValueError(f\"{save_path} must be json or yaml\")\n[docs]    def tool_run_logging_kwargs(self) -> Dict:\n        return {}\n[docs]class AgentOutputParser(BaseOutputParser):\n[docs]    @abstractmethod\n    def parse(self, text: str) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Parse text into agent action/finish.\"\"\"\n[docs]class LLMSingleActionAgent(BaseSingleActionAgent):\n    llm_chain: LLMChain\n    output_parser: AgentOutputParser\n    stop: List[str]\n    @property\n    def input_keys(self) -> List[str]:\n        return list(set(self.llm_chain.input_keys) - {\"intermediate_steps\"})\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return dictionary representation of agent.\"\"\"\n        _dict = super().dict()\n        del _dict[\"output_parser\"]\n        return _dict\n[docs]    def plan(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}486{"id": "df388dc9008a-6", "text": "return _dict\n[docs]    def plan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n        output = self.llm_chain.run(\n            intermediate_steps=intermediate_steps,\n            stop=self.stop,\n            callbacks=callbacks,\n            **kwargs,\n        )\n        return self.output_parser.parse(output)\n[docs]    async def aplan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n        output = await self.llm_chain.arun(\n            intermediate_steps=intermediate_steps,\n            stop=self.stop,\n            callbacks=callbacks,\n            **kwargs,\n        )\n        return self.output_parser.parse(output)\n[docs]    def tool_run_logging_kwargs(self) -> Dict:\n        return {\n            \"llm_prefix\": \"\",\n            \"observation_prefix\": \"\" if len(self.stop) == 0 else self.stop[0],\n        }", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}487{"id": "df388dc9008a-7", "text": "}\n[docs]class Agent(BaseSingleActionAgent):\n    \"\"\"Class responsible for calling the language model and deciding the action.\n    This is driven by an LLMChain. The prompt in the LLMChain MUST include\n    a variable called \"agent_scratchpad\" where the agent can put its\n    intermediary work.\n    \"\"\"\n    llm_chain: LLMChain\n    output_parser: AgentOutputParser\n    allowed_tools: Optional[List[str]] = None\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return dictionary representation of agent.\"\"\"\n        _dict = super().dict()\n        del _dict[\"output_parser\"]\n        return _dict\n[docs]    def get_allowed_tools(self) -> Optional[List[str]]:\n        return self.allowed_tools\n    @property\n    def return_values(self) -> List[str]:\n        return [\"output\"]\n    def _fix_text(self, text: str) -> str:\n        \"\"\"Fix the text.\"\"\"\n        raise ValueError(\"fix_text not implemented for this agent.\")\n    @property\n    def _stop(self) -> List[str]:\n        return [\n            f\"\\n{self.observation_prefix.rstrip()}\",\n            f\"\\n\\t{self.observation_prefix.rstrip()}\",\n        ]\n    def _construct_scratchpad(\n        self, intermediate_steps: List[Tuple[AgentAction, str]]\n    ) -> Union[str, List[BaseMessage]]:\n        \"\"\"Construct the scratchpad that lets the agent continue its thought process.\"\"\"\n        thoughts = \"\"\n        for action, observation in intermediate_steps:\n            thoughts += action.log\n            thoughts += f\"\\n{self.observation_prefix}{observation}\\n{self.llm_prefix}\"\n        return thoughts\n[docs]    def plan(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}488{"id": "df388dc9008a-8", "text": "return thoughts\n[docs]    def plan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n        full_inputs = self.get_full_inputs(intermediate_steps, **kwargs)\n        full_output = self.llm_chain.predict(callbacks=callbacks, **full_inputs)\n        return self.output_parser.parse(full_output)\n[docs]    async def aplan(\n        self,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        callbacks: Callbacks = None,\n        **kwargs: Any,\n    ) -> Union[AgentAction, AgentFinish]:\n        \"\"\"Given input, decided what to do.\n        Args:\n            intermediate_steps: Steps the LLM has taken to date,\n                along with observations\n            callbacks: Callbacks to run.\n            **kwargs: User inputs.\n        Returns:\n            Action specifying what tool to use.\n        \"\"\"\n        full_inputs = self.get_full_inputs(intermediate_steps, **kwargs)\n        full_output = await self.llm_chain.apredict(callbacks=callbacks, **full_inputs)\n        return self.output_parser.parse(full_output)\n[docs]    def get_full_inputs(\n        self, intermediate_steps: List[Tuple[AgentAction, str]], **kwargs: Any\n    ) -> Dict[str, Any]:\n        \"\"\"Create the full inputs for the LLMChain from intermediate steps.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}489{"id": "df388dc9008a-9", "text": "\"\"\"Create the full inputs for the LLMChain from intermediate steps.\"\"\"\n        thoughts = self._construct_scratchpad(intermediate_steps)\n        new_inputs = {\"agent_scratchpad\": thoughts, \"stop\": self._stop}\n        full_inputs = {**kwargs, **new_inputs}\n        return full_inputs\n    @property\n    def input_keys(self) -> List[str]:\n        \"\"\"Return the input keys.\n        :meta private:\n        \"\"\"\n        return list(set(self.llm_chain.input_keys) - {\"agent_scratchpad\"})\n    @root_validator()\n    def validate_prompt(cls, values: Dict) -> Dict:\n        \"\"\"Validate that prompt matches format.\"\"\"\n        prompt = values[\"llm_chain\"].prompt\n        if \"agent_scratchpad\" not in prompt.input_variables:\n            logger.warning(\n                \"`agent_scratchpad` should be a variable in prompt.input_variables.\"\n                \" Did not find it, so adding it at the end.\"\n            )\n            prompt.input_variables.append(\"agent_scratchpad\")\n            if isinstance(prompt, PromptTemplate):\n                prompt.template += \"\\n{agent_scratchpad}\"\n            elif isinstance(prompt, FewShotPromptTemplate):\n                prompt.suffix += \"\\n{agent_scratchpad}\"\n            else:\n                raise ValueError(f\"Got unexpected prompt type {type(prompt)}\")\n        return values\n    @property\n    @abstractmethod\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n    @property\n    @abstractmethod\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the LLM call with.\"\"\"\n[docs]    @classmethod\n    @abstractmethod\n    def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:\n        \"\"\"Create a prompt for this class.\"\"\"\n    @classmethod", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}490{"id": "df388dc9008a-10", "text": "\"\"\"Create a prompt for this class.\"\"\"\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        \"\"\"Validate that appropriate tools are passed in.\"\"\"\n        pass\n    @classmethod\n    @abstractmethod\n    def _get_default_output_parser(cls, **kwargs: Any) -> AgentOutputParser:\n        \"\"\"Get default output parser for this class.\"\"\"\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        output_parser: Optional[AgentOutputParser] = None,\n        **kwargs: Any,\n    ) -> Agent:\n        \"\"\"Construct an agent from an LLM and tools.\"\"\"\n        cls._validate_tools(tools)\n        llm_chain = LLMChain(\n            llm=llm,\n            prompt=cls.create_prompt(tools),\n            callback_manager=callback_manager,\n        )\n        tool_names = [tool.name for tool in tools]\n        _output_parser = output_parser or cls._get_default_output_parser()\n        return cls(\n            llm_chain=llm_chain,\n            allowed_tools=tool_names,\n            output_parser=_output_parser,\n            **kwargs,\n        )\n[docs]    def return_stopped_response(\n        self,\n        early_stopping_method: str,\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        **kwargs: Any,\n    ) -> AgentFinish:\n        \"\"\"Return response when agent has been stopped due to max iterations.\"\"\"\n        if early_stopping_method == \"force\":\n            # `force` just returns a constant string\n            return AgentFinish(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}491{"id": "df388dc9008a-11", "text": "# `force` just returns a constant string\n            return AgentFinish(\n                {\"output\": \"Agent stopped due to iteration limit or time limit.\"}, \"\"\n            )\n        elif early_stopping_method == \"generate\":\n            # Generate does one final forward pass\n            thoughts = \"\"\n            for action, observation in intermediate_steps:\n                thoughts += action.log\n                thoughts += (\n                    f\"\\n{self.observation_prefix}{observation}\\n{self.llm_prefix}\"\n                )\n            # Adding to the previous steps, we now tell the LLM to make a final pred\n            thoughts += (\n                \"\\n\\nI now need to return a final answer based on the previous steps:\"\n            )\n            new_inputs = {\"agent_scratchpad\": thoughts, \"stop\": self._stop}\n            full_inputs = {**kwargs, **new_inputs}\n            full_output = self.llm_chain.predict(**full_inputs)\n            # We try to extract a final answer\n            parsed_output = self.output_parser.parse(full_output)\n            if isinstance(parsed_output, AgentFinish):\n                # If we can extract, we send the correct stuff\n                return parsed_output\n            else:\n                # If we can extract, but the tool is not the final tool,\n                # we just return the full output\n                return AgentFinish({\"output\": full_output}, full_output)\n        else:\n            raise ValueError(\n                \"early_stopping_method should be one of `force` or `generate`, \"\n                f\"got {early_stopping_method}\"\n            )\n[docs]    def tool_run_logging_kwargs(self) -> Dict:\n        return {\n            \"llm_prefix\": self.llm_prefix,\n            \"observation_prefix\": self.observation_prefix,\n        }\nclass ExceptionTool(BaseTool):\n    name = \"_Exception\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}492{"id": "df388dc9008a-12", "text": "}\nclass ExceptionTool(BaseTool):\n    name = \"_Exception\"\n    description = \"Exception tool\"\n    def _run(\n        self,\n        query: str,\n        run_manager: Optional[CallbackManagerForToolRun] = None,\n    ) -> str:\n        return query\n    async def _arun(\n        self,\n        query: str,\n        run_manager: Optional[AsyncCallbackManagerForToolRun] = None,\n    ) -> str:\n        return query\n[docs]class AgentExecutor(Chain):\n    \"\"\"Consists of an agent using tools.\"\"\"\n    agent: Union[BaseSingleActionAgent, BaseMultiActionAgent]\n    tools: Sequence[BaseTool]\n    return_intermediate_steps: bool = False\n    max_iterations: Optional[int] = 15\n    max_execution_time: Optional[float] = None\n    early_stopping_method: str = \"force\"\n    handle_parsing_errors: Union[\n        bool, str, Callable[[OutputParserException], str]\n    ] = False\n[docs]    @classmethod\n    def from_agent_and_tools(\n        cls,\n        agent: Union[BaseSingleActionAgent, BaseMultiActionAgent],\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        **kwargs: Any,\n    ) -> AgentExecutor:\n        \"\"\"Create from agent and tools.\"\"\"\n        return cls(\n            agent=agent, tools=tools, callback_manager=callback_manager, **kwargs\n        )\n    @root_validator()\n    def validate_tools(cls, values: Dict) -> Dict:\n        \"\"\"Validate that tools are compatible with agent.\"\"\"\n        agent = values[\"agent\"]\n        tools = values[\"tools\"]\n        allowed_tools = agent.get_allowed_tools()", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}493{"id": "df388dc9008a-13", "text": "tools = values[\"tools\"]\n        allowed_tools = agent.get_allowed_tools()\n        if allowed_tools is not None:\n            if set(allowed_tools) != set([tool.name for tool in tools]):\n                raise ValueError(\n                    f\"Allowed tools ({allowed_tools}) different than \"\n                    f\"provided tools ({[tool.name for tool in tools]})\"\n                )\n        return values\n    @root_validator()\n    def validate_return_direct_tool(cls, values: Dict) -> Dict:\n        \"\"\"Validate that tools are compatible with agent.\"\"\"\n        agent = values[\"agent\"]\n        tools = values[\"tools\"]\n        if isinstance(agent, BaseMultiActionAgent):\n            for tool in tools:\n                if tool.return_direct:\n                    raise ValueError(\n                        \"Tools that have `return_direct=True` are not allowed \"\n                        \"in multi-action agents\"\n                    )\n        return values\n[docs]    def save(self, file_path: Union[Path, str]) -> None:\n        \"\"\"Raise error - saving not supported for Agent Executors.\"\"\"\n        raise ValueError(\n            \"Saving not supported for agent executors. \"\n            \"If you are trying to save the agent, please use the \"\n            \"`.save_agent(...)`\"\n        )\n[docs]    def save_agent(self, file_path: Union[Path, str]) -> None:\n        \"\"\"Save the underlying agent.\"\"\"\n        return self.agent.save(file_path)\n    @property\n    def input_keys(self) -> List[str]:\n        \"\"\"Return the input keys.\n        :meta private:\n        \"\"\"\n        return self.agent.input_keys\n    @property\n    def output_keys(self) -> List[str]:\n        \"\"\"Return the singular output key.\n        :meta private:\n        \"\"\"\n        if self.return_intermediate_steps:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}494{"id": "df388dc9008a-14", "text": ":meta private:\n        \"\"\"\n        if self.return_intermediate_steps:\n            return self.agent.return_values + [\"intermediate_steps\"]\n        else:\n            return self.agent.return_values\n[docs]    def lookup_tool(self, name: str) -> BaseTool:\n        \"\"\"Lookup tool by name.\"\"\"\n        return {tool.name: tool for tool in self.tools}[name]\n    def _should_continue(self, iterations: int, time_elapsed: float) -> bool:\n        if self.max_iterations is not None and iterations >= self.max_iterations:\n            return False\n        if (\n            self.max_execution_time is not None\n            and time_elapsed >= self.max_execution_time\n        ):\n            return False\n        return True\n    def _return(\n        self,\n        output: AgentFinish,\n        intermediate_steps: list,\n        run_manager: Optional[CallbackManagerForChainRun] = None,\n    ) -> Dict[str, Any]:\n        if run_manager:\n            run_manager.on_agent_finish(output, color=\"green\", verbose=self.verbose)\n        final_output = output.return_values\n        if self.return_intermediate_steps:\n            final_output[\"intermediate_steps\"] = intermediate_steps\n        return final_output\n    async def _areturn(\n        self,\n        output: AgentFinish,\n        intermediate_steps: list,\n        run_manager: Optional[AsyncCallbackManagerForChainRun] = None,\n    ) -> Dict[str, Any]:\n        if run_manager:\n            await run_manager.on_agent_finish(\n                output, color=\"green\", verbose=self.verbose\n            )\n        final_output = output.return_values\n        if self.return_intermediate_steps:\n            final_output[\"intermediate_steps\"] = intermediate_steps\n        return final_output\n    def _take_next_step(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}495{"id": "df388dc9008a-15", "text": "return final_output\n    def _take_next_step(\n        self,\n        name_to_tool_map: Dict[str, BaseTool],\n        color_mapping: Dict[str, str],\n        inputs: Dict[str, str],\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        run_manager: Optional[CallbackManagerForChainRun] = None,\n    ) -> Union[AgentFinish, List[Tuple[AgentAction, str]]]:\n        \"\"\"Take a single step in the thought-action-observation loop.\n        Override this to take control of how the agent makes and acts on choices.\n        \"\"\"\n        try:\n            # Call the LLM to see what to do.\n            output = self.agent.plan(\n                intermediate_steps,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **inputs,\n            )\n        except OutputParserException as e:\n            if isinstance(self.handle_parsing_errors, bool):\n                raise_error = not self.handle_parsing_errors\n            else:\n                raise_error = False\n            if raise_error:\n                raise e\n            text = str(e)\n            if isinstance(self.handle_parsing_errors, bool):\n                if e.send_to_llm:\n                    observation = str(e.observation)\n                    text = str(e.llm_output)\n                else:\n                    observation = \"Invalid or incomplete response\"\n            elif isinstance(self.handle_parsing_errors, str):\n                observation = self.handle_parsing_errors\n            elif callable(self.handle_parsing_errors):\n                observation = self.handle_parsing_errors(e)\n            else:\n                raise ValueError(\"Got unexpected type of `handle_parsing_errors`\")\n            output = AgentAction(\"_Exception\", observation, text)\n            if run_manager:\n                run_manager.on_agent_action(output, color=\"green\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}496{"id": "df388dc9008a-16", "text": "if run_manager:\n                run_manager.on_agent_action(output, color=\"green\")\n            tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n            observation = ExceptionTool().run(\n                output.tool_input,\n                verbose=self.verbose,\n                color=None,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **tool_run_kwargs,\n            )\n            return [(output, observation)]\n        # If the tool chosen is the finishing tool, then we end and return.\n        if isinstance(output, AgentFinish):\n            return output\n        actions: List[AgentAction]\n        if isinstance(output, AgentAction):\n            actions = [output]\n        else:\n            actions = output\n        result = []\n        for agent_action in actions:\n            if run_manager:\n                run_manager.on_agent_action(agent_action, color=\"green\")\n            # Otherwise we lookup the tool\n            if agent_action.tool in name_to_tool_map:\n                tool = name_to_tool_map[agent_action.tool]\n                return_direct = tool.return_direct\n                color = color_mapping[agent_action.tool]\n                tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n                if return_direct:\n                    tool_run_kwargs[\"llm_prefix\"] = \"\"\n                # We then call the tool on the tool input to get an observation\n                observation = tool.run(\n                    agent_action.tool_input,\n                    verbose=self.verbose,\n                    color=color,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **tool_run_kwargs,\n                )\n            else:\n                tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n                observation = InvalidTool().run(\n                    agent_action.tool,\n                    verbose=self.verbose,\n                    color=None,\n                    callbacks=run_manager.get_child() if run_manager else None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}497{"id": "df388dc9008a-17", "text": "color=None,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **tool_run_kwargs,\n                )\n            result.append((agent_action, observation))\n        return result\n    async def _atake_next_step(\n        self,\n        name_to_tool_map: Dict[str, BaseTool],\n        color_mapping: Dict[str, str],\n        inputs: Dict[str, str],\n        intermediate_steps: List[Tuple[AgentAction, str]],\n        run_manager: Optional[AsyncCallbackManagerForChainRun] = None,\n    ) -> Union[AgentFinish, List[Tuple[AgentAction, str]]]:\n        \"\"\"Take a single step in the thought-action-observation loop.\n        Override this to take control of how the agent makes and acts on choices.\n        \"\"\"\n        try:\n            # Call the LLM to see what to do.\n            output = await self.agent.aplan(\n                intermediate_steps,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **inputs,\n            )\n        except OutputParserException as e:\n            if isinstance(self.handle_parsing_errors, bool):\n                raise_error = not self.handle_parsing_errors\n            else:\n                raise_error = False\n            if raise_error:\n                raise e\n            text = str(e)\n            if isinstance(self.handle_parsing_errors, bool):\n                observation = \"Invalid or incomplete response\"\n            elif isinstance(self.handle_parsing_errors, str):\n                observation = self.handle_parsing_errors\n            elif callable(self.handle_parsing_errors):\n                observation = self.handle_parsing_errors(e)\n            else:\n                raise ValueError(\"Got unexpected type of `handle_parsing_errors`\")\n            output = AgentAction(\"_Exception\", observation, text)\n            tool_run_kwargs = self.agent.tool_run_logging_kwargs()", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}498{"id": "df388dc9008a-18", "text": "tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n            observation = await ExceptionTool().arun(\n                output.tool_input,\n                verbose=self.verbose,\n                color=None,\n                callbacks=run_manager.get_child() if run_manager else None,\n                **tool_run_kwargs,\n            )\n            return [(output, observation)]\n        # If the tool chosen is the finishing tool, then we end and return.\n        if isinstance(output, AgentFinish):\n            return output\n        actions: List[AgentAction]\n        if isinstance(output, AgentAction):\n            actions = [output]\n        else:\n            actions = output\n        async def _aperform_agent_action(\n            agent_action: AgentAction,\n        ) -> Tuple[AgentAction, str]:\n            if run_manager:\n                await run_manager.on_agent_action(\n                    agent_action, verbose=self.verbose, color=\"green\"\n                )\n            # Otherwise we lookup the tool\n            if agent_action.tool in name_to_tool_map:\n                tool = name_to_tool_map[agent_action.tool]\n                return_direct = tool.return_direct\n                color = color_mapping[agent_action.tool]\n                tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n                if return_direct:\n                    tool_run_kwargs[\"llm_prefix\"] = \"\"\n                # We then call the tool on the tool input to get an observation\n                observation = await tool.arun(\n                    agent_action.tool_input,\n                    verbose=self.verbose,\n                    color=color,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **tool_run_kwargs,\n                )\n            else:\n                tool_run_kwargs = self.agent.tool_run_logging_kwargs()\n                observation = await InvalidTool().arun(\n                    agent_action.tool,\n                    verbose=self.verbose,\n                    color=None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}499{"id": "df388dc9008a-19", "text": "agent_action.tool,\n                    verbose=self.verbose,\n                    color=None,\n                    callbacks=run_manager.get_child() if run_manager else None,\n                    **tool_run_kwargs,\n                )\n            return agent_action, observation\n        # Use asyncio.gather to run multiple tool.arun() calls concurrently\n        result = await asyncio.gather(\n            *[_aperform_agent_action(agent_action) for agent_action in actions]\n        )\n        return list(result)\n    def _call(\n        self,\n        inputs: Dict[str, str],\n        run_manager: Optional[CallbackManagerForChainRun] = None,\n    ) -> Dict[str, Any]:\n        \"\"\"Run text through and get agent response.\"\"\"\n        # Construct a mapping of tool name to tool for easy lookup\n        name_to_tool_map = {tool.name: tool for tool in self.tools}\n        # We construct a mapping from each tool to a color, used for logging.\n        color_mapping = get_color_mapping(\n            [tool.name for tool in self.tools], excluded_colors=[\"green\"]\n        )\n        intermediate_steps: List[Tuple[AgentAction, str]] = []\n        # Let's start tracking the number of iterations and time elapsed\n        iterations = 0\n        time_elapsed = 0.0\n        start_time = time.time()\n        # We now enter the agent loop (until it returns something).\n        while self._should_continue(iterations, time_elapsed):\n            next_step_output = self._take_next_step(\n                name_to_tool_map,\n                color_mapping,\n                inputs,\n                intermediate_steps,\n                run_manager=run_manager,\n            )\n            if isinstance(next_step_output, AgentFinish):\n                return self._return(\n                    next_step_output, intermediate_steps, run_manager=run_manager\n                )", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}500{"id": "df388dc9008a-20", "text": "next_step_output, intermediate_steps, run_manager=run_manager\n                )\n            intermediate_steps.extend(next_step_output)\n            if len(next_step_output) == 1:\n                next_step_action = next_step_output[0]\n                # See if tool should return directly\n                tool_return = self._get_tool_return(next_step_action)\n                if tool_return is not None:\n                    return self._return(\n                        tool_return, intermediate_steps, run_manager=run_manager\n                    )\n            iterations += 1\n            time_elapsed = time.time() - start_time\n        output = self.agent.return_stopped_response(\n            self.early_stopping_method, intermediate_steps, **inputs\n        )\n        return self._return(output, intermediate_steps, run_manager=run_manager)\n    async def _acall(\n        self,\n        inputs: Dict[str, str],\n        run_manager: Optional[AsyncCallbackManagerForChainRun] = None,\n    ) -> Dict[str, str]:\n        \"\"\"Run text through and get agent response.\"\"\"\n        # Construct a mapping of tool name to tool for easy lookup\n        name_to_tool_map = {tool.name: tool for tool in self.tools}\n        # We construct a mapping from each tool to a color, used for logging.\n        color_mapping = get_color_mapping(\n            [tool.name for tool in self.tools], excluded_colors=[\"green\"]\n        )\n        intermediate_steps: List[Tuple[AgentAction, str]] = []\n        # Let's start tracking the number of iterations and time elapsed\n        iterations = 0\n        time_elapsed = 0.0\n        start_time = time.time()\n        # We now enter the agent loop (until it returns something).\n        async with asyncio_timeout(self.max_execution_time):\n            try:\n                while self._should_continue(iterations, time_elapsed):", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}501{"id": "df388dc9008a-21", "text": "try:\n                while self._should_continue(iterations, time_elapsed):\n                    next_step_output = await self._atake_next_step(\n                        name_to_tool_map,\n                        color_mapping,\n                        inputs,\n                        intermediate_steps,\n                        run_manager=run_manager,\n                    )\n                    if isinstance(next_step_output, AgentFinish):\n                        return await self._areturn(\n                            next_step_output,\n                            intermediate_steps,\n                            run_manager=run_manager,\n                        )\n                    intermediate_steps.extend(next_step_output)\n                    if len(next_step_output) == 1:\n                        next_step_action = next_step_output[0]\n                        # See if tool should return directly\n                        tool_return = self._get_tool_return(next_step_action)\n                        if tool_return is not None:\n                            return await self._areturn(\n                                tool_return, intermediate_steps, run_manager=run_manager\n                            )\n                    iterations += 1\n                    time_elapsed = time.time() - start_time\n                output = self.agent.return_stopped_response(\n                    self.early_stopping_method, intermediate_steps, **inputs\n                )\n                return await self._areturn(\n                    output, intermediate_steps, run_manager=run_manager\n                )\n            except TimeoutError:\n                # stop early when interrupted by the async timeout\n                output = self.agent.return_stopped_response(\n                    self.early_stopping_method, intermediate_steps, **inputs\n                )\n                return await self._areturn(\n                    output, intermediate_steps, run_manager=run_manager\n                )\n    def _get_tool_return(\n        self, next_step_output: Tuple[AgentAction, str]\n    ) -> Optional[AgentFinish]:\n        \"\"\"Check if the tool is a returning tool.\"\"\"\n        agent_action, observation = next_step_output", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}502{"id": "df388dc9008a-22", "text": "agent_action, observation = next_step_output\n        name_to_tool_map = {tool.name: tool for tool in self.tools}\n        # Invalid tools won't be in the map, so we return False.\n        if agent_action.tool in name_to_tool_map:\n            if name_to_tool_map[agent_action.tool].return_direct:\n                return AgentFinish(\n                    {self.agent.return_values[0]: observation},\n                    \"\",\n                )\n        return None\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent.html"}503{"id": "8ae79f1bf5aa-0", "text": "Source code for langchain.agents.loading\n\"\"\"Functionality for loading agents.\"\"\"\nimport json\nimport logging\nfrom pathlib import Path\nfrom typing import Any, List, Optional, Union\nimport yaml\nfrom langchain.agents.agent import BaseSingleActionAgent\nfrom langchain.agents.tools import Tool\nfrom langchain.agents.types import AGENT_TO_CLASS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.loading import load_chain, load_chain_from_config\nfrom langchain.utilities.loading import try_load_from_hub\nlogger = logging.getLogger(__file__)\nURL_BASE = \"https://raw.githubusercontent.com/hwchase17/langchain-hub/master/agents/\"\ndef _load_agent_from_tools(\n    config: dict, llm: BaseLanguageModel, tools: List[Tool], **kwargs: Any\n) -> BaseSingleActionAgent:\n    config_type = config.pop(\"_type\")\n    if config_type not in AGENT_TO_CLASS:\n        raise ValueError(f\"Loading {config_type} agent not supported\")\n    agent_cls = AGENT_TO_CLASS[config_type]\n    combined_config = {**config, **kwargs}\n    return agent_cls.from_llm_and_tools(llm, tools, **combined_config)\ndef load_agent_from_config(\n    config: dict,\n    llm: Optional[BaseLanguageModel] = None,\n    tools: Optional[List[Tool]] = None,\n    **kwargs: Any,\n) -> BaseSingleActionAgent:\n    \"\"\"Load agent from Config Dict.\"\"\"\n    if \"_type\" not in config:\n        raise ValueError(\"Must specify an agent Type in config\")\n    load_from_tools = config.pop(\"load_from_llm_and_tools\", False)\n    if load_from_tools:\n        if llm is None:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/loading.html"}504{"id": "8ae79f1bf5aa-1", "text": "if load_from_tools:\n        if llm is None:\n            raise ValueError(\n                \"If `load_from_llm_and_tools` is set to True, \"\n                \"then LLM must be provided\"\n            )\n        if tools is None:\n            raise ValueError(\n                \"If `load_from_llm_and_tools` is set to True, \"\n                \"then tools must be provided\"\n            )\n        return _load_agent_from_tools(config, llm, tools, **kwargs)\n    config_type = config.pop(\"_type\")\n    if config_type not in AGENT_TO_CLASS:\n        raise ValueError(f\"Loading {config_type} agent not supported\")\n    agent_cls = AGENT_TO_CLASS[config_type]\n    if \"llm_chain\" in config:\n        config[\"llm_chain\"] = load_chain_from_config(config.pop(\"llm_chain\"))\n    elif \"llm_chain_path\" in config:\n        config[\"llm_chain\"] = load_chain(config.pop(\"llm_chain_path\"))\n    else:\n        raise ValueError(\"One of `llm_chain` and `llm_chain_path` should be specified.\")\n    if \"output_parser\" in config:\n        logger.warning(\n            \"Currently loading output parsers on agent is not supported, \"\n            \"will just use the default one.\"\n        )\n        del config[\"output_parser\"]\n    combined_config = {**config, **kwargs}\n    return agent_cls(**combined_config)  # type: ignore\n[docs]def load_agent(path: Union[str, Path], **kwargs: Any) -> BaseSingleActionAgent:\n    \"\"\"Unified method for loading a agent from LangChainHub or local fs.\"\"\"\n    if hub_result := try_load_from_hub(\n        path, _load_agent_from_file, \"agents\", {\"json\", \"yaml\"}\n    ):", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/loading.html"}505{"id": "8ae79f1bf5aa-2", "text": "):\n        return hub_result\n    else:\n        return _load_agent_from_file(path, **kwargs)\ndef _load_agent_from_file(\n    file: Union[str, Path], **kwargs: Any\n) -> BaseSingleActionAgent:\n    \"\"\"Load agent from file.\"\"\"\n    # Convert file to Path object.\n    if isinstance(file, str):\n        file_path = Path(file)\n    else:\n        file_path = file\n    # Load from either json or yaml.\n    if file_path.suffix == \".json\":\n        with open(file_path) as f:\n            config = json.load(f)\n    elif file_path.suffix == \".yaml\":\n        with open(file_path, \"r\") as f:\n            config = yaml.safe_load(f)\n    else:\n        raise ValueError(\"File type must be json or yaml\")\n    # Load the agent from the config now.\n    return load_agent_from_config(config, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/loading.html"}506{"id": "799a872483e5-0", "text": "Source code for langchain.agents.load_tools\n# flake8: noqa\n\"\"\"Load tools.\"\"\"\nimport warnings\nfrom typing import Any, Dict, List, Optional, Callable, Tuple\nfrom mypy_extensions import Arg, KwArg\nfrom langchain.agents.tools import Tool\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.callbacks.manager import Callbacks\nfrom langchain.chains.api import news_docs, open_meteo_docs, podcast_docs, tmdb_docs\nfrom langchain.chains.api.base import APIChain\nfrom langchain.chains.llm_math.base import LLMMathChain\nfrom langchain.chains.pal.base import PALChain\nfrom langchain.requests import TextRequestsWrapper\nfrom langchain.tools.arxiv.tool import ArxivQueryRun\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.bing_search.tool import BingSearchRun\nfrom langchain.tools.ddg_search.tool import DuckDuckGoSearchRun\nfrom langchain.tools.google_search.tool import GoogleSearchResults, GoogleSearchRun\nfrom langchain.tools.metaphor_search.tool import MetaphorSearchResults\nfrom langchain.tools.google_serper.tool import GoogleSerperResults, GoogleSerperRun\nfrom langchain.tools.graphql.tool import BaseGraphQLTool\nfrom langchain.tools.human.tool import HumanInputRun\nfrom langchain.tools.python.tool import PythonREPLTool\nfrom langchain.tools.requests.tool import (\n    RequestsDeleteTool,\n    RequestsGetTool,\n    RequestsPatchTool,\n    RequestsPostTool,\n    RequestsPutTool,\n)\nfrom langchain.tools.scenexplain.tool import SceneXplainTool\nfrom langchain.tools.searx_search.tool import SearxSearchResults, SearxSearchRun\nfrom langchain.tools.shell.tool import ShellTool\nfrom langchain.tools.wikipedia.tool import WikipediaQueryRun", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}507{"id": "799a872483e5-1", "text": "from langchain.tools.shell.tool import ShellTool\nfrom langchain.tools.wikipedia.tool import WikipediaQueryRun\nfrom langchain.tools.wolfram_alpha.tool import WolframAlphaQueryRun\nfrom langchain.tools.openweathermap.tool import OpenWeatherMapQueryRun\nfrom langchain.utilities import ArxivAPIWrapper\nfrom langchain.utilities.bing_search import BingSearchAPIWrapper\nfrom langchain.utilities.duckduckgo_search import DuckDuckGoSearchAPIWrapper\nfrom langchain.utilities.google_search import GoogleSearchAPIWrapper\nfrom langchain.utilities.google_serper import GoogleSerperAPIWrapper\nfrom langchain.utilities.metaphor_search import MetaphorSearchAPIWrapper\nfrom langchain.utilities.awslambda import LambdaWrapper\nfrom langchain.utilities.graphql import GraphQLAPIWrapper\nfrom langchain.utilities.searx_search import SearxSearchWrapper\nfrom langchain.utilities.serpapi import SerpAPIWrapper\nfrom langchain.utilities.twilio import TwilioAPIWrapper\nfrom langchain.utilities.wikipedia import WikipediaAPIWrapper\nfrom langchain.utilities.wolfram_alpha import WolframAlphaAPIWrapper\nfrom langchain.utilities.openweathermap import OpenWeatherMapAPIWrapper\ndef _get_python_repl() -> BaseTool:\n    return PythonREPLTool()\ndef _get_tools_requests_get() -> BaseTool:\n    return RequestsGetTool(requests_wrapper=TextRequestsWrapper())\ndef _get_tools_requests_post() -> BaseTool:\n    return RequestsPostTool(requests_wrapper=TextRequestsWrapper())\ndef _get_tools_requests_patch() -> BaseTool:\n    return RequestsPatchTool(requests_wrapper=TextRequestsWrapper())\ndef _get_tools_requests_put() -> BaseTool:\n    return RequestsPutTool(requests_wrapper=TextRequestsWrapper())\ndef _get_tools_requests_delete() -> BaseTool:\n    return RequestsDeleteTool(requests_wrapper=TextRequestsWrapper())\ndef _get_terminal() -> BaseTool:\n    return ShellTool()", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}508{"id": "799a872483e5-2", "text": "def _get_terminal() -> BaseTool:\n    return ShellTool()\n_BASE_TOOLS: Dict[str, Callable[[], BaseTool]] = {\n    \"python_repl\": _get_python_repl,\n    \"requests\": _get_tools_requests_get,  # preserved for backwards compatability\n    \"requests_get\": _get_tools_requests_get,\n    \"requests_post\": _get_tools_requests_post,\n    \"requests_patch\": _get_tools_requests_patch,\n    \"requests_put\": _get_tools_requests_put,\n    \"requests_delete\": _get_tools_requests_delete,\n    \"terminal\": _get_terminal,\n}\ndef _get_pal_math(llm: BaseLanguageModel) -> BaseTool:\n    return Tool(\n        name=\"PAL-MATH\",\n        description=\"A language model that is really good at solving complex word math problems. Input should be a fully worded hard word math problem.\",\n        func=PALChain.from_math_prompt(llm).run,\n    )\ndef _get_pal_colored_objects(llm: BaseLanguageModel) -> BaseTool:\n    return Tool(\n        name=\"PAL-COLOR-OBJ\",\n        description=\"A language model that is really good at reasoning about position and the color attributes of objects. Input should be a fully worded hard reasoning problem. Make sure to include all information about the objects AND the final question you want to answer.\",\n        func=PALChain.from_colored_object_prompt(llm).run,\n    )\ndef _get_llm_math(llm: BaseLanguageModel) -> BaseTool:\n    return Tool(\n        name=\"Calculator\",\n        description=\"Useful for when you need to answer questions about math.\",\n        func=LLMMathChain.from_llm(llm=llm).run,\n        coroutine=LLMMathChain.from_llm(llm=llm).arun,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}509{"id": "799a872483e5-3", "text": "coroutine=LLMMathChain.from_llm(llm=llm).arun,\n    )\ndef _get_open_meteo_api(llm: BaseLanguageModel) -> BaseTool:\n    chain = APIChain.from_llm_and_api_docs(llm, open_meteo_docs.OPEN_METEO_DOCS)\n    return Tool(\n        name=\"Open Meteo API\",\n        description=\"Useful for when you want to get weather information from the OpenMeteo API. The input should be a question in natural language that this API can answer.\",\n        func=chain.run,\n    )\n_LLM_TOOLS: Dict[str, Callable[[BaseLanguageModel], BaseTool]] = {\n    \"pal-math\": _get_pal_math,\n    \"pal-colored-objects\": _get_pal_colored_objects,\n    \"llm-math\": _get_llm_math,\n    \"open-meteo-api\": _get_open_meteo_api,\n}\ndef _get_news_api(llm: BaseLanguageModel, **kwargs: Any) -> BaseTool:\n    news_api_key = kwargs[\"news_api_key\"]\n    chain = APIChain.from_llm_and_api_docs(\n        llm, news_docs.NEWS_DOCS, headers={\"X-Api-Key\": news_api_key}\n    )\n    return Tool(\n        name=\"News API\",\n        description=\"Use this when you want to get information about the top headlines of current news stories. The input should be a question in natural language that this API can answer.\",\n        func=chain.run,\n    )\ndef _get_tmdb_api(llm: BaseLanguageModel, **kwargs: Any) -> BaseTool:\n    tmdb_bearer_token = kwargs[\"tmdb_bearer_token\"]\n    chain = APIChain.from_llm_and_api_docs(\n        llm,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}510{"id": "799a872483e5-4", "text": "chain = APIChain.from_llm_and_api_docs(\n        llm,\n        tmdb_docs.TMDB_DOCS,\n        headers={\"Authorization\": f\"Bearer {tmdb_bearer_token}\"},\n    )\n    return Tool(\n        name=\"TMDB API\",\n        description=\"Useful for when you want to get information from The Movie Database. The input should be a question in natural language that this API can answer.\",\n        func=chain.run,\n    )\ndef _get_podcast_api(llm: BaseLanguageModel, **kwargs: Any) -> BaseTool:\n    listen_api_key = kwargs[\"listen_api_key\"]\n    chain = APIChain.from_llm_and_api_docs(\n        llm,\n        podcast_docs.PODCAST_DOCS,\n        headers={\"X-ListenAPI-Key\": listen_api_key},\n    )\n    return Tool(\n        name=\"Podcast API\",\n        description=\"Use the Listen Notes Podcast API to search all podcasts or episodes. The input should be a question in natural language that this API can answer.\",\n        func=chain.run,\n    )\ndef _get_lambda_api(**kwargs: Any) -> BaseTool:\n    return Tool(\n        name=kwargs[\"awslambda_tool_name\"],\n        description=kwargs[\"awslambda_tool_description\"],\n        func=LambdaWrapper(**kwargs).run,\n    )\ndef _get_wolfram_alpha(**kwargs: Any) -> BaseTool:\n    return WolframAlphaQueryRun(api_wrapper=WolframAlphaAPIWrapper(**kwargs))\ndef _get_google_search(**kwargs: Any) -> BaseTool:\n    return GoogleSearchRun(api_wrapper=GoogleSearchAPIWrapper(**kwargs))\ndef _get_wikipedia(**kwargs: Any) -> BaseTool:\n    return WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper(**kwargs))", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}511{"id": "799a872483e5-5", "text": "return WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper(**kwargs))\ndef _get_arxiv(**kwargs: Any) -> BaseTool:\n    return ArxivQueryRun(api_wrapper=ArxivAPIWrapper(**kwargs))\ndef _get_google_serper(**kwargs: Any) -> BaseTool:\n    return GoogleSerperRun(api_wrapper=GoogleSerperAPIWrapper(**kwargs))\ndef _get_google_serper_results_json(**kwargs: Any) -> BaseTool:\n    return GoogleSerperResults(api_wrapper=GoogleSerperAPIWrapper(**kwargs))\ndef _get_google_search_results_json(**kwargs: Any) -> BaseTool:\n    return GoogleSearchResults(api_wrapper=GoogleSearchAPIWrapper(**kwargs))\ndef _get_serpapi(**kwargs: Any) -> BaseTool:\n    return Tool(\n        name=\"Search\",\n        description=\"A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\",\n        func=SerpAPIWrapper(**kwargs).run,\n        coroutine=SerpAPIWrapper(**kwargs).arun,\n    )\ndef _get_twilio(**kwargs: Any) -> BaseTool:\n    return Tool(\n        name=\"Text Message\",\n        description=\"Useful for when you need to send a text message to a provided phone number.\",\n        func=TwilioAPIWrapper(**kwargs).run,\n    )\ndef _get_searx_search(**kwargs: Any) -> BaseTool:\n    return SearxSearchRun(wrapper=SearxSearchWrapper(**kwargs))\ndef _get_searx_search_results_json(**kwargs: Any) -> BaseTool:\n    wrapper_kwargs = {k: v for k, v in kwargs.items() if k != \"num_results\"}\n    return SearxSearchResults(wrapper=SearxSearchWrapper(**wrapper_kwargs), **kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}512{"id": "799a872483e5-6", "text": "return SearxSearchResults(wrapper=SearxSearchWrapper(**wrapper_kwargs), **kwargs)\ndef _get_bing_search(**kwargs: Any) -> BaseTool:\n    return BingSearchRun(api_wrapper=BingSearchAPIWrapper(**kwargs))\ndef _get_metaphor_search(**kwargs: Any) -> BaseTool:\n    return MetaphorSearchResults(api_wrapper=MetaphorSearchAPIWrapper(**kwargs))\ndef _get_ddg_search(**kwargs: Any) -> BaseTool:\n    return DuckDuckGoSearchRun(api_wrapper=DuckDuckGoSearchAPIWrapper(**kwargs))\ndef _get_human_tool(**kwargs: Any) -> BaseTool:\n    return HumanInputRun(**kwargs)\ndef _get_scenexplain(**kwargs: Any) -> BaseTool:\n    return SceneXplainTool(**kwargs)\ndef _get_graphql_tool(**kwargs: Any) -> BaseTool:\n    graphql_endpoint = kwargs[\"graphql_endpoint\"]\n    wrapper = GraphQLAPIWrapper(graphql_endpoint=graphql_endpoint)\n    return BaseGraphQLTool(graphql_wrapper=wrapper)\ndef _get_openweathermap(**kwargs: Any) -> BaseTool:\n    return OpenWeatherMapQueryRun(api_wrapper=OpenWeatherMapAPIWrapper(**kwargs))\n_EXTRA_LLM_TOOLS: Dict[\n    str,\n    Tuple[Callable[[Arg(BaseLanguageModel, \"llm\"), KwArg(Any)], BaseTool], List[str]],\n] = {\n    \"news-api\": (_get_news_api, [\"news_api_key\"]),\n    \"tmdb-api\": (_get_tmdb_api, [\"tmdb_bearer_token\"]),\n    \"podcast-api\": (_get_podcast_api, [\"listen_api_key\"]),\n}\n_EXTRA_OPTIONAL_TOOLS: Dict[str, Tuple[Callable[[KwArg(Any)], BaseTool], List[str]]] = {", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}513{"id": "799a872483e5-7", "text": "\"wolfram-alpha\": (_get_wolfram_alpha, [\"wolfram_alpha_appid\"]),\n    \"google-search\": (_get_google_search, [\"google_api_key\", \"google_cse_id\"]),\n    \"google-search-results-json\": (\n        _get_google_search_results_json,\n        [\"google_api_key\", \"google_cse_id\", \"num_results\"],\n    ),\n    \"searx-search-results-json\": (\n        _get_searx_search_results_json,\n        [\"searx_host\", \"engines\", \"num_results\", \"aiosession\"],\n    ),\n    \"bing-search\": (_get_bing_search, [\"bing_subscription_key\", \"bing_search_url\"]),\n    \"metaphor-search\": (_get_metaphor_search, [\"metaphor_api_key\"]),\n    \"ddg-search\": (_get_ddg_search, []),\n    \"google-serper\": (_get_google_serper, [\"serper_api_key\", \"aiosession\"]),\n    \"google-serper-results-json\": (\n        _get_google_serper_results_json,\n        [\"serper_api_key\", \"aiosession\"],\n    ),\n    \"serpapi\": (_get_serpapi, [\"serpapi_api_key\", \"aiosession\"]),\n    \"twilio\": (_get_twilio, [\"account_sid\", \"auth_token\", \"from_number\"]),\n    \"searx-search\": (_get_searx_search, [\"searx_host\", \"engines\", \"aiosession\"]),\n    \"wikipedia\": (_get_wikipedia, [\"top_k_results\", \"lang\"]),\n    \"arxiv\": (\n        _get_arxiv,\n        [\"top_k_results\", \"load_max_docs\", \"load_all_available_meta\"],\n    ),\n    \"human\": (_get_human_tool, [\"prompt_func\", \"input_func\"]),", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}514{"id": "799a872483e5-8", "text": "),\n    \"human\": (_get_human_tool, [\"prompt_func\", \"input_func\"]),\n    \"awslambda\": (\n        _get_lambda_api,\n        [\"awslambda_tool_name\", \"awslambda_tool_description\", \"function_name\"],\n    ),\n    \"sceneXplain\": (_get_scenexplain, []),\n    \"graphql\": (_get_graphql_tool, [\"graphql_endpoint\"]),\n    \"openweathermap-api\": (_get_openweathermap, [\"openweathermap_api_key\"]),\n}\ndef _handle_callbacks(\n    callback_manager: Optional[BaseCallbackManager], callbacks: Callbacks\n) -> Callbacks:\n    if callback_manager is not None:\n        warnings.warn(\n            \"callback_manager is deprecated. Please use callbacks instead.\",\n            DeprecationWarning,\n        )\n        if callbacks is not None:\n            raise ValueError(\n                \"Cannot specify both callback_manager and callbacks arguments.\"\n            )\n        return callback_manager\n    return callbacks\n[docs]def load_huggingface_tool(\n    task_or_repo_id: str,\n    model_repo_id: Optional[str] = None,\n    token: Optional[str] = None,\n    remote: bool = False,\n    **kwargs: Any,\n) -> BaseTool:\n    try:\n        from transformers import load_tool\n    except ImportError:\n        raise ValueError(\n            \"HuggingFace tools require the libraries `transformers>=4.29.0`\"\n            \" and `huggingface_hub>=0.14.1` to be installed.\"\n            \" Please install it with\"\n            \" `pip install --upgrade transformers huggingface_hub`.\"\n        )\n    hf_tool = load_tool(\n        task_or_repo_id,\n        model_repo_id=model_repo_id,\n        token=token,\n        remote=remote,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}515{"id": "799a872483e5-9", "text": "model_repo_id=model_repo_id,\n        token=token,\n        remote=remote,\n        **kwargs,\n    )\n    outputs = hf_tool.outputs\n    if set(outputs) != {\"text\"}:\n        raise NotImplementedError(\"Multimodal outputs not supported yet.\")\n    inputs = hf_tool.inputs\n    if set(inputs) != {\"text\"}:\n        raise NotImplementedError(\"Multimodal inputs not supported yet.\")\n    return Tool.from_function(\n        hf_tool.__call__, name=hf_tool.name, description=hf_tool.description\n    )\n[docs]def load_tools(\n    tool_names: List[str],\n    llm: Optional[BaseLanguageModel] = None,\n    callbacks: Callbacks = None,\n    **kwargs: Any,\n) -> List[BaseTool]:\n    \"\"\"Load tools based on their name.\n    Args:\n        tool_names: name of tools to load.\n        llm: Optional language model, may be needed to initialize certain tools.\n        callbacks: Optional callback manager or list of callback handlers.\n            If not provided, default global callback manager will be used.\n    Returns:\n        List of tools.\n    \"\"\"\n    tools = []\n    callbacks = _handle_callbacks(\n        callback_manager=kwargs.get(\"callback_manager\"), callbacks=callbacks\n    )\n    for name in tool_names:\n        if name == \"requests\":\n            warnings.warn(\n                \"tool name `requests` is deprecated - \"\n                \"please use `requests_all` or specify the requests method\"\n            )\n        if name == \"requests_all\":\n            # expand requests into various methods\n            requests_method_tools = [\n                _tool for _tool in _BASE_TOOLS if _tool.startswith(\"requests_\")\n            ]\n            tool_names.extend(requests_method_tools)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}516{"id": "799a872483e5-10", "text": "]\n            tool_names.extend(requests_method_tools)\n        elif name in _BASE_TOOLS:\n            tools.append(_BASE_TOOLS[name]())\n        elif name in _LLM_TOOLS:\n            if llm is None:\n                raise ValueError(f\"Tool {name} requires an LLM to be provided\")\n            tool = _LLM_TOOLS[name](llm)\n            tools.append(tool)\n        elif name in _EXTRA_LLM_TOOLS:\n            if llm is None:\n                raise ValueError(f\"Tool {name} requires an LLM to be provided\")\n            _get_llm_tool_func, extra_keys = _EXTRA_LLM_TOOLS[name]\n            missing_keys = set(extra_keys).difference(kwargs)\n            if missing_keys:\n                raise ValueError(\n                    f\"Tool {name} requires some parameters that were not \"\n                    f\"provided: {missing_keys}\"\n                )\n            sub_kwargs = {k: kwargs[k] for k in extra_keys}\n            tool = _get_llm_tool_func(llm=llm, **sub_kwargs)\n            tools.append(tool)\n        elif name in _EXTRA_OPTIONAL_TOOLS:\n            _get_tool_func, extra_keys = _EXTRA_OPTIONAL_TOOLS[name]\n            sub_kwargs = {k: kwargs[k] for k in extra_keys if k in kwargs}\n            tool = _get_tool_func(**sub_kwargs)\n            tools.append(tool)\n        else:\n            raise ValueError(f\"Got unknown tool {name}\")\n    if callbacks is not None:\n        for tool in tools:\n            tool.callbacks = callbacks\n    return tools\n[docs]def get_all_tool_names() -> List[str]:\n    \"\"\"Get a list of all possible tool names.\"\"\"\n    return (\n        list(_BASE_TOOLS)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}517{"id": "799a872483e5-11", "text": "return (\n        list(_BASE_TOOLS)\n        + list(_EXTRA_OPTIONAL_TOOLS)\n        + list(_EXTRA_LLM_TOOLS)\n        + list(_LLM_TOOLS)\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/load_tools.html"}518{"id": "165c72fd8502-0", "text": "Source code for langchain.agents.agent_types\nfrom enum import Enum\n[docs]class AgentType(str, Enum):\n    ZERO_SHOT_REACT_DESCRIPTION = \"zero-shot-react-description\"\n    REACT_DOCSTORE = \"react-docstore\"\n    SELF_ASK_WITH_SEARCH = \"self-ask-with-search\"\n    CONVERSATIONAL_REACT_DESCRIPTION = \"conversational-react-description\"\n    CHAT_ZERO_SHOT_REACT_DESCRIPTION = \"chat-zero-shot-react-description\"\n    CHAT_CONVERSATIONAL_REACT_DESCRIPTION = \"chat-conversational-react-description\"\n    STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION = (\n        \"structured-chat-zero-shot-react-description\"\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_types.html"}519{"id": "d78b88234c4b-0", "text": "Source code for langchain.agents.self_ask_with_search.base\n\"\"\"Chain that does self ask with search.\"\"\"\nfrom typing import Any, Sequence, Union\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentExecutor, AgentOutputParser\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.self_ask_with_search.output_parser import SelfAskOutputParser\nfrom langchain.agents.self_ask_with_search.prompt import PROMPT\nfrom langchain.agents.tools import Tool\nfrom langchain.agents.utils import validate_tools_single_input\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.tools.base import BaseTool\nfrom langchain.utilities.google_serper import GoogleSerperAPIWrapper\nfrom langchain.utilities.serpapi import SerpAPIWrapper\nclass SelfAskWithSearchAgent(Agent):\n    \"\"\"Agent for the self-ask-with-search paper.\"\"\"\n    output_parser: AgentOutputParser = Field(default_factory=SelfAskOutputParser)\n    @classmethod\n    def _get_default_output_parser(cls, **kwargs: Any) -> AgentOutputParser:\n        return SelfAskOutputParser()\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        return AgentType.SELF_ASK_WITH_SEARCH\n    @classmethod\n    def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:\n        \"\"\"Prompt does not depend on tools.\"\"\"\n        return PROMPT\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        validate_tools_single_input(cls.__name__, tools)\n        super()._validate_tools(tools)\n        if len(tools) != 1:\n            raise ValueError(f\"Exactly one tool must be specified, but got {tools}\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/self_ask_with_search/base.html"}520{"id": "d78b88234c4b-1", "text": "raise ValueError(f\"Exactly one tool must be specified, but got {tools}\")\n        tool_names = {tool.name for tool in tools}\n        if tool_names != {\"Intermediate Answer\"}:\n            raise ValueError(\n                f\"Tool name should be Intermediate Answer, got {tool_names}\"\n            )\n    @property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Intermediate answer: \"\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the LLM call with.\"\"\"\n        return \"\"\n[docs]class SelfAskWithSearchChain(AgentExecutor):\n    \"\"\"Chain that does self ask with search.\n    Example:\n        .. code-block:: python\n            from langchain import SelfAskWithSearchChain, OpenAI, GoogleSerperAPIWrapper\n            search_chain = GoogleSerperAPIWrapper()\n            self_ask = SelfAskWithSearchChain(llm=OpenAI(), search_chain=search_chain)\n    \"\"\"\n    def __init__(\n        self,\n        llm: BaseLanguageModel,\n        search_chain: Union[GoogleSerperAPIWrapper, SerpAPIWrapper],\n        **kwargs: Any,\n    ):\n        \"\"\"Initialize with just an LLM and a search chain.\"\"\"\n        search_tool = Tool(\n            name=\"Intermediate Answer\", func=search_chain.run, description=\"Search\"\n        )\n        agent = SelfAskWithSearchAgent.from_llm_and_tools(llm, [search_tool])\n        super().__init__(agent=agent, tools=[search_tool], **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/self_ask_with_search/base.html"}521{"id": "233ed01767dd-0", "text": "Source code for langchain.agents.mrkl.base\n\"\"\"Attempt to implement MRKL systems as described in arxiv.org/pdf/2205.00445.pdf.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, Callable, List, NamedTuple, Optional, Sequence\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentExecutor, AgentOutputParser\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.mrkl.output_parser import MRKLOutputParser\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS, PREFIX, SUFFIX\nfrom langchain.agents.tools import Tool\nfrom langchain.agents.utils import validate_tools_single_input\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain.tools.base import BaseTool\nclass ChainConfig(NamedTuple):\n    \"\"\"Configuration for chain to use in MRKL system.\n    Args:\n        action_name: Name of the action.\n        action: Action function to call.\n        action_description: Description of the action.\n    \"\"\"\n    action_name: str\n    action: Callable\n    action_description: str\n[docs]class ZeroShotAgent(Agent):\n    \"\"\"Agent for the MRKL chain.\"\"\"\n    output_parser: AgentOutputParser = Field(default_factory=MRKLOutputParser)\n    @classmethod\n    def _get_default_output_parser(cls, **kwargs: Any) -> AgentOutputParser:\n        return MRKLOutputParser()\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        return AgentType.ZERO_SHOT_REACT_DESCRIPTION\n    @property\n    def observation_prefix(self) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/mrkl/base.html"}522{"id": "233ed01767dd-1", "text": "@property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Observation: \"\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the llm call with.\"\"\"\n        return \"Thought:\"\n[docs]    @classmethod\n    def create_prompt(\n        cls,\n        tools: Sequence[BaseTool],\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        input_variables: Optional[List[str]] = None,\n    ) -> PromptTemplate:\n        \"\"\"Create prompt in the style of the zero shot agent.\n        Args:\n            tools: List of tools the agent will have access to, used to format the\n                prompt.\n            prefix: String to put before the list of tools.\n            suffix: String to put after the list of tools.\n            input_variables: List of input variables the final prompt will expect.\n        Returns:\n            A PromptTemplate with the template assembled from the pieces here.\n        \"\"\"\n        tool_strings = \"\\n\".join([f\"{tool.name}: {tool.description}\" for tool in tools])\n        tool_names = \", \".join([tool.name for tool in tools])\n        format_instructions = format_instructions.format(tool_names=tool_names)\n        template = \"\\n\\n\".join([prefix, tool_strings, format_instructions, suffix])\n        if input_variables is None:\n            input_variables = [\"input\", \"agent_scratchpad\"]\n        return PromptTemplate(template=template, input_variables=input_variables)\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/mrkl/base.html"}523{"id": "233ed01767dd-2", "text": "llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        output_parser: Optional[AgentOutputParser] = None,\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        input_variables: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> Agent:\n        \"\"\"Construct an agent from an LLM and tools.\"\"\"\n        cls._validate_tools(tools)\n        prompt = cls.create_prompt(\n            tools,\n            prefix=prefix,\n            suffix=suffix,\n            format_instructions=format_instructions,\n            input_variables=input_variables,\n        )\n        llm_chain = LLMChain(\n            llm=llm,\n            prompt=prompt,\n            callback_manager=callback_manager,\n        )\n        tool_names = [tool.name for tool in tools]\n        _output_parser = output_parser or cls._get_default_output_parser()\n        return cls(\n            llm_chain=llm_chain,\n            allowed_tools=tool_names,\n            output_parser=_output_parser,\n            **kwargs,\n        )\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        validate_tools_single_input(cls.__name__, tools)\n        for tool in tools:\n            if tool.description is None:\n                raise ValueError(\n                    f\"Got a tool {tool.name} without a description. For this agent, \"\n                    f\"a description must always be provided.\"\n                )\n        super()._validate_tools(tools)\n[docs]class MRKLChain(AgentExecutor):\n    \"\"\"Chain that implements the MRKL system.\n    Example:\n        .. code-block:: python", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/mrkl/base.html"}524{"id": "233ed01767dd-3", "text": "Example:\n        .. code-block:: python\n            from langchain import OpenAI, MRKLChain\n            from langchain.chains.mrkl.base import ChainConfig\n            llm = OpenAI(temperature=0)\n            prompt = PromptTemplate(...)\n            chains = [...]\n            mrkl = MRKLChain.from_chains(llm=llm, prompt=prompt)\n    \"\"\"\n[docs]    @classmethod\n    def from_chains(\n        cls, llm: BaseLanguageModel, chains: List[ChainConfig], **kwargs: Any\n    ) -> AgentExecutor:\n        \"\"\"User friendly way to initialize the MRKL chain.\n        This is intended to be an easy way to get up and running with the\n        MRKL chain.\n        Args:\n            llm: The LLM to use as the agent LLM.\n            chains: The chains the MRKL system has access to.\n            **kwargs: parameters to be passed to initialization.\n        Returns:\n            An initialized MRKL chain.\n        Example:\n            .. code-block:: python\n                from langchain import LLMMathChain, OpenAI, SerpAPIWrapper, MRKLChain\n                from langchain.chains.mrkl.base import ChainConfig\n                llm = OpenAI(temperature=0)\n                search = SerpAPIWrapper()\n                llm_math_chain = LLMMathChain(llm=llm)\n                chains = [\n                    ChainConfig(\n                        action_name = \"Search\",\n                        action=search.search,\n                        action_description=\"useful for searching\"\n                    ),\n                    ChainConfig(\n                        action_name=\"Calculator\",\n                        action=llm_math_chain.run,\n                        action_description=\"useful for doing math\"\n                    )\n                ]\n                mrkl = MRKLChain.from_chains(llm, chains)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/mrkl/base.html"}525{"id": "233ed01767dd-4", "text": "]\n                mrkl = MRKLChain.from_chains(llm, chains)\n        \"\"\"\n        tools = [\n            Tool(\n                name=c.action_name,\n                func=c.action,\n                description=c.action_description,\n            )\n            for c in chains\n        ]\n        agent = ZeroShotAgent.from_llm_and_tools(llm, tools)\n        return cls(agent=agent, tools=tools, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/mrkl/base.html"}526{"id": "66ab8508fab5-0", "text": "Source code for langchain.agents.structured_chat.base\nimport re\nfrom typing import Any, List, Optional, Sequence, Tuple\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentOutputParser\nfrom langchain.agents.structured_chat.output_parser import (\n    StructuredChatOutputParserWithRetries,\n)\nfrom langchain.agents.structured_chat.prompt import FORMAT_INSTRUCTIONS, PREFIX, SUFFIX\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.prompts.chat import (\n    ChatPromptTemplate,\n    HumanMessagePromptTemplate,\n    SystemMessagePromptTemplate,\n)\nfrom langchain.schema import AgentAction\nfrom langchain.tools import BaseTool\nHUMAN_MESSAGE_TEMPLATE = \"{input}\\n\\n{agent_scratchpad}\"\n[docs]class StructuredChatAgent(Agent):\n    output_parser: AgentOutputParser = Field(\n        default_factory=StructuredChatOutputParserWithRetries\n    )\n    @property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Observation: \"\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the llm call with.\"\"\"\n        return \"Thought:\"\n    def _construct_scratchpad(\n        self, intermediate_steps: List[Tuple[AgentAction, str]]\n    ) -> str:\n        agent_scratchpad = super()._construct_scratchpad(intermediate_steps)\n        if not isinstance(agent_scratchpad, str):\n            raise ValueError(\"agent_scratchpad should be of type string.\")\n        if agent_scratchpad:\n            return (\n                f\"This was your previous work \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/structured_chat/base.html"}527{"id": "66ab8508fab5-1", "text": "if agent_scratchpad:\n            return (\n                f\"This was your previous work \"\n                f\"(but I haven't seen any of it! I only see what \"\n                f\"you return as final answer):\\n{agent_scratchpad}\"\n            )\n        else:\n            return agent_scratchpad\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        pass\n    @classmethod\n    def _get_default_output_parser(\n        cls, llm: Optional[BaseLanguageModel] = None, **kwargs: Any\n    ) -> AgentOutputParser:\n        return StructuredChatOutputParserWithRetries.from_llm(llm=llm)\n    @property\n    def _stop(self) -> List[str]:\n        return [\"Observation:\"]\n[docs]    @classmethod\n    def create_prompt(\n        cls,\n        tools: Sequence[BaseTool],\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        human_message_template: str = HUMAN_MESSAGE_TEMPLATE,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        input_variables: Optional[List[str]] = None,\n        memory_prompts: Optional[List[BasePromptTemplate]] = None,\n    ) -> BasePromptTemplate:\n        tool_strings = []\n        for tool in tools:\n            args_schema = re.sub(\"}\", \"}}}}\", re.sub(\"{\", \"{{{{\", str(tool.args)))\n            tool_strings.append(f\"{tool.name}: {tool.description}, args: {args_schema}\")\n        formatted_tools = \"\\n\".join(tool_strings)\n        tool_names = \", \".join([tool.name for tool in tools])\n        format_instructions = format_instructions.format(tool_names=tool_names)\n        template = \"\\n\\n\".join([prefix, formatted_tools, format_instructions, suffix])", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/structured_chat/base.html"}528{"id": "66ab8508fab5-2", "text": "template = \"\\n\\n\".join([prefix, formatted_tools, format_instructions, suffix])\n        if input_variables is None:\n            input_variables = [\"input\", \"agent_scratchpad\"]\n        _memory_prompts = memory_prompts or []\n        messages = [\n            SystemMessagePromptTemplate.from_template(template),\n            *_memory_prompts,\n            HumanMessagePromptTemplate.from_template(human_message_template),\n        ]\n        return ChatPromptTemplate(input_variables=input_variables, messages=messages)\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        output_parser: Optional[AgentOutputParser] = None,\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        human_message_template: str = HUMAN_MESSAGE_TEMPLATE,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        input_variables: Optional[List[str]] = None,\n        memory_prompts: Optional[List[BasePromptTemplate]] = None,\n        **kwargs: Any,\n    ) -> Agent:\n        \"\"\"Construct an agent from an LLM and tools.\"\"\"\n        cls._validate_tools(tools)\n        prompt = cls.create_prompt(\n            tools,\n            prefix=prefix,\n            suffix=suffix,\n            human_message_template=human_message_template,\n            format_instructions=format_instructions,\n            input_variables=input_variables,\n            memory_prompts=memory_prompts,\n        )\n        llm_chain = LLMChain(\n            llm=llm,\n            prompt=prompt,\n            callback_manager=callback_manager,\n        )\n        tool_names = [tool.name for tool in tools]", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/structured_chat/base.html"}529{"id": "66ab8508fab5-3", "text": ")\n        tool_names = [tool.name for tool in tools]\n        _output_parser = output_parser or cls._get_default_output_parser(llm=llm)\n        return cls(\n            llm_chain=llm_chain,\n            allowed_tools=tool_names,\n            output_parser=_output_parser,\n            **kwargs,\n        )\n    @property\n    def _agent_type(self) -> str:\n        raise ValueError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/structured_chat/base.html"}530{"id": "8555ddc84c2e-0", "text": "Source code for langchain.agents.conversational_chat.base\n\"\"\"An agent designed to hold a conversation in addition to using tools.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, List, Optional, Sequence, Tuple\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentOutputParser\nfrom langchain.agents.conversational_chat.output_parser import ConvoOutputParser\nfrom langchain.agents.conversational_chat.prompt import (\n    PREFIX,\n    SUFFIX,\n    TEMPLATE_TOOL_RESPONSE,\n)\nfrom langchain.agents.utils import validate_tools_single_input\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains import LLMChain\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.prompts.chat import (\n    ChatPromptTemplate,\n    HumanMessagePromptTemplate,\n    MessagesPlaceholder,\n    SystemMessagePromptTemplate,\n)\nfrom langchain.schema import (\n    AgentAction,\n    AIMessage,\n    BaseMessage,\n    BaseOutputParser,\n    HumanMessage,\n)\nfrom langchain.tools.base import BaseTool\n[docs]class ConversationalChatAgent(Agent):\n    \"\"\"An agent designed to hold a conversation in addition to using tools.\"\"\"\n    output_parser: AgentOutputParser = Field(default_factory=ConvoOutputParser)\n    template_tool_response: str = TEMPLATE_TOOL_RESPONSE\n    @classmethod\n    def _get_default_output_parser(cls, **kwargs: Any) -> AgentOutputParser:\n        return ConvoOutputParser()\n    @property\n    def _agent_type(self) -> str:\n        raise NotImplementedError\n    @property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Observation: \"\n    @property", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational_chat/base.html"}531{"id": "8555ddc84c2e-1", "text": "return \"Observation: \"\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the llm call with.\"\"\"\n        return \"Thought:\"\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        super()._validate_tools(tools)\n        validate_tools_single_input(cls.__name__, tools)\n[docs]    @classmethod\n    def create_prompt(\n        cls,\n        tools: Sequence[BaseTool],\n        system_message: str = PREFIX,\n        human_message: str = SUFFIX,\n        input_variables: Optional[List[str]] = None,\n        output_parser: Optional[BaseOutputParser] = None,\n    ) -> BasePromptTemplate:\n        tool_strings = \"\\n\".join(\n            [f\"> {tool.name}: {tool.description}\" for tool in tools]\n        )\n        tool_names = \", \".join([tool.name for tool in tools])\n        _output_parser = output_parser or cls._get_default_output_parser()\n        format_instructions = human_message.format(\n            format_instructions=_output_parser.get_format_instructions()\n        )\n        final_prompt = format_instructions.format(\n            tool_names=tool_names, tools=tool_strings\n        )\n        if input_variables is None:\n            input_variables = [\"input\", \"chat_history\", \"agent_scratchpad\"]\n        messages = [\n            SystemMessagePromptTemplate.from_template(system_message),\n            MessagesPlaceholder(variable_name=\"chat_history\"),\n            HumanMessagePromptTemplate.from_template(final_prompt),\n            MessagesPlaceholder(variable_name=\"agent_scratchpad\"),\n        ]\n        return ChatPromptTemplate(input_variables=input_variables, messages=messages)\n    def _construct_scratchpad(\n        self, intermediate_steps: List[Tuple[AgentAction, str]]\n    ) -> List[BaseMessage]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational_chat/base.html"}532{"id": "8555ddc84c2e-2", "text": ") -> List[BaseMessage]:\n        \"\"\"Construct the scratchpad that lets the agent continue its thought process.\"\"\"\n        thoughts: List[BaseMessage] = []\n        for action, observation in intermediate_steps:\n            thoughts.append(AIMessage(content=action.log))\n            human_message = HumanMessage(\n                content=self.template_tool_response.format(observation=observation)\n            )\n            thoughts.append(human_message)\n        return thoughts\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        output_parser: Optional[AgentOutputParser] = None,\n        system_message: str = PREFIX,\n        human_message: str = SUFFIX,\n        input_variables: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> Agent:\n        \"\"\"Construct an agent from an LLM and tools.\"\"\"\n        cls._validate_tools(tools)\n        _output_parser = output_parser or cls._get_default_output_parser()\n        prompt = cls.create_prompt(\n            tools,\n            system_message=system_message,\n            human_message=human_message,\n            input_variables=input_variables,\n            output_parser=_output_parser,\n        )\n        llm_chain = LLMChain(\n            llm=llm,\n            prompt=prompt,\n            callback_manager=callback_manager,\n        )\n        tool_names = [tool.name for tool in tools]\n        return cls(\n            llm_chain=llm_chain,\n            allowed_tools=tool_names,\n            output_parser=_output_parser,\n            **kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational_chat/base.html"}533{"id": "8555ddc84c2e-3", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational_chat/base.html"}534{"id": "8b640105eed3-0", "text": "Source code for langchain.agents.agent_toolkits.spark_sql.base\n\"\"\"Spark SQL agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.spark_sql.prompt import SQL_PREFIX, SQL_SUFFIX\nfrom langchain.agents.agent_toolkits.spark_sql.toolkit import SparkSQLToolkit\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\n[docs]def create_spark_sql_agent(\n    llm: BaseLanguageModel,\n    toolkit: SparkSQLToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = SQL_PREFIX,\n    suffix: str = SQL_SUFFIX,\n    format_instructions: str = FORMAT_INSTRUCTIONS,\n    input_variables: Optional[List[str]] = None,\n    top_k: int = 10,\n    max_iterations: Optional[int] = 15,\n    max_execution_time: Optional[float] = None,\n    early_stopping_method: str = \"force\",\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a sql agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prefix = prefix.format(top_k=top_k)\n    prompt = ZeroShotAgent.create_prompt(\n        tools,\n        prefix=prefix,\n        suffix=suffix,\n        format_instructions=format_instructions,\n        input_variables=input_variables,\n    )\n    llm_chain = LLMChain(\n        llm=llm,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/spark_sql/base.html"}535{"id": "8b640105eed3-1", "text": ")\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        max_iterations=max_iterations,\n        max_execution_time=max_execution_time,\n        early_stopping_method=early_stopping_method,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/spark_sql/base.html"}536{"id": "06d05d3b34fd-0", "text": "Source code for langchain.agents.agent_toolkits.spark_sql.toolkit\n\"\"\"Toolkit for interacting with Spark SQL.\"\"\"\nfrom typing import List\nfrom pydantic import Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.tools import BaseTool\nfrom langchain.tools.spark_sql.tool import (\n    InfoSparkSQLTool,\n    ListSparkSQLTool,\n    QueryCheckerTool,\n    QuerySparkSQLTool,\n)\nfrom langchain.utilities.spark_sql import SparkSQL\n[docs]class SparkSQLToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with Spark SQL.\"\"\"\n    db: SparkSQL = Field(exclude=True)\n    llm: BaseLanguageModel = Field(exclude=True)\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return [\n            QuerySparkSQLTool(db=self.db),\n            InfoSparkSQLTool(db=self.db),\n            ListSparkSQLTool(db=self.db),\n            QueryCheckerTool(db=self.db, llm=self.llm),\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/spark_sql/toolkit.html"}537{"id": "e3755439334b-0", "text": "Source code for langchain.agents.agent_toolkits.nla.toolkit\n\"\"\"Toolkit for interacting with API's using natural language.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, List, Optional, Sequence\nfrom pydantic import Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.agents.agent_toolkits.nla.tool import NLATool\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.requests import Requests\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.openapi.utils.openapi_utils import OpenAPISpec\nfrom langchain.tools.plugin import AIPlugin\n[docs]class NLAToolkit(BaseToolkit):\n    \"\"\"Natural Language API Toolkit Definition.\"\"\"\n    nla_tools: Sequence[NLATool] = Field(...)\n    \"\"\"List of API Endpoint Tools.\"\"\"\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools for all the API operations.\"\"\"\n        return list(self.nla_tools)\n    @staticmethod\n    def _get_http_operation_tools(\n        llm: BaseLanguageModel,\n        spec: OpenAPISpec,\n        requests: Optional[Requests] = None,\n        verbose: bool = False,\n        **kwargs: Any,\n    ) -> List[NLATool]:\n        \"\"\"Get the tools for all the API operations.\"\"\"\n        if not spec.paths:\n            return []\n        http_operation_tools = []\n        for path in spec.paths:\n            for method in spec.get_methods_for_path(path):\n                endpoint_tool = NLATool.from_llm_and_method(\n                    llm=llm,\n                    path=path,\n                    method=method,\n                    spec=spec,\n                    requests=requests,\n                    verbose=verbose,\n                    **kwargs,\n                )\n                http_operation_tools.append(endpoint_tool)\n        return http_operation_tools", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/nla/toolkit.html"}538{"id": "e3755439334b-1", "text": ")\n                http_operation_tools.append(endpoint_tool)\n        return http_operation_tools\n[docs]    @classmethod\n    def from_llm_and_spec(\n        cls,\n        llm: BaseLanguageModel,\n        spec: OpenAPISpec,\n        requests: Optional[Requests] = None,\n        verbose: bool = False,\n        **kwargs: Any,\n    ) -> NLAToolkit:\n        \"\"\"Instantiate the toolkit by creating tools for each operation.\"\"\"\n        http_operation_tools = cls._get_http_operation_tools(\n            llm=llm, spec=spec, requests=requests, verbose=verbose, **kwargs\n        )\n        return cls(nla_tools=http_operation_tools)\n[docs]    @classmethod\n    def from_llm_and_url(\n        cls,\n        llm: BaseLanguageModel,\n        open_api_url: str,\n        requests: Optional[Requests] = None,\n        verbose: bool = False,\n        **kwargs: Any,\n    ) -> NLAToolkit:\n        \"\"\"Instantiate the toolkit from an OpenAPI Spec URL\"\"\"\n        spec = OpenAPISpec.from_url(open_api_url)\n        return cls.from_llm_and_spec(\n            llm=llm, spec=spec, requests=requests, verbose=verbose, **kwargs\n        )\n[docs]    @classmethod\n    def from_llm_and_ai_plugin(\n        cls,\n        llm: BaseLanguageModel,\n        ai_plugin: AIPlugin,\n        requests: Optional[Requests] = None,\n        verbose: bool = False,\n        **kwargs: Any,\n    ) -> NLAToolkit:\n        \"\"\"Instantiate the toolkit from an OpenAPI Spec URL\"\"\"\n        spec = OpenAPISpec.from_url(ai_plugin.api.url)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/nla/toolkit.html"}539{"id": "e3755439334b-2", "text": "spec = OpenAPISpec.from_url(ai_plugin.api.url)\n        # TODO: Merge optional Auth information with the `requests` argument\n        return cls.from_llm_and_spec(\n            llm=llm,\n            spec=spec,\n            requests=requests,\n            verbose=verbose,\n            **kwargs,\n        )\n[docs]    @classmethod\n    def from_llm_and_ai_plugin_url(\n        cls,\n        llm: BaseLanguageModel,\n        ai_plugin_url: str,\n        requests: Optional[Requests] = None,\n        verbose: bool = False,\n        **kwargs: Any,\n    ) -> NLAToolkit:\n        \"\"\"Instantiate the toolkit from an OpenAPI Spec URL\"\"\"\n        plugin = AIPlugin.from_url(ai_plugin_url)\n        return cls.from_llm_and_ai_plugin(\n            llm=llm, ai_plugin=plugin, requests=requests, verbose=verbose, **kwargs\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/nla/toolkit.html"}540{"id": "42adfb4fa851-0", "text": "Source code for langchain.agents.agent_toolkits.python.base\n\"\"\"Python agent.\"\"\"\nfrom typing import Any, Dict, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.python.prompt import PREFIX\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.tools.python.tool import PythonREPLTool\n[docs]def create_python_agent(\n    llm: BaseLanguageModel,\n    tool: PythonREPLTool,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    verbose: bool = False,\n    prefix: str = PREFIX,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a python agent from an LLM and tool.\"\"\"\n    tools = [tool]\n    prompt = ZeroShotAgent.create_prompt(tools, prefix=prefix)\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/python/base.html"}541{"id": "e18caa3c811a-0", "text": "Source code for langchain.agents.agent_toolkits.zapier.toolkit\n\"\"\"Zapier Toolkit.\"\"\"\nfrom typing import List\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools import BaseTool\nfrom langchain.tools.zapier.tool import ZapierNLARunAction\nfrom langchain.utilities.zapier import ZapierNLAWrapper\n[docs]class ZapierToolkit(BaseToolkit):\n    \"\"\"Zapier Toolkit.\"\"\"\n    tools: List[BaseTool] = []\n[docs]    @classmethod\n    def from_zapier_nla_wrapper(\n        cls, zapier_nla_wrapper: ZapierNLAWrapper\n    ) -> \"ZapierToolkit\":\n        \"\"\"Create a toolkit from a ZapierNLAWrapper.\"\"\"\n        actions = zapier_nla_wrapper.list()\n        tools = [\n            ZapierNLARunAction(\n                action_id=action[\"id\"],\n                zapier_description=action[\"description\"],\n                params_schema=action[\"params\"],\n                api_wrapper=zapier_nla_wrapper,\n            )\n            for action in actions\n        ]\n        return cls(tools=tools)\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return self.tools\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/zapier/toolkit.html"}542{"id": "d5189c69669b-0", "text": "Source code for langchain.agents.agent_toolkits.azure_cognitive_services.toolkit\nfrom __future__ import annotations\nimport sys\nfrom typing import List\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools.azure_cognitive_services import (\n    AzureCogsFormRecognizerTool,\n    AzureCogsImageAnalysisTool,\n    AzureCogsSpeech2TextTool,\n    AzureCogsText2SpeechTool,\n)\nfrom langchain.tools.base import BaseTool\n[docs]class AzureCognitiveServicesToolkit(BaseToolkit):\n    \"\"\"Toolkit for Azure Cognitive Services.\"\"\"\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        tools = [\n            AzureCogsFormRecognizerTool(),\n            AzureCogsSpeech2TextTool(),\n            AzureCogsText2SpeechTool(),\n        ]\n        # TODO: Remove check once azure-ai-vision supports MacOS.\n        if sys.platform.startswith(\"linux\") or sys.platform.startswith(\"win\"):\n            tools.append(AzureCogsImageAnalysisTool())\n        return tools\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/azure_cognitive_services/toolkit.html"}543{"id": "b3e3a0cc1255-0", "text": "Source code for langchain.agents.agent_toolkits.file_management.toolkit\n\"\"\"Toolkit for interacting with the local filesystem.\"\"\"\nfrom __future__ import annotations\nfrom typing import List, Optional\nfrom pydantic import root_validator\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools import BaseTool\nfrom langchain.tools.file_management.copy import CopyFileTool\nfrom langchain.tools.file_management.delete import DeleteFileTool\nfrom langchain.tools.file_management.file_search import FileSearchTool\nfrom langchain.tools.file_management.list_dir import ListDirectoryTool\nfrom langchain.tools.file_management.move import MoveFileTool\nfrom langchain.tools.file_management.read import ReadFileTool\nfrom langchain.tools.file_management.write import WriteFileTool\n_FILE_TOOLS = {\n    tool_cls.__fields__[\"name\"].default: tool_cls\n    for tool_cls in [\n        CopyFileTool,\n        DeleteFileTool,\n        FileSearchTool,\n        MoveFileTool,\n        ReadFileTool,\n        WriteFileTool,\n        ListDirectoryTool,\n    ]\n}\n[docs]class FileManagementToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with a Local Files.\"\"\"\n    root_dir: Optional[str] = None\n    \"\"\"If specified, all file operations are made relative to root_dir.\"\"\"\n    selected_tools: Optional[List[str]] = None\n    \"\"\"If provided, only provide the selected tools. Defaults to all.\"\"\"\n    @root_validator\n    def validate_tools(cls, values: dict) -> dict:\n        selected_tools = values.get(\"selected_tools\") or []\n        for tool_name in selected_tools:\n            if tool_name not in _FILE_TOOLS:\n                raise ValueError(\n                    f\"File Tool of name {tool_name} not supported.\"\n                    f\" Permitted tools: {list(_FILE_TOOLS)}\"\n                )\n        return values", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/file_management/toolkit.html"}544{"id": "b3e3a0cc1255-1", "text": ")\n        return values\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        allowed_tools = self.selected_tools or _FILE_TOOLS.keys()\n        tools: List[BaseTool] = []\n        for tool in allowed_tools:\n            tool_cls = _FILE_TOOLS[tool]\n            tools.append(tool_cls(root_dir=self.root_dir))  # type: ignore\n        return tools\n__all__ = [\"FileManagementToolkit\"]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/file_management/toolkit.html"}545{"id": "8e2f4ea02755-0", "text": "Source code for langchain.agents.agent_toolkits.playwright.toolkit\n\"\"\"Playwright web browser toolkit.\"\"\"\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, List, Optional, Type, cast\nfrom pydantic import Extra, root_validator\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools.base import BaseTool\nfrom langchain.tools.playwright.base import (\n    BaseBrowserTool,\n    lazy_import_playwright_browsers,\n)\nfrom langchain.tools.playwright.click import ClickTool\nfrom langchain.tools.playwright.current_page import CurrentWebPageTool\nfrom langchain.tools.playwright.extract_hyperlinks import ExtractHyperlinksTool\nfrom langchain.tools.playwright.extract_text import ExtractTextTool\nfrom langchain.tools.playwright.get_elements import GetElementsTool\nfrom langchain.tools.playwright.navigate import NavigateTool\nfrom langchain.tools.playwright.navigate_back import NavigateBackTool\nif TYPE_CHECKING:\n    from playwright.async_api import Browser as AsyncBrowser\n    from playwright.sync_api import Browser as SyncBrowser\nelse:\n    try:\n        # We do this so pydantic can resolve the types when instantiating\n        from playwright.async_api import Browser as AsyncBrowser\n        from playwright.sync_api import Browser as SyncBrowser\n    except ImportError:\n        pass\n[docs]class PlayWrightBrowserToolkit(BaseToolkit):\n    \"\"\"Toolkit for web browser tools.\"\"\"\n    sync_browser: Optional[\"SyncBrowser\"] = None\n    async_browser: Optional[\"AsyncBrowser\"] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator\n    def validate_imports_and_browser_provided(cls, values: dict) -> dict:\n        \"\"\"Check that the arguments are valid.\"\"\"\n        lazy_import_playwright_browsers()", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/playwright/toolkit.html"}546{"id": "8e2f4ea02755-1", "text": "\"\"\"Check that the arguments are valid.\"\"\"\n        lazy_import_playwright_browsers()\n        if values.get(\"async_browser\") is None and values.get(\"sync_browser\") is None:\n            raise ValueError(\"Either async_browser or sync_browser must be specified.\")\n        return values\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        tool_classes: List[Type[BaseBrowserTool]] = [\n            ClickTool,\n            NavigateTool,\n            NavigateBackTool,\n            ExtractTextTool,\n            ExtractHyperlinksTool,\n            GetElementsTool,\n            CurrentWebPageTool,\n        ]\n        tools = [\n            tool_cls.from_browser(\n                sync_browser=self.sync_browser, async_browser=self.async_browser\n            )\n            for tool_cls in tool_classes\n        ]\n        return cast(List[BaseTool], tools)\n[docs]    @classmethod\n    def from_browser(\n        cls,\n        sync_browser: Optional[SyncBrowser] = None,\n        async_browser: Optional[AsyncBrowser] = None,\n    ) -> PlayWrightBrowserToolkit:\n        \"\"\"Instantiate the toolkit.\"\"\"\n        # This is to raise a better error than the forward ref ones Pydantic would have\n        lazy_import_playwright_browsers()\n        return cls(sync_browser=sync_browser, async_browser=async_browser)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/playwright/toolkit.html"}547{"id": "d033aa0fd063-0", "text": "Source code for langchain.agents.agent_toolkits.openapi.base\n\"\"\"OpenAPI spec agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.openapi.prompt import (\n    OPENAPI_PREFIX,\n    OPENAPI_SUFFIX,\n)\nfrom langchain.agents.agent_toolkits.openapi.toolkit import OpenAPIToolkit\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\n[docs]def create_openapi_agent(\n    llm: BaseLanguageModel,\n    toolkit: OpenAPIToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = OPENAPI_PREFIX,\n    suffix: str = OPENAPI_SUFFIX,\n    format_instructions: str = FORMAT_INSTRUCTIONS,\n    input_variables: Optional[List[str]] = None,\n    max_iterations: Optional[int] = 15,\n    max_execution_time: Optional[float] = None,\n    early_stopping_method: str = \"force\",\n    verbose: bool = False,\n    return_intermediate_steps: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a json agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prompt = ZeroShotAgent.create_prompt(\n        tools,\n        prefix=prefix,\n        suffix=suffix,\n        format_instructions=format_instructions,\n        input_variables=input_variables,\n    )\n    llm_chain = LLMChain(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/openapi/base.html"}548{"id": "d033aa0fd063-1", "text": "input_variables=input_variables,\n    )\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        return_intermediate_steps=return_intermediate_steps,\n        max_iterations=max_iterations,\n        max_execution_time=max_execution_time,\n        early_stopping_method=early_stopping_method,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/openapi/base.html"}549{"id": "78b131a6a35b-0", "text": "Source code for langchain.agents.agent_toolkits.openapi.toolkit\n\"\"\"Requests toolkit.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, List\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.agents.agent_toolkits.json.base import create_json_agent\nfrom langchain.agents.agent_toolkits.json.toolkit import JsonToolkit\nfrom langchain.agents.agent_toolkits.openapi.prompt import DESCRIPTION\nfrom langchain.agents.tools import Tool\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.requests import TextRequestsWrapper\nfrom langchain.tools import BaseTool\nfrom langchain.tools.json.tool import JsonSpec\nfrom langchain.tools.requests.tool import (\n    RequestsDeleteTool,\n    RequestsGetTool,\n    RequestsPatchTool,\n    RequestsPostTool,\n    RequestsPutTool,\n)\nclass RequestsToolkit(BaseToolkit):\n    \"\"\"Toolkit for making requests.\"\"\"\n    requests_wrapper: TextRequestsWrapper\n    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Return a list of tools.\"\"\"\n        return [\n            RequestsGetTool(requests_wrapper=self.requests_wrapper),\n            RequestsPostTool(requests_wrapper=self.requests_wrapper),\n            RequestsPatchTool(requests_wrapper=self.requests_wrapper),\n            RequestsPutTool(requests_wrapper=self.requests_wrapper),\n            RequestsDeleteTool(requests_wrapper=self.requests_wrapper),\n        ]\n[docs]class OpenAPIToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with a OpenAPI api.\"\"\"\n    json_agent: AgentExecutor\n    requests_wrapper: TextRequestsWrapper\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        json_agent_tool = Tool(\n            name=\"json_explorer\",\n            func=self.json_agent.run,\n            description=DESCRIPTION,\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/openapi/toolkit.html"}550{"id": "78b131a6a35b-1", "text": "func=self.json_agent.run,\n            description=DESCRIPTION,\n        )\n        request_toolkit = RequestsToolkit(requests_wrapper=self.requests_wrapper)\n        return [*request_toolkit.get_tools(), json_agent_tool]\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        json_spec: JsonSpec,\n        requests_wrapper: TextRequestsWrapper,\n        **kwargs: Any,\n    ) -> OpenAPIToolkit:\n        \"\"\"Create json agent from llm, then initialize.\"\"\"\n        json_agent = create_json_agent(llm, JsonToolkit(spec=json_spec), **kwargs)\n        return cls(json_agent=json_agent, requests_wrapper=requests_wrapper)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/openapi/toolkit.html"}551{"id": "1839072ac2e5-0", "text": "Source code for langchain.agents.agent_toolkits.json.base\n\"\"\"Json agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.json.prompt import JSON_PREFIX, JSON_SUFFIX\nfrom langchain.agents.agent_toolkits.json.toolkit import JsonToolkit\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\n[docs]def create_json_agent(\n    llm: BaseLanguageModel,\n    toolkit: JsonToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = JSON_PREFIX,\n    suffix: str = JSON_SUFFIX,\n    format_instructions: str = FORMAT_INSTRUCTIONS,\n    input_variables: Optional[List[str]] = None,\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a json agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prompt = ZeroShotAgent.create_prompt(\n        tools,\n        prefix=prefix,\n        suffix=suffix,\n        format_instructions=format_instructions,\n        input_variables=input_variables,\n    )\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/json/base.html"}552{"id": "1839072ac2e5-1", "text": "return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/json/base.html"}553{"id": "25ac0edb2a41-0", "text": "Source code for langchain.agents.agent_toolkits.json.toolkit\n\"\"\"Toolkit for interacting with a JSON spec.\"\"\"\nfrom __future__ import annotations\nfrom typing import List\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools import BaseTool\nfrom langchain.tools.json.tool import JsonGetValueTool, JsonListKeysTool, JsonSpec\n[docs]class JsonToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with a JSON spec.\"\"\"\n    spec: JsonSpec\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return [\n            JsonListKeysTool(spec=self.spec),\n            JsonGetValueTool(spec=self.spec),\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/json/toolkit.html"}554{"id": "b19b45e9f510-0", "text": "Source code for langchain.agents.agent_toolkits.jira.toolkit\n\"\"\"Jira Toolkit.\"\"\"\nfrom typing import List\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools import BaseTool\nfrom langchain.tools.jira.tool import JiraAction\nfrom langchain.utilities.jira import JiraAPIWrapper\n[docs]class JiraToolkit(BaseToolkit):\n    \"\"\"Jira Toolkit.\"\"\"\n    tools: List[BaseTool] = []\n[docs]    @classmethod\n    def from_jira_api_wrapper(cls, jira_api_wrapper: JiraAPIWrapper) -> \"JiraToolkit\":\n        actions = jira_api_wrapper.list()\n        tools = [\n            JiraAction(\n                name=action[\"name\"],\n                description=action[\"description\"],\n                mode=action[\"mode\"],\n                api_wrapper=jira_api_wrapper,\n            )\n            for action in actions\n        ]\n        return cls(tools=tools)\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return self.tools\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/jira/toolkit.html"}555{"id": "09eb2fc59200-0", "text": "Source code for langchain.agents.agent_toolkits.powerbi.chat_base\n\"\"\"Power BI agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents import AgentExecutor\nfrom langchain.agents.agent import AgentOutputParser\nfrom langchain.agents.agent_toolkits.powerbi.prompt import (\n    POWERBI_CHAT_PREFIX,\n    POWERBI_CHAT_SUFFIX,\n)\nfrom langchain.agents.agent_toolkits.powerbi.toolkit import PowerBIToolkit\nfrom langchain.agents.conversational_chat.base import ConversationalChatAgent\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.memory import ConversationBufferMemory\nfrom langchain.memory.chat_memory import BaseChatMemory\nfrom langchain.utilities.powerbi import PowerBIDataset\n[docs]def create_pbi_chat_agent(\n    llm: BaseChatModel,\n    toolkit: Optional[PowerBIToolkit],\n    powerbi: Optional[PowerBIDataset] = None,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    output_parser: Optional[AgentOutputParser] = None,\n    prefix: str = POWERBI_CHAT_PREFIX,\n    suffix: str = POWERBI_CHAT_SUFFIX,\n    examples: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    memory: Optional[BaseChatMemory] = None,\n    top_k: int = 10,\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a pbi agent from an Chat LLM and tools.\n    If you supply only a toolkit and no powerbi dataset, the same LLM is used for both.\n    \"\"\"\n    if toolkit is None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/chat_base.html"}556{"id": "09eb2fc59200-1", "text": "\"\"\"\n    if toolkit is None:\n        if powerbi is None:\n            raise ValueError(\"Must provide either a toolkit or powerbi dataset\")\n        toolkit = PowerBIToolkit(powerbi=powerbi, llm=llm, examples=examples)\n    tools = toolkit.get_tools()\n    agent = ConversationalChatAgent.from_llm_and_tools(\n        llm=llm,\n        tools=tools,\n        system_message=prefix.format(top_k=top_k),\n        human_message=suffix,\n        input_variables=input_variables,\n        callback_manager=callback_manager,\n        output_parser=output_parser,\n        verbose=verbose,\n        **kwargs,\n    )\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        memory=memory\n        or ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True),\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/chat_base.html"}557{"id": "23953858bb91-0", "text": "Source code for langchain.agents.agent_toolkits.powerbi.base\n\"\"\"Power BI agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents import AgentExecutor\nfrom langchain.agents.agent_toolkits.powerbi.prompt import (\n    POWERBI_PREFIX,\n    POWERBI_SUFFIX,\n)\nfrom langchain.agents.agent_toolkits.powerbi.toolkit import PowerBIToolkit\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.utilities.powerbi import PowerBIDataset\n[docs]def create_pbi_agent(\n    llm: BaseLanguageModel,\n    toolkit: Optional[PowerBIToolkit],\n    powerbi: Optional[PowerBIDataset] = None,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = POWERBI_PREFIX,\n    suffix: str = POWERBI_SUFFIX,\n    format_instructions: str = FORMAT_INSTRUCTIONS,\n    examples: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    top_k: int = 10,\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a pbi agent from an LLM and tools.\"\"\"\n    if toolkit is None:\n        if powerbi is None:\n            raise ValueError(\"Must provide either a toolkit or powerbi dataset\")\n        toolkit = PowerBIToolkit(powerbi=powerbi, llm=llm, examples=examples)\n    tools = toolkit.get_tools()", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/base.html"}558{"id": "23953858bb91-1", "text": "tools = toolkit.get_tools()\n    agent = ZeroShotAgent(\n        llm_chain=LLMChain(\n            llm=llm,\n            prompt=ZeroShotAgent.create_prompt(\n                tools,\n                prefix=prefix.format(top_k=top_k),\n                suffix=suffix,\n                format_instructions=format_instructions,\n                input_variables=input_variables,\n            ),\n            callback_manager=callback_manager,  # type: ignore\n            verbose=verbose,\n        ),\n        allowed_tools=[tool.name for tool in tools],\n        **kwargs,\n    )\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/base.html"}559{"id": "6e7fb74be7bc-0", "text": "Source code for langchain.agents.agent_toolkits.powerbi.toolkit\n\"\"\"Toolkit for interacting with a Power BI dataset.\"\"\"\nfrom typing import List, Optional\nfrom pydantic import Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain.tools import BaseTool\nfrom langchain.tools.powerbi.prompt import QUESTION_TO_QUERY\nfrom langchain.tools.powerbi.tool import (\n    InfoPowerBITool,\n    ListPowerBITool,\n    QueryPowerBITool,\n)\nfrom langchain.utilities.powerbi import PowerBIDataset\n[docs]class PowerBIToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with PowerBI dataset.\"\"\"\n    powerbi: PowerBIDataset = Field(exclude=True)\n    llm: BaseLanguageModel = Field(exclude=True)\n    examples: Optional[str] = None\n    max_iterations: int = 5\n    callback_manager: Optional[BaseCallbackManager] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        if self.callback_manager:\n            chain = LLMChain(\n                llm=self.llm,\n                callback_manager=self.callback_manager,\n                prompt=PromptTemplate(\n                    template=QUESTION_TO_QUERY,\n                    input_variables=[\"tool_input\", \"tables\", \"schemas\", \"examples\"],\n                ),\n            )\n        else:\n            chain = LLMChain(\n                llm=self.llm,\n                prompt=PromptTemplate(\n                    template=QUESTION_TO_QUERY,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/toolkit.html"}560{"id": "6e7fb74be7bc-1", "text": "prompt=PromptTemplate(\n                    template=QUESTION_TO_QUERY,\n                    input_variables=[\"tool_input\", \"tables\", \"schemas\", \"examples\"],\n                ),\n            )\n        return [\n            QueryPowerBITool(\n                llm_chain=chain,\n                powerbi=self.powerbi,\n                examples=self.examples,\n                max_iterations=self.max_iterations,\n            ),\n            InfoPowerBITool(powerbi=self.powerbi),\n            ListPowerBITool(powerbi=self.powerbi),\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/powerbi/toolkit.html"}561{"id": "10fcedf3c3f4-0", "text": "Source code for langchain.agents.agent_toolkits.csv.base\n\"\"\"Agent for working with csvs.\"\"\"\nfrom typing import Any, List, Optional, Union\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\nfrom langchain.base_language import BaseLanguageModel\n[docs]def create_csv_agent(\n    llm: BaseLanguageModel,\n    path: Union[str, List[str]],\n    pandas_kwargs: Optional[dict] = None,\n    **kwargs: Any,\n) -> AgentExecutor:\n    \"\"\"Create csv agent by loading to a dataframe and using pandas agent.\"\"\"\n    try:\n        import pandas as pd\n    except ImportError:\n        raise ValueError(\n            \"pandas package not found, please install with `pip install pandas`\"\n        )\n    _kwargs = pandas_kwargs or {}\n    if isinstance(path, str):\n        df = pd.read_csv(path, **_kwargs)\n    elif isinstance(path, list):\n        df = []\n        for item in path:\n            if not isinstance(item, str):\n                raise ValueError(f\"Expected str, got {type(path)}\")\n            df.append(pd.read_csv(item, **_kwargs))\n    else:\n        raise ValueError(f\"Expected str or list, got {type(path)}\")\n    return create_pandas_dataframe_agent(llm, df, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/csv/base.html"}562{"id": "47773f249872-0", "text": "Source code for langchain.agents.agent_toolkits.gmail.toolkit\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, List\nfrom pydantic import Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.tools import BaseTool\nfrom langchain.tools.gmail.create_draft import GmailCreateDraft\nfrom langchain.tools.gmail.get_message import GmailGetMessage\nfrom langchain.tools.gmail.get_thread import GmailGetThread\nfrom langchain.tools.gmail.search import GmailSearch\nfrom langchain.tools.gmail.send_message import GmailSendMessage\nfrom langchain.tools.gmail.utils import build_resource_service\nif TYPE_CHECKING:\n    # This is for linting and IDE typehints\n    from googleapiclient.discovery import Resource\nelse:\n    try:\n        # We do this so pydantic can resolve the types when instantiating\n        from googleapiclient.discovery import Resource\n    except ImportError:\n        pass\nSCOPES = [\"https://mail.google.com/\"]\n[docs]class GmailToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with Gmail.\"\"\"\n    api_resource: Resource = Field(default_factory=build_resource_service)\n    class Config:\n        \"\"\"Pydantic config.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return [\n            GmailCreateDraft(api_resource=self.api_resource),\n            GmailSendMessage(api_resource=self.api_resource),\n            GmailSearch(api_resource=self.api_resource),\n            GmailGetMessage(api_resource=self.api_resource),\n            GmailGetThread(api_resource=self.api_resource),\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/gmail/toolkit.html"}563{"id": "740ad6575d99-0", "text": "Source code for langchain.agents.agent_toolkits.spark.base\n\"\"\"Agent for working with pandas objects.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.spark.prompt import PREFIX, SUFFIX\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.llms.base import BaseLLM\nfrom langchain.tools.python.tool import PythonAstREPLTool\ndef _validate_spark_df(df: Any) -> bool:\n    try:\n        from pyspark.sql import DataFrame as SparkLocalDataFrame\n        return isinstance(df, SparkLocalDataFrame)\n    except ImportError:\n        return False\ndef _validate_spark_connect_df(df: Any) -> bool:\n    try:\n        from pyspark.sql.connect.dataframe import DataFrame as SparkConnectDataFrame\n        return isinstance(df, SparkConnectDataFrame)\n    except ImportError:\n        return False\n[docs]def create_spark_dataframe_agent(\n    llm: BaseLLM,\n    df: Any,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = PREFIX,\n    suffix: str = SUFFIX,\n    input_variables: Optional[List[str]] = None,\n    verbose: bool = False,\n    return_intermediate_steps: bool = False,\n    max_iterations: Optional[int] = 15,\n    max_execution_time: Optional[float] = None,\n    early_stopping_method: str = \"force\",\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a spark agent from an LLM and dataframe.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/spark/base.html"}564{"id": "740ad6575d99-1", "text": ") -> AgentExecutor:\n    \"\"\"Construct a spark agent from an LLM and dataframe.\"\"\"\n    if not _validate_spark_df(df) and not _validate_spark_connect_df(df):\n        raise ValueError(\"Spark is not installed. run `pip install pyspark`.\")\n    if input_variables is None:\n        input_variables = [\"df\", \"input\", \"agent_scratchpad\"]\n    tools = [PythonAstREPLTool(locals={\"df\": df})]\n    prompt = ZeroShotAgent.create_prompt(\n        tools, prefix=prefix, suffix=suffix, input_variables=input_variables\n    )\n    partial_prompt = prompt.partial(df=str(df.first()))\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=partial_prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(\n        llm_chain=llm_chain,\n        allowed_tools=tool_names,\n        callback_manager=callback_manager,\n        **kwargs,\n    )\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        return_intermediate_steps=return_intermediate_steps,\n        max_iterations=max_iterations,\n        max_execution_time=max_execution_time,\n        early_stopping_method=early_stopping_method,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/spark/base.html"}565{"id": "bcd42c26e7cd-0", "text": "Source code for langchain.agents.agent_toolkits.pandas.base\n\"\"\"Agent for working with pandas objects.\"\"\"\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.pandas.prompt import (\n    MULTI_DF_PREFIX,\n    PREFIX,\n    SUFFIX_NO_DF,\n    SUFFIX_WITH_DF,\n    SUFFIX_WITH_MULTI_DF,\n)\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.tools.python.tool import PythonAstREPLTool\ndef _get_multi_prompt(\n    dfs: List[Any],\n    prefix: Optional[str] = None,\n    suffix: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    include_df_in_prompt: Optional[bool] = True,\n) -> Tuple[BasePromptTemplate, List[PythonAstREPLTool]]:\n    num_dfs = len(dfs)\n    if suffix is not None:\n        suffix_to_use = suffix\n        include_dfs_head = True\n    elif include_df_in_prompt:\n        suffix_to_use = SUFFIX_WITH_MULTI_DF\n        include_dfs_head = True\n    else:\n        suffix_to_use = SUFFIX_NO_DF\n        include_dfs_head = False\n    if input_variables is None:\n        input_variables = [\"input\", \"agent_scratchpad\", \"num_dfs\"]\n        if include_dfs_head:\n            input_variables += [\"dfs_head\"]\n    if prefix is None:\n        prefix = MULTI_DF_PREFIX\n    df_locals = {}\n    for i, dataframe in enumerate(dfs):", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/pandas/base.html"}566{"id": "bcd42c26e7cd-1", "text": "df_locals = {}\n    for i, dataframe in enumerate(dfs):\n        df_locals[f\"df{i + 1}\"] = dataframe\n    tools = [PythonAstREPLTool(locals=df_locals)]\n    prompt = ZeroShotAgent.create_prompt(\n        tools, prefix=prefix, suffix=suffix_to_use, input_variables=input_variables\n    )\n    partial_prompt = prompt.partial()\n    if \"dfs_head\" in input_variables:\n        dfs_head = \"\\n\\n\".join([d.head().to_markdown() for d in dfs])\n        partial_prompt = partial_prompt.partial(num_dfs=str(num_dfs), dfs_head=dfs_head)\n    if \"num_dfs\" in input_variables:\n        partial_prompt = partial_prompt.partial(num_dfs=str(num_dfs))\n    return partial_prompt, tools\ndef _get_single_prompt(\n    df: Any,\n    prefix: Optional[str] = None,\n    suffix: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    include_df_in_prompt: Optional[bool] = True,\n) -> Tuple[BasePromptTemplate, List[PythonAstREPLTool]]:\n    if suffix is not None:\n        suffix_to_use = suffix\n        include_df_head = True\n    elif include_df_in_prompt:\n        suffix_to_use = SUFFIX_WITH_DF\n        include_df_head = True\n    else:\n        suffix_to_use = SUFFIX_NO_DF\n        include_df_head = False\n    if input_variables is None:\n        input_variables = [\"input\", \"agent_scratchpad\"]\n        if include_df_head:\n            input_variables += [\"df_head\"]\n    if prefix is None:\n        prefix = PREFIX\n    tools = [PythonAstREPLTool(locals={\"df\": df})]\n    prompt = ZeroShotAgent.create_prompt(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/pandas/base.html"}567{"id": "bcd42c26e7cd-2", "text": "prompt = ZeroShotAgent.create_prompt(\n        tools, prefix=prefix, suffix=suffix_to_use, input_variables=input_variables\n    )\n    partial_prompt = prompt.partial()\n    if \"df_head\" in input_variables:\n        partial_prompt = partial_prompt.partial(df_head=str(df.head().to_markdown()))\n    return partial_prompt, tools\ndef _get_prompt_and_tools(\n    df: Any,\n    prefix: Optional[str] = None,\n    suffix: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    include_df_in_prompt: Optional[bool] = True,\n) -> Tuple[BasePromptTemplate, List[PythonAstREPLTool]]:\n    try:\n        import pandas as pd\n    except ImportError:\n        raise ValueError(\n            \"pandas package not found, please install with `pip install pandas`\"\n        )\n    if include_df_in_prompt is not None and suffix is not None:\n        raise ValueError(\"If suffix is specified, include_df_in_prompt should not be.\")\n    if isinstance(df, list):\n        for item in df:\n            if not isinstance(item, pd.DataFrame):\n                raise ValueError(f\"Expected pandas object, got {type(df)}\")\n        return _get_multi_prompt(\n            df,\n            prefix=prefix,\n            suffix=suffix,\n            input_variables=input_variables,\n            include_df_in_prompt=include_df_in_prompt,\n        )\n    else:\n        if not isinstance(df, pd.DataFrame):\n            raise ValueError(f\"Expected pandas object, got {type(df)}\")\n        return _get_single_prompt(\n            df,\n            prefix=prefix,\n            suffix=suffix,\n            input_variables=input_variables,\n            include_df_in_prompt=include_df_in_prompt,\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/pandas/base.html"}568{"id": "bcd42c26e7cd-3", "text": "include_df_in_prompt=include_df_in_prompt,\n        )\n[docs]def create_pandas_dataframe_agent(\n    llm: BaseLanguageModel,\n    df: Any,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: Optional[str] = None,\n    suffix: Optional[str] = None,\n    input_variables: Optional[List[str]] = None,\n    verbose: bool = False,\n    return_intermediate_steps: bool = False,\n    max_iterations: Optional[int] = 15,\n    max_execution_time: Optional[float] = None,\n    early_stopping_method: str = \"force\",\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    include_df_in_prompt: Optional[bool] = True,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a pandas agent from an LLM and dataframe.\"\"\"\n    prompt, tools = _get_prompt_and_tools(\n        df,\n        prefix=prefix,\n        suffix=suffix,\n        input_variables=input_variables,\n        include_df_in_prompt=include_df_in_prompt,\n    )\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(\n        llm_chain=llm_chain,\n        allowed_tools=tool_names,\n        callback_manager=callback_manager,\n        **kwargs,\n    )\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        return_intermediate_steps=return_intermediate_steps,\n        max_iterations=max_iterations,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/pandas/base.html"}569{"id": "bcd42c26e7cd-4", "text": "return_intermediate_steps=return_intermediate_steps,\n        max_iterations=max_iterations,\n        max_execution_time=max_execution_time,\n        early_stopping_method=early_stopping_method,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/pandas/base.html"}570{"id": "ec7938bd52c1-0", "text": "Source code for langchain.agents.agent_toolkits.vectorstore.base\n\"\"\"VectorStore agent.\"\"\"\nfrom typing import Any, Dict, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.vectorstore.prompt import PREFIX, ROUTER_PREFIX\nfrom langchain.agents.agent_toolkits.vectorstore.toolkit import (\n    VectorStoreRouterToolkit,\n    VectorStoreToolkit,\n)\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\n[docs]def create_vectorstore_agent(\n    llm: BaseLanguageModel,\n    toolkit: VectorStoreToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = PREFIX,\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a vectorstore agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prompt = ZeroShotAgent.create_prompt(tools, prefix=prefix)\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\n[docs]def create_vectorstore_router_agent(", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/vectorstore/base.html"}571{"id": "ec7938bd52c1-1", "text": ")\n[docs]def create_vectorstore_router_agent(\n    llm: BaseLanguageModel,\n    toolkit: VectorStoreRouterToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = ROUTER_PREFIX,\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a vectorstore router agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prompt = ZeroShotAgent.create_prompt(tools, prefix=prefix)\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/vectorstore/base.html"}572{"id": "1fdb897ab45e-0", "text": "Source code for langchain.agents.agent_toolkits.vectorstore.toolkit\n\"\"\"Toolkit for interacting with a vector store.\"\"\"\nfrom typing import List\nfrom pydantic import BaseModel, Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.llms.openai import OpenAI\nfrom langchain.tools import BaseTool\nfrom langchain.tools.vectorstore.tool import (\n    VectorStoreQATool,\n    VectorStoreQAWithSourcesTool,\n)\nfrom langchain.vectorstores.base import VectorStore\n[docs]class VectorStoreInfo(BaseModel):\n    \"\"\"Information about a vectorstore.\"\"\"\n    vectorstore: VectorStore = Field(exclude=True)\n    name: str\n    description: str\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]class VectorStoreToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with a vector store.\"\"\"\n    vectorstore_info: VectorStoreInfo = Field(exclude=True)\n    llm: BaseLanguageModel = Field(default_factory=lambda: OpenAI(temperature=0))\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        description = VectorStoreQATool.get_description(\n            self.vectorstore_info.name, self.vectorstore_info.description\n        )\n        qa_tool = VectorStoreQATool(\n            name=self.vectorstore_info.name,\n            description=description,\n            vectorstore=self.vectorstore_info.vectorstore,\n            llm=self.llm,\n        )\n        description = VectorStoreQAWithSourcesTool.get_description(\n            self.vectorstore_info.name, self.vectorstore_info.description\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/vectorstore/toolkit.html"}573{"id": "1fdb897ab45e-1", "text": "self.vectorstore_info.name, self.vectorstore_info.description\n        )\n        qa_with_sources_tool = VectorStoreQAWithSourcesTool(\n            name=f\"{self.vectorstore_info.name}_with_sources\",\n            description=description,\n            vectorstore=self.vectorstore_info.vectorstore,\n            llm=self.llm,\n        )\n        return [qa_tool, qa_with_sources_tool]\n[docs]class VectorStoreRouterToolkit(BaseToolkit):\n    \"\"\"Toolkit for routing between vectorstores.\"\"\"\n    vectorstores: List[VectorStoreInfo] = Field(exclude=True)\n    llm: BaseLanguageModel = Field(default_factory=lambda: OpenAI(temperature=0))\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        tools: List[BaseTool] = []\n        for vectorstore_info in self.vectorstores:\n            description = VectorStoreQATool.get_description(\n                vectorstore_info.name, vectorstore_info.description\n            )\n            qa_tool = VectorStoreQATool(\n                name=vectorstore_info.name,\n                description=description,\n                vectorstore=vectorstore_info.vectorstore,\n                llm=self.llm,\n            )\n            tools.append(qa_tool)\n        return tools\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/vectorstore/toolkit.html"}574{"id": "af387847ea1c-0", "text": "Source code for langchain.agents.agent_toolkits.sql.base\n\"\"\"SQL agent.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom langchain.agents.agent import AgentExecutor\nfrom langchain.agents.agent_toolkits.sql.prompt import SQL_PREFIX, SQL_SUFFIX\nfrom langchain.agents.agent_toolkits.sql.toolkit import SQLDatabaseToolkit\nfrom langchain.agents.mrkl.base import ZeroShotAgent\nfrom langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains.llm import LLMChain\n[docs]def create_sql_agent(\n    llm: BaseLanguageModel,\n    toolkit: SQLDatabaseToolkit,\n    callback_manager: Optional[BaseCallbackManager] = None,\n    prefix: str = SQL_PREFIX,\n    suffix: str = SQL_SUFFIX,\n    format_instructions: str = FORMAT_INSTRUCTIONS,\n    input_variables: Optional[List[str]] = None,\n    top_k: int = 10,\n    max_iterations: Optional[int] = 15,\n    max_execution_time: Optional[float] = None,\n    early_stopping_method: str = \"force\",\n    verbose: bool = False,\n    agent_executor_kwargs: Optional[Dict[str, Any]] = None,\n    **kwargs: Dict[str, Any],\n) -> AgentExecutor:\n    \"\"\"Construct a sql agent from an LLM and tools.\"\"\"\n    tools = toolkit.get_tools()\n    prefix = prefix.format(dialect=toolkit.dialect, top_k=top_k)\n    prompt = ZeroShotAgent.create_prompt(\n        tools,\n        prefix=prefix,\n        suffix=suffix,\n        format_instructions=format_instructions,\n        input_variables=input_variables,\n    )\n    llm_chain = LLMChain(\n        llm=llm,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/sql/base.html"}575{"id": "af387847ea1c-1", "text": ")\n    llm_chain = LLMChain(\n        llm=llm,\n        prompt=prompt,\n        callback_manager=callback_manager,\n    )\n    tool_names = [tool.name for tool in tools]\n    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)\n    return AgentExecutor.from_agent_and_tools(\n        agent=agent,\n        tools=tools,\n        callback_manager=callback_manager,\n        verbose=verbose,\n        max_iterations=max_iterations,\n        max_execution_time=max_execution_time,\n        early_stopping_method=early_stopping_method,\n        **(agent_executor_kwargs or {}),\n    )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/sql/base.html"}576{"id": "729d7a297537-0", "text": "Source code for langchain.agents.agent_toolkits.sql.toolkit\n\"\"\"Toolkit for interacting with a SQL database.\"\"\"\nfrom typing import List\nfrom pydantic import Field\nfrom langchain.agents.agent_toolkits.base import BaseToolkit\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.sql_database import SQLDatabase\nfrom langchain.tools import BaseTool\nfrom langchain.tools.sql_database.tool import (\n    InfoSQLDatabaseTool,\n    ListSQLDatabaseTool,\n    QueryCheckerTool,\n    QuerySQLDataBaseTool,\n)\n[docs]class SQLDatabaseToolkit(BaseToolkit):\n    \"\"\"Toolkit for interacting with SQL databases.\"\"\"\n    db: SQLDatabase = Field(exclude=True)\n    llm: BaseLanguageModel = Field(exclude=True)\n    @property\n    def dialect(self) -> str:\n        \"\"\"Return string representation of dialect to use.\"\"\"\n        return self.db.dialect\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n[docs]    def get_tools(self) -> List[BaseTool]:\n        \"\"\"Get the tools in the toolkit.\"\"\"\n        return [\n            QuerySQLDataBaseTool(db=self.db),\n            InfoSQLDatabaseTool(db=self.db),\n            ListSQLDatabaseTool(db=self.db),\n            QueryCheckerTool(db=self.db, llm=self.llm),\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/agent_toolkits/sql/toolkit.html"}577{"id": "4e95b7e9f32f-0", "text": "Source code for langchain.agents.conversational.base\n\"\"\"An agent designed to hold a conversation in addition to using tools.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, List, Optional, Sequence\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentOutputParser\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.conversational.output_parser import ConvoOutputParser\nfrom langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS, PREFIX, SUFFIX\nfrom langchain.agents.utils import validate_tools_single_input\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.callbacks.base import BaseCallbackManager\nfrom langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain.tools.base import BaseTool\n[docs]class ConversationalAgent(Agent):\n    \"\"\"An agent designed to hold a conversation in addition to using tools.\"\"\"\n    ai_prefix: str = \"AI\"\n    output_parser: AgentOutputParser = Field(default_factory=ConvoOutputParser)\n    @classmethod\n    def _get_default_output_parser(\n        cls, ai_prefix: str = \"AI\", **kwargs: Any\n    ) -> AgentOutputParser:\n        return ConvoOutputParser(ai_prefix=ai_prefix)\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        return AgentType.CONVERSATIONAL_REACT_DESCRIPTION\n    @property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Observation: \"\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the llm call with.\"\"\"\n        return \"Thought:\"\n[docs]    @classmethod\n    def create_prompt(\n        cls,", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational/base.html"}578{"id": "4e95b7e9f32f-1", "text": "[docs]    @classmethod\n    def create_prompt(\n        cls,\n        tools: Sequence[BaseTool],\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        ai_prefix: str = \"AI\",\n        human_prefix: str = \"Human\",\n        input_variables: Optional[List[str]] = None,\n    ) -> PromptTemplate:\n        \"\"\"Create prompt in the style of the zero shot agent.\n        Args:\n            tools: List of tools the agent will have access to, used to format the\n                prompt.\n            prefix: String to put before the list of tools.\n            suffix: String to put after the list of tools.\n            ai_prefix: String to use before AI output.\n            human_prefix: String to use before human output.\n            input_variables: List of input variables the final prompt will expect.\n        Returns:\n            A PromptTemplate with the template assembled from the pieces here.\n        \"\"\"\n        tool_strings = \"\\n\".join(\n            [f\"> {tool.name}: {tool.description}\" for tool in tools]\n        )\n        tool_names = \", \".join([tool.name for tool in tools])\n        format_instructions = format_instructions.format(\n            tool_names=tool_names, ai_prefix=ai_prefix, human_prefix=human_prefix\n        )\n        template = \"\\n\\n\".join([prefix, tool_strings, format_instructions, suffix])\n        if input_variables is None:\n            input_variables = [\"input\", \"chat_history\", \"agent_scratchpad\"]\n        return PromptTemplate(template=template, input_variables=input_variables)\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        super()._validate_tools(tools)\n        validate_tools_single_input(cls.__name__, tools)", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational/base.html"}579{"id": "4e95b7e9f32f-2", "text": "super()._validate_tools(tools)\n        validate_tools_single_input(cls.__name__, tools)\n[docs]    @classmethod\n    def from_llm_and_tools(\n        cls,\n        llm: BaseLanguageModel,\n        tools: Sequence[BaseTool],\n        callback_manager: Optional[BaseCallbackManager] = None,\n        output_parser: Optional[AgentOutputParser] = None,\n        prefix: str = PREFIX,\n        suffix: str = SUFFIX,\n        format_instructions: str = FORMAT_INSTRUCTIONS,\n        ai_prefix: str = \"AI\",\n        human_prefix: str = \"Human\",\n        input_variables: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> Agent:\n        \"\"\"Construct an agent from an LLM and tools.\"\"\"\n        cls._validate_tools(tools)\n        prompt = cls.create_prompt(\n            tools,\n            ai_prefix=ai_prefix,\n            human_prefix=human_prefix,\n            prefix=prefix,\n            suffix=suffix,\n            format_instructions=format_instructions,\n            input_variables=input_variables,\n        )\n        llm_chain = LLMChain(\n            llm=llm,\n            prompt=prompt,\n            callback_manager=callback_manager,\n        )\n        tool_names = [tool.name for tool in tools]\n        _output_parser = output_parser or cls._get_default_output_parser(\n            ai_prefix=ai_prefix\n        )\n        return cls(\n            llm_chain=llm_chain,\n            allowed_tools=tool_names,\n            ai_prefix=ai_prefix,\n            output_parser=_output_parser,\n            **kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/conversational/base.html"}580{"id": "1979855fa776-0", "text": "Source code for langchain.agents.react.base\n\"\"\"Chain that implements the ReAct paper from https://arxiv.org/pdf/2210.03629.pdf.\"\"\"\nfrom typing import Any, List, Optional, Sequence\nfrom pydantic import Field\nfrom langchain.agents.agent import Agent, AgentExecutor, AgentOutputParser\nfrom langchain.agents.agent_types import AgentType\nfrom langchain.agents.react.output_parser import ReActOutputParser\nfrom langchain.agents.react.textworld_prompt import TEXTWORLD_PROMPT\nfrom langchain.agents.react.wiki_prompt import WIKI_PROMPT\nfrom langchain.agents.tools import Tool\nfrom langchain.agents.utils import validate_tools_single_input\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.docstore.base import Docstore\nfrom langchain.docstore.document import Document\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.tools.base import BaseTool\nclass ReActDocstoreAgent(Agent):\n    \"\"\"Agent for the ReAct chain.\"\"\"\n    output_parser: AgentOutputParser = Field(default_factory=ReActOutputParser)\n    @classmethod\n    def _get_default_output_parser(cls, **kwargs: Any) -> AgentOutputParser:\n        return ReActOutputParser()\n    @property\n    def _agent_type(self) -> str:\n        \"\"\"Return Identifier of agent type.\"\"\"\n        return AgentType.REACT_DOCSTORE\n    @classmethod\n    def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:\n        \"\"\"Return default prompt.\"\"\"\n        return WIKI_PROMPT\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        validate_tools_single_input(cls.__name__, tools)\n        super()._validate_tools(tools)\n        if len(tools) != 2:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/react/base.html"}581{"id": "1979855fa776-1", "text": "super()._validate_tools(tools)\n        if len(tools) != 2:\n            raise ValueError(f\"Exactly two tools must be specified, but got {tools}\")\n        tool_names = {tool.name for tool in tools}\n        if tool_names != {\"Lookup\", \"Search\"}:\n            raise ValueError(\n                f\"Tool names should be Lookup and Search, got {tool_names}\"\n            )\n    @property\n    def observation_prefix(self) -> str:\n        \"\"\"Prefix to append the observation with.\"\"\"\n        return \"Observation: \"\n    @property\n    def _stop(self) -> List[str]:\n        return [\"\\nObservation:\"]\n    @property\n    def llm_prefix(self) -> str:\n        \"\"\"Prefix to append the LLM call with.\"\"\"\n        return \"Thought:\"\nclass DocstoreExplorer:\n    \"\"\"Class to assist with exploration of a document store.\"\"\"\n    def __init__(self, docstore: Docstore):\n        \"\"\"Initialize with a docstore, and set initial document to None.\"\"\"\n        self.docstore = docstore\n        self.document: Optional[Document] = None\n        self.lookup_str = \"\"\n        self.lookup_index = 0\n    def search(self, term: str) -> str:\n        \"\"\"Search for a term in the docstore, and if found save.\"\"\"\n        result = self.docstore.search(term)\n        if isinstance(result, Document):\n            self.document = result\n            return self._summary\n        else:\n            self.document = None\n            return result\n    def lookup(self, term: str) -> str:\n        \"\"\"Lookup a term in document (if saved).\"\"\"\n        if self.document is None:\n            raise ValueError(\"Cannot lookup without a successful search first\")\n        if term.lower() != self.lookup_str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/react/base.html"}582{"id": "1979855fa776-2", "text": "if term.lower() != self.lookup_str:\n            self.lookup_str = term.lower()\n            self.lookup_index = 0\n        else:\n            self.lookup_index += 1\n        lookups = [p for p in self._paragraphs if self.lookup_str in p.lower()]\n        if len(lookups) == 0:\n            return \"No Results\"\n        elif self.lookup_index >= len(lookups):\n            return \"No More Results\"\n        else:\n            result_prefix = f\"(Result {self.lookup_index + 1}/{len(lookups)})\"\n            return f\"{result_prefix} {lookups[self.lookup_index]}\"\n    @property\n    def _summary(self) -> str:\n        return self._paragraphs[0]\n    @property\n    def _paragraphs(self) -> List[str]:\n        if self.document is None:\n            raise ValueError(\"Cannot get paragraphs without a document\")\n        return self.document.page_content.split(\"\\n\\n\")\n[docs]class ReActTextWorldAgent(ReActDocstoreAgent):\n    \"\"\"Agent for the ReAct TextWorld chain.\"\"\"\n[docs]    @classmethod\n    def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:\n        \"\"\"Return default prompt.\"\"\"\n        return TEXTWORLD_PROMPT\n    @classmethod\n    def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:\n        validate_tools_single_input(cls.__name__, tools)\n        super()._validate_tools(tools)\n        if len(tools) != 1:\n            raise ValueError(f\"Exactly one tool must be specified, but got {tools}\")\n        tool_names = {tool.name for tool in tools}\n        if tool_names != {\"Play\"}:\n            raise ValueError(f\"Tool name should be Play, got {tool_names}\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/react/base.html"}583{"id": "1979855fa776-3", "text": "raise ValueError(f\"Tool name should be Play, got {tool_names}\")\n[docs]class ReActChain(AgentExecutor):\n    \"\"\"Chain that implements the ReAct paper.\n    Example:\n        .. code-block:: python\n            from langchain import ReActChain, OpenAI\n            react = ReAct(llm=OpenAI())\n    \"\"\"\n    def __init__(self, llm: BaseLanguageModel, docstore: Docstore, **kwargs: Any):\n        \"\"\"Initialize with the LLM and a docstore.\"\"\"\n        docstore_explorer = DocstoreExplorer(docstore)\n        tools = [\n            Tool(\n                name=\"Search\",\n                func=docstore_explorer.search,\n                description=\"Search for a term in the docstore.\",\n            ),\n            Tool(\n                name=\"Lookup\",\n                func=docstore_explorer.lookup,\n                description=\"Lookup a term in the docstore.\",\n            ),\n        ]\n        agent = ReActDocstoreAgent.from_llm_and_tools(llm, tools)\n        super().__init__(agent=agent, tools=tools, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/agents/react/base.html"}584{"id": "ffb4a1e80409-0", "text": "Source code for langchain.utilities.spark_sql\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, Any, Iterable, List, Optional\nif TYPE_CHECKING:\n    from pyspark.sql import DataFrame, Row, SparkSession\n[docs]class SparkSQL:\n    def __init__(\n        self,\n        spark_session: Optional[SparkSession] = None,\n        catalog: Optional[str] = None,\n        schema: Optional[str] = None,\n        ignore_tables: Optional[List[str]] = None,\n        include_tables: Optional[List[str]] = None,\n        sample_rows_in_table_info: int = 3,\n    ):\n        try:\n            from pyspark.sql import SparkSession\n        except ImportError:\n            raise ValueError(\n                \"pyspark is not installed. Please install it with `pip install pyspark`\"\n            )\n        self._spark = (\n            spark_session if spark_session else SparkSession.builder.getOrCreate()\n        )\n        if catalog is not None:\n            self._spark.catalog.setCurrentCatalog(catalog)\n        if schema is not None:\n            self._spark.catalog.setCurrentDatabase(schema)\n        self._all_tables = set(self._get_all_table_names())\n        self._include_tables = set(include_tables) if include_tables else set()\n        if self._include_tables:\n            missing_tables = self._include_tables - self._all_tables\n            if missing_tables:\n                raise ValueError(\n                    f\"include_tables {missing_tables} not found in database\"\n                )\n        self._ignore_tables = set(ignore_tables) if ignore_tables else set()\n        if self._ignore_tables:\n            missing_tables = self._ignore_tables - self._all_tables\n            if missing_tables:\n                raise ValueError(\n                    f\"ignore_tables {missing_tables} not found in database\"\n                )", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/spark_sql.html"}585{"id": "ffb4a1e80409-1", "text": "f\"ignore_tables {missing_tables} not found in database\"\n                )\n        usable_tables = self.get_usable_table_names()\n        self._usable_tables = set(usable_tables) if usable_tables else self._all_tables\n        if not isinstance(sample_rows_in_table_info, int):\n            raise TypeError(\"sample_rows_in_table_info must be an integer\")\n        self._sample_rows_in_table_info = sample_rows_in_table_info\n[docs]    @classmethod\n    def from_uri(\n        cls, database_uri: str, engine_args: Optional[dict] = None, **kwargs: Any\n    ) -> SparkSQL:\n        \"\"\"Creating a remote Spark Session via Spark connect.\n        For example: SparkSQL.from_uri(\"sc://localhost:15002\")\n        \"\"\"\n        try:\n            from pyspark.sql import SparkSession\n        except ImportError:\n            raise ValueError(\n                \"pyspark is not installed. Please install it with `pip install pyspark`\"\n            )\n        spark = SparkSession.builder.remote(database_uri).getOrCreate()\n        return cls(spark, **kwargs)\n[docs]    def get_usable_table_names(self) -> Iterable[str]:\n        \"\"\"Get names of tables available.\"\"\"\n        if self._include_tables:\n            return self._include_tables\n        # sorting the result can help LLM understanding it.\n        return sorted(self._all_tables - self._ignore_tables)\n    def _get_all_table_names(self) -> Iterable[str]:\n        rows = self._spark.sql(\"SHOW TABLES\").select(\"tableName\").collect()\n        return list(map(lambda row: row.tableName, rows))\n    def _get_create_table_stmt(self, table: str) -> str:\n        statement = (\n            self._spark.sql(f\"SHOW CREATE TABLE {table}\").collect()[0].createtab_stmt", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/spark_sql.html"}586{"id": "ffb4a1e80409-2", "text": ")\n        # Ignore the data source provider and options to reduce the number of tokens.\n        using_clause_index = statement.find(\"USING\")\n        return statement[:using_clause_index] + \";\"\n[docs]    def get_table_info(self, table_names: Optional[List[str]] = None) -> str:\n        all_table_names = self.get_usable_table_names()\n        if table_names is not None:\n            missing_tables = set(table_names).difference(all_table_names)\n            if missing_tables:\n                raise ValueError(f\"table_names {missing_tables} not found in database\")\n            all_table_names = table_names\n        tables = []\n        for table_name in all_table_names:\n            table_info = self._get_create_table_stmt(table_name)\n            if self._sample_rows_in_table_info:\n                table_info += \"\\n\\n/*\"\n                table_info += f\"\\n{self._get_sample_spark_rows(table_name)}\\n\"\n                table_info += \"*/\"\n            tables.append(table_info)\n        final_str = \"\\n\\n\".join(tables)\n        return final_str\n    def _get_sample_spark_rows(self, table: str) -> str:\n        query = f\"SELECT * FROM {table} LIMIT {self._sample_rows_in_table_info}\"\n        df = self._spark.sql(query)\n        columns_str = \"\\t\".join(list(map(lambda f: f.name, df.schema.fields)))\n        try:\n            sample_rows = self._get_dataframe_results(df)\n            # save the sample rows in string format\n            sample_rows_str = \"\\n\".join([\"\\t\".join(row) for row in sample_rows])\n        except Exception:\n            sample_rows_str = \"\"\n        return (\n            f\"{self._sample_rows_in_table_info} rows from {table} table:\\n\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/spark_sql.html"}587{"id": "ffb4a1e80409-3", "text": "f\"{self._sample_rows_in_table_info} rows from {table} table:\\n\"\n            f\"{columns_str}\\n\"\n            f\"{sample_rows_str}\"\n        )\n    def _convert_row_as_tuple(self, row: Row) -> tuple:\n        return tuple(map(str, row.asDict().values()))\n    def _get_dataframe_results(self, df: DataFrame) -> list:\n        return list(map(self._convert_row_as_tuple, df.collect()))\n[docs]    def run(self, command: str, fetch: str = \"all\") -> str:\n        df = self._spark.sql(command)\n        if fetch == \"one\":\n            df = df.limit(1)\n        return str(self._get_dataframe_results(df))\n[docs]    def get_table_info_no_throw(self, table_names: Optional[List[str]] = None) -> str:\n        \"\"\"Get information about specified tables.\n        Follows best practices as specified in: Rajkumar et al, 2022\n        (https://arxiv.org/abs/2204.00498)\n        If `sample_rows_in_table_info`, the specified number of sample rows will be\n        appended to each table description. This can increase performance as\n        demonstrated in the paper.\n        \"\"\"\n        try:\n            return self.get_table_info(table_names)\n        except ValueError as e:\n            \"\"\"Format the error message\"\"\"\n            return f\"Error: {e}\"\n[docs]    def run_no_throw(self, command: str, fetch: str = \"all\") -> str:\n        \"\"\"Execute a SQL command and return a string representing the results.\n        If the statement returns rows, a string of the results is returned.\n        If the statement returns no rows, an empty string is returned.\n        If the statement throws an error, the error message is returned.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/spark_sql.html"}588{"id": "ffb4a1e80409-4", "text": "If the statement throws an error, the error message is returned.\n        \"\"\"\n        try:\n            from pyspark.errors import PySparkException\n        except ImportError:\n            raise ValueError(\n                \"pyspark is not installed. Please install it with `pip install pyspark`\"\n            )\n        try:\n            return self.run(command, fetch)\n        except PySparkException as e:\n            \"\"\"Format the error message\"\"\"\n            return f\"Error: {e}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/spark_sql.html"}589{"id": "565423240eb0-0", "text": "Source code for langchain.utilities.python\nimport sys\nfrom io import StringIO\nfrom typing import Dict, Optional\nfrom pydantic import BaseModel, Field\n[docs]class PythonREPL(BaseModel):\n    \"\"\"Simulates a standalone Python REPL.\"\"\"\n    globals: Optional[Dict] = Field(default_factory=dict, alias=\"_globals\")\n    locals: Optional[Dict] = Field(default_factory=dict, alias=\"_locals\")\n[docs]    def run(self, command: str) -> str:\n        \"\"\"Run command with own globals/locals and returns anything printed.\"\"\"\n        old_stdout = sys.stdout\n        sys.stdout = mystdout = StringIO()\n        try:\n            exec(command, self.globals, self.locals)\n            sys.stdout = old_stdout\n            output = mystdout.getvalue()\n        except Exception as e:\n            sys.stdout = old_stdout\n            output = repr(e)\n        return output\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/python.html"}590{"id": "a8a5b9481081-0", "text": "Source code for langchain.utilities.bash\n\"\"\"Wrapper around subprocess to run commands.\"\"\"\nfrom __future__ import annotations\nimport platform\nimport re\nimport subprocess\nfrom typing import TYPE_CHECKING, List, Union\nfrom uuid import uuid4\nif TYPE_CHECKING:\n    import pexpect\ndef _lazy_import_pexpect() -> pexpect:\n    \"\"\"Import pexpect only when needed.\"\"\"\n    if platform.system() == \"Windows\":\n        raise ValueError(\"Persistent bash processes are not yet supported on Windows.\")\n    try:\n        import pexpect\n    except ImportError:\n        raise ImportError(\n            \"pexpect required for persistent bash processes.\"\n            \" To install, run `pip install pexpect`.\"\n        )\n    return pexpect\n[docs]class BashProcess:\n    \"\"\"Executes bash commands and returns the output.\"\"\"\n    def __init__(\n        self,\n        strip_newlines: bool = False,\n        return_err_output: bool = False,\n        persistent: bool = False,\n    ):\n        \"\"\"Initialize with stripping newlines.\"\"\"\n        self.strip_newlines = strip_newlines\n        self.return_err_output = return_err_output\n        self.prompt = \"\"\n        self.process = None\n        if persistent:\n            self.prompt = str(uuid4())\n            self.process = self._initialize_persistent_process(self.prompt)\n    @staticmethod\n    def _initialize_persistent_process(prompt: str) -> pexpect.spawn:\n        # Start bash in a clean environment\n        # Doesn't work on windows\n        pexpect = _lazy_import_pexpect()\n        process = pexpect.spawn(\n            \"env\", [\"-i\", \"bash\", \"--norc\", \"--noprofile\"], encoding=\"utf-8\"\n        )\n        # Set the custom prompt\n        process.sendline(\"PS1=\" + prompt)", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bash.html"}591{"id": "a8a5b9481081-1", "text": "# Set the custom prompt\n        process.sendline(\"PS1=\" + prompt)\n        process.expect_exact(prompt, timeout=10)\n        return process\n[docs]    def run(self, commands: Union[str, List[str]]) -> str:\n        \"\"\"Run commands and return final output.\"\"\"\n        if isinstance(commands, str):\n            commands = [commands]\n        commands = \";\".join(commands)\n        if self.process is not None:\n            return self._run_persistent(\n                commands,\n            )\n        else:\n            return self._run(commands)\n    def _run(self, command: str) -> str:\n        \"\"\"Run commands and return final output.\"\"\"\n        try:\n            output = subprocess.run(\n                command,\n                shell=True,\n                check=True,\n                stdout=subprocess.PIPE,\n                stderr=subprocess.STDOUT,\n            ).stdout.decode()\n        except subprocess.CalledProcessError as error:\n            if self.return_err_output:\n                return error.stdout.decode()\n            return str(error)\n        if self.strip_newlines:\n            output = output.strip()\n        return output\n[docs]    def process_output(self, output: str, command: str) -> str:\n        # Remove the command from the output using a regular expression\n        pattern = re.escape(command) + r\"\\s*\\n\"\n        output = re.sub(pattern, \"\", output, count=1)\n        return output.strip()\n    def _run_persistent(self, command: str) -> str:\n        \"\"\"Run commands and return final output.\"\"\"\n        pexpect = _lazy_import_pexpect()\n        if self.process is None:\n            raise ValueError(\"Process not initialized\")\n        self.process.sendline(command)\n        # Clear the output with an empty string\n        self.process.expect(self.prompt, timeout=10)", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bash.html"}592{"id": "a8a5b9481081-2", "text": "self.process.expect(self.prompt, timeout=10)\n        self.process.sendline(\"\")\n        try:\n            self.process.expect([self.prompt, pexpect.EOF], timeout=10)\n        except pexpect.TIMEOUT:\n            return f\"Timeout error while executing command {command}\"\n        if self.process.after == pexpect.EOF:\n            return f\"Exited with error status: {self.process.exitstatus}\"\n        output = self.process.before\n        output = self.process_output(output, command)\n        if self.strip_newlines:\n            return output.strip()\n        return output\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bash.html"}593{"id": "5f9b097e0978-0", "text": "Source code for langchain.utilities.google_places_api\n\"\"\"Chain that calls Google Places API.\n\"\"\"\nimport logging\nfrom typing import Any, Dict, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\n[docs]class GooglePlacesAPIWrapper(BaseModel):\n    \"\"\"Wrapper around Google Places API.\n    To use, you should have the ``googlemaps`` python package installed,\n     **an API key for the google maps platform**,\n     and the enviroment variable ''GPLACES_API_KEY''\n     set with your API key , or pass 'gplaces_api_key'\n     as a named parameter to the constructor.\n    By default, this will return the all the results on the input query.\n     You can use the top_k_results argument to limit the number of results.\n    Example:\n        .. code-block:: python\n            from langchain import GooglePlacesAPIWrapper\n            gplaceapi = GooglePlacesAPIWrapper()\n    \"\"\"\n    gplaces_api_key: Optional[str] = None\n    google_map_client: Any  #: :meta private:\n    top_k_results: Optional[int] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key is in your environment variable.\"\"\"\n        gplaces_api_key = get_from_dict_or_env(\n            values, \"gplaces_api_key\", \"GPLACES_API_KEY\"\n        )\n        values[\"gplaces_api_key\"] = gplaces_api_key\n        try:\n            import googlemaps\n            values[\"google_map_client\"] = googlemaps.Client(gplaces_api_key)\n        except ImportError:\n            raise ImportError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_places_api.html"}594{"id": "5f9b097e0978-1", "text": "except ImportError:\n            raise ImportError(\n                \"Could not import googlemaps python package. \"\n                \"Please install it with `pip install googlemaps`.\"\n            )\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run Places search and get k number of places that exists that match.\"\"\"\n        search_results = self.google_map_client.places(query)[\"results\"]\n        num_to_return = len(search_results)\n        places = []\n        if num_to_return == 0:\n            return \"Google Places did not find any places that match the description\"\n        num_to_return = (\n            num_to_return\n            if self.top_k_results is None\n            else min(num_to_return, self.top_k_results)\n        )\n        for i in range(num_to_return):\n            result = search_results[i]\n            details = self.fetch_place_details(result[\"place_id\"])\n            if details is not None:\n                places.append(details)\n        return \"\\n\".join([f\"{i+1}. {item}\" for i, item in enumerate(places)])\n[docs]    def fetch_place_details(self, place_id: str) -> Optional[str]:\n        try:\n            place_details = self.google_map_client.place(place_id)\n            formatted_details = self.format_place_details(place_details)\n            return formatted_details\n        except Exception as e:\n            logging.error(f\"An Error occurred while fetching place details: {e}\")\n            return None\n[docs]    def format_place_details(self, place_details: Dict[str, Any]) -> Optional[str]:\n        try:\n            name = place_details.get(\"result\", {}).get(\"name\", \"Unkown\")\n            address = place_details.get(\"result\", {}).get(\n                \"formatted_address\", \"Unknown\"\n            )", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_places_api.html"}595{"id": "5f9b097e0978-2", "text": "\"formatted_address\", \"Unknown\"\n            )\n            phone_number = place_details.get(\"result\", {}).get(\n                \"formatted_phone_number\", \"Unknown\"\n            )\n            website = place_details.get(\"result\", {}).get(\"website\", \"Unknown\")\n            formatted_details = (\n                f\"{name}\\nAddress: {address}\\n\"\n                f\"Phone: {phone_number}\\nWebsite: {website}\\n\\n\"\n            )\n            return formatted_details\n        except Exception as e:\n            logging.error(f\"An error occurred while formatting place details: {e}\")\n            return None\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_places_api.html"}596{"id": "451d01e155c6-0", "text": "Source code for langchain.utilities.graphql\nimport json\nfrom typing import Any, Callable, Dict, Optional\nfrom pydantic import BaseModel, Extra, root_validator\n[docs]class GraphQLAPIWrapper(BaseModel):\n    \"\"\"Wrapper around GraphQL API.\n    To use, you should have the ``gql`` python package installed.\n    This wrapper will use the GraphQL API to conduct queries.\n    \"\"\"\n    custom_headers: Optional[Dict[str, str]] = None\n    graphql_endpoint: str\n    gql_client: Any  #: :meta private:\n    gql_function: Callable[[str], Any]  #: :meta private:\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that the python package exists in the environment.\"\"\"\n        try:\n            from gql import Client, gql\n            from gql.transport.requests import RequestsHTTPTransport\n        except ImportError as e:\n            raise ImportError(\n                \"Could not import gql python package. \"\n                f\"Try installing it with `pip install gql`. Received error: {e}\"\n            )\n        headers = values.get(\"custom_headers\")\n        transport = RequestsHTTPTransport(\n            url=values[\"graphql_endpoint\"],\n            headers=headers,\n        )\n        client = Client(transport=transport, fetch_schema_from_transport=True)\n        values[\"gql_client\"] = client\n        values[\"gql_function\"] = gql\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run a GraphQL query and get the results.\"\"\"\n        result = self._execute_query(query)\n        return json.dumps(result, indent=2)", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/graphql.html"}597{"id": "451d01e155c6-1", "text": "result = self._execute_query(query)\n        return json.dumps(result, indent=2)\n    def _execute_query(self, query: str) -> Dict[str, Any]:\n        \"\"\"Execute a GraphQL query and return the results.\"\"\"\n        document_node = self.gql_function(query)\n        result = self.gql_client.execute(document_node)\n        return result\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/graphql.html"}598{"id": "ffcd814928d6-0", "text": "Source code for langchain.utilities.duckduckgo_search\n\"\"\"Util that calls DuckDuckGo Search.\nNo setup required. Free.\nhttps://pypi.org/project/duckduckgo-search/\n\"\"\"\nfrom typing import Dict, List, Optional\nfrom pydantic import BaseModel, Extra\nfrom pydantic.class_validators import root_validator\n[docs]class DuckDuckGoSearchAPIWrapper(BaseModel):\n    \"\"\"Wrapper for DuckDuckGo Search API.\n    Free and does not require any setup\n    \"\"\"\n    k: int = 10\n    region: Optional[str] = \"wt-wt\"\n    safesearch: str = \"moderate\"\n    time: Optional[str] = \"y\"\n    max_results: int = 5\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that python package exists in environment.\"\"\"\n        try:\n            from duckduckgo_search import ddg  # noqa: F401\n        except ImportError:\n            raise ValueError(\n                \"Could not import duckduckgo-search python package. \"\n                \"Please install it with `pip install duckduckgo-search`.\"\n            )\n        return values\n[docs]    def get_snippets(self, query: str) -> List[str]:\n        \"\"\"Run query through DuckDuckGo and return concatenated results.\"\"\"\n        from duckduckgo_search import ddg\n        results = ddg(\n            query,\n            region=self.region,\n            safesearch=self.safesearch,\n            time=self.time,\n            max_results=self.max_results,\n        )\n        if results is None or len(results) == 0:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/duckduckgo_search.html"}599{"id": "ffcd814928d6-1", "text": ")\n        if results is None or len(results) == 0:\n            return [\"No good DuckDuckGo Search Result was found\"]\n        snippets = [result[\"body\"] for result in results]\n        return snippets\n[docs]    def run(self, query: str) -> str:\n        snippets = self.get_snippets(query)\n        return \" \".join(snippets)\n[docs]    def results(self, query: str, num_results: int) -> List[Dict[str, str]]:\n        \"\"\"Run query through DuckDuckGo and return metadata.\n        Args:\n            query: The query to search for.\n            num_results: The number of results to return.\n        Returns:\n            A list of dictionaries with the following keys:\n                snippet - The description of the result.\n                title - The title of the result.\n                link - The link to the result.\n        \"\"\"\n        from duckduckgo_search import ddg\n        results = ddg(\n            query,\n            region=self.region,\n            safesearch=self.safesearch,\n            time=self.time,\n            max_results=num_results,\n        )\n        if results is None or len(results) == 0:\n            return [{\"Result\": \"No good DuckDuckGo Search Result was found\"}]\n        def to_metadata(result: Dict) -> Dict[str, str]:\n            return {\n                \"snippet\": result[\"body\"],\n                \"title\": result[\"title\"],\n                \"link\": result[\"href\"],\n            }\n        return [to_metadata(result) for result in results]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/duckduckgo_search.html"}600{"id": "fe9d545ce648-0", "text": "Source code for langchain.utilities.apify\nfrom typing import Any, Callable, Dict, Optional\nfrom pydantic import BaseModel, root_validator\nfrom langchain.document_loaders import ApifyDatasetLoader\nfrom langchain.document_loaders.base import Document\nfrom langchain.utils import get_from_dict_or_env\n[docs]class ApifyWrapper(BaseModel):\n    \"\"\"Wrapper around Apify.\n    To use, you should have the ``apify-client`` python package installed,\n    and the environment variable ``APIFY_API_TOKEN`` set with your API key, or pass\n    `apify_api_token` as a named parameter to the constructor.\n    \"\"\"\n    apify_client: Any\n    apify_client_async: Any\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate environment.\n        Validate that an Apify API token is set and the apify-client\n        Python package exists in the current environment.\n        \"\"\"\n        apify_api_token = get_from_dict_or_env(\n            values, \"apify_api_token\", \"APIFY_API_TOKEN\"\n        )\n        try:\n            from apify_client import ApifyClient, ApifyClientAsync\n            values[\"apify_client\"] = ApifyClient(apify_api_token)\n            values[\"apify_client_async\"] = ApifyClientAsync(apify_api_token)\n        except ImportError:\n            raise ValueError(\n                \"Could not import apify-client Python package. \"\n                \"Please install it with `pip install apify-client`.\"\n            )\n        return values\n[docs]    def call_actor(\n        self,\n        actor_id: str,\n        run_input: Dict,\n        dataset_mapping_function: Callable[[Dict], Document],\n        *,\n        build: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/apify.html"}601{"id": "fe9d545ce648-1", "text": "*,\n        build: Optional[str] = None,\n        memory_mbytes: Optional[int] = None,\n        timeout_secs: Optional[int] = None,\n    ) -> ApifyDatasetLoader:\n        \"\"\"Run an Actor on the Apify platform and wait for results to be ready.\n        Args:\n            actor_id (str): The ID or name of the Actor on the Apify platform.\n            run_input (Dict): The input object of the Actor that you're trying to run.\n            dataset_mapping_function (Callable): A function that takes a single\n                dictionary (an Apify dataset item) and converts it to an\n                instance of the Document class.\n            build (str, optional): Optionally specifies the actor build to run.\n                It can be either a build tag or build number.\n            memory_mbytes (int, optional): Optional memory limit for the run,\n                in megabytes.\n            timeout_secs (int, optional): Optional timeout for the run, in seconds.\n        Returns:\n            ApifyDatasetLoader: A loader that will fetch the records from the\n                Actor run's default dataset.\n        \"\"\"\n        actor_call = self.apify_client.actor(actor_id).call(\n            run_input=run_input,\n            build=build,\n            memory_mbytes=memory_mbytes,\n            timeout_secs=timeout_secs,\n        )\n        return ApifyDatasetLoader(\n            dataset_id=actor_call[\"defaultDatasetId\"],\n            dataset_mapping_function=dataset_mapping_function,\n        )\n[docs]    async def acall_actor(\n        self,\n        actor_id: str,\n        run_input: Dict,\n        dataset_mapping_function: Callable[[Dict], Document],\n        *,\n        build: Optional[str] = None,\n        memory_mbytes: Optional[int] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/apify.html"}602{"id": "fe9d545ce648-2", "text": "memory_mbytes: Optional[int] = None,\n        timeout_secs: Optional[int] = None,\n    ) -> ApifyDatasetLoader:\n        \"\"\"Run an Actor on the Apify platform and wait for results to be ready.\n        Args:\n            actor_id (str): The ID or name of the Actor on the Apify platform.\n            run_input (Dict): The input object of the Actor that you're trying to run.\n            dataset_mapping_function (Callable): A function that takes a single\n                dictionary (an Apify dataset item) and converts it to\n                an instance of the Document class.\n            build (str, optional): Optionally specifies the actor build to run.\n                It can be either a build tag or build number.\n            memory_mbytes (int, optional): Optional memory limit for the run,\n                in megabytes.\n            timeout_secs (int, optional): Optional timeout for the run, in seconds.\n        Returns:\n            ApifyDatasetLoader: A loader that will fetch the records from the\n                Actor run's default dataset.\n        \"\"\"\n        actor_call = await self.apify_client_async.actor(actor_id).call(\n            run_input=run_input,\n            build=build,\n            memory_mbytes=memory_mbytes,\n            timeout_secs=timeout_secs,\n        )\n        return ApifyDatasetLoader(\n            dataset_id=actor_call[\"defaultDatasetId\"],\n            dataset_mapping_function=dataset_mapping_function,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/apify.html"}603{"id": "7fdf084c9a19-0", "text": "Source code for langchain.utilities.twilio\n\"\"\"Util that calls Twilio.\"\"\"\nfrom typing import Any, Dict, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\n[docs]class TwilioAPIWrapper(BaseModel):\n    \"\"\"Sms Client using Twilio.\n    To use, you should have the ``twilio`` python package installed,\n    and the environment variables ``TWILIO_ACCOUNT_SID``, ``TWILIO_AUTH_TOKEN``, and\n    ``TWILIO_FROM_NUMBER``, or pass `account_sid`, `auth_token`, and `from_number` as\n    named parameters to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.utilities.twilio import TwilioAPIWrapper\n            twilio = TwilioAPIWrapper(\n                account_sid=\"ACxxx\",\n                auth_token=\"xxx\",\n                from_number=\"+10123456789\"\n            )\n            twilio.run('test', '+12484345508')\n    \"\"\"\n    client: Any  #: :meta private:\n    account_sid: Optional[str] = None\n    \"\"\"Twilio account string identifier.\"\"\"\n    auth_token: Optional[str] = None\n    \"\"\"Twilio auth token.\"\"\"\n    from_number: Optional[str] = None\n    \"\"\"A Twilio phone number in [E.164](https://www.twilio.com/docs/glossary/what-e164) \n        format, an \n        [alphanumeric sender ID](https://www.twilio.com/docs/sms/send-messages#use-an-alphanumeric-sender-id), \n        or a [Channel Endpoint address](https://www.twilio.com/docs/sms/channels#channel-addresses) \n        that is enabled for the type of message you want to send. Phone numbers or", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/twilio.html"}604{"id": "7fdf084c9a19-1", "text": "that is enabled for the type of message you want to send. Phone numbers or \n        [short codes](https://www.twilio.com/docs/sms/api/short-code) purchased from \n        Twilio also work here. You cannot, for example, spoof messages from a private \n        cell phone number. If you are using `messaging_service_sid`, this parameter \n        must be empty.\n    \"\"\"  # noqa: E501\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = False\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        try:\n            from twilio.rest import Client\n        except ImportError:\n            raise ImportError(\n                \"Could not import twilio python package. \"\n                \"Please install it with `pip install twilio`.\"\n            )\n        account_sid = get_from_dict_or_env(values, \"account_sid\", \"TWILIO_ACCOUNT_SID\")\n        auth_token = get_from_dict_or_env(values, \"auth_token\", \"TWILIO_AUTH_TOKEN\")\n        values[\"from_number\"] = get_from_dict_or_env(\n            values, \"from_number\", \"TWILIO_FROM_NUMBER\"\n        )\n        values[\"client\"] = Client(account_sid, auth_token)\n        return values\n[docs]    def run(self, body: str, to: str) -> str:\n        \"\"\"Run body through Twilio and respond with message sid.\n        Args:\n            body: The text of the message you want to send. Can be up to 1,600\n                characters in length.\n            to: The destination phone number in", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/twilio.html"}605{"id": "7fdf084c9a19-2", "text": "characters in length.\n            to: The destination phone number in\n                [E.164](https://www.twilio.com/docs/glossary/what-e164) format for\n                SMS/MMS or\n                [Channel user address](https://www.twilio.com/docs/sms/channels#channel-addresses)\n                for other 3rd-party channels.\n        \"\"\"  # noqa: E501\n        message = self.client.messages.create(to, from_=self.from_number, body=body)\n        return message.sid\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/twilio.html"}606{"id": "f6b5ad050170-0", "text": "Source code for langchain.utilities.searx_search\n\"\"\"Utility for using SearxNG meta search API.\nSearxNG is a privacy-friendly free metasearch engine that aggregates results from\n`multiple search engines\n<https://docs.searxng.org/admin/engines/configured_engines.html>`_ and databases and\nsupports the `OpenSearch \n<https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md>`_\nspecification.\nMore detailes on the installtion instructions `here. <../../integrations/searx.html>`_\nFor the search API refer to https://docs.searxng.org/dev/search_api.html\nQuick Start\n-----------\nIn order to use this utility you need to provide the searx host. This can be done\nby passing the named parameter :attr:`searx_host <SearxSearchWrapper.searx_host>`\nor exporting the environment variable SEARX_HOST.\nNote: this is the only required parameter.\nThen create a searx search instance like this:\n    .. code-block:: python\n        from langchain.utilities import SearxSearchWrapper\n        # when the host starts with `http` SSL is disabled and the connection\n        # is assumed to be on a private network\n        searx_host='http://self.hosted'\n        search = SearxSearchWrapper(searx_host=searx_host)\nYou can now use the ``search`` instance to query the searx API.\nSearching\n---------\nUse the :meth:`run() <SearxSearchWrapper.run>` and\n:meth:`results() <SearxSearchWrapper.results>` methods to query the searx API.\nOther methods are are available for convenience.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}607{"id": "f6b5ad050170-1", "text": "Other methods are are available for convenience.\n:class:`SearxResults` is a convenience wrapper around the raw json result.\nExample usage of the ``run`` method to make a search:\n    .. code-block:: python\n        s.run(query=\"what is the best search engine?\")\nEngine Parameters\n-----------------\nYou can pass any `accepted searx search API\n<https://docs.searxng.org/dev/search_api.html>`_ parameters to the\n:py:class:`SearxSearchWrapper` instance.\nIn the following example we are using the\n:attr:`engines <SearxSearchWrapper.engines>` and the ``language`` parameters:\n    .. code-block:: python\n        # assuming the searx host is set as above or exported as an env variable\n        s = SearxSearchWrapper(engines=['google', 'bing'],\n                            language='es')\nSearch Tips\n-----------\nSearx offers a special\n`search syntax <https://docs.searxng.org/user/index.html#search-syntax>`_\nthat can also be used instead of passing engine parameters.\nFor example the following query:\n    .. code-block:: python\n        s = SearxSearchWrapper(\"langchain library\", engines=['github'])\n        # can also be written as:\n        s = SearxSearchWrapper(\"langchain library !github\")\n        # or even:\n        s = SearxSearchWrapper(\"langchain library !gh\")\nIn some situations you might want to pass an extra string to the search query.\nFor example when the `run()` method is called by an agent. The search suffix can\nalso be used as a way to pass extra parameters to searx or the underlying search\nengines.\n    .. code-block:: python\n        # select the github engine and pass the search suffix", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}608{"id": "f6b5ad050170-2", "text": ".. code-block:: python\n        # select the github engine and pass the search suffix\n        s = SearchWrapper(\"langchain library\", query_suffix=\"!gh\")\n        s = SearchWrapper(\"langchain library\")\n        # select github the conventional google search syntax\n        s.run(\"large language models\", query_suffix=\"site:github.com\")\n*NOTE*: A search suffix can be defined on both the instance and the method level.\nThe resulting query will be the concatenation of the two with the former taking\nprecedence.\nSee `SearxNG Configured Engines\n<https://docs.searxng.org/admin/engines/configured_engines.html>`_ and\n`SearxNG Search Syntax <https://docs.searxng.org/user/index.html#id1>`_\nfor more details.\nNotes\n-----\nThis wrapper is based on the SearxNG fork https://github.com/searxng/searxng which is\nbetter maintained than the original Searx project and offers more features.\nPublic searxNG instances often use a rate limiter for API usage, so you might want to\nuse a self hosted instance and disable the rate limiter.\nIf you are self-hosting an instance you can customize the rate limiter for your\nown network as described `here <https://github.com/searxng/searxng/pull/2129>`_.\nFor a list of public SearxNG instances see https://searx.space/\n\"\"\"\nimport json\nfrom typing import Any, Dict, List, Optional\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel, Extra, Field, PrivateAttr, root_validator, validator\nfrom langchain.utils import get_from_dict_or_env\ndef _get_default_params() -> dict:\n    return {\"language\": \"en\", \"format\": \"json\"}", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}609{"id": "f6b5ad050170-3", "text": "return {\"language\": \"en\", \"format\": \"json\"}\n[docs]class SearxResults(dict):\n    \"\"\"Dict like wrapper around search api results.\"\"\"\n    _data = \"\"\n    def __init__(self, data: str):\n        \"\"\"Take a raw result from Searx and make it into a dict like object.\"\"\"\n        json_data = json.loads(data)\n        super().__init__(json_data)\n        self.__dict__ = self\n    def __str__(self) -> str:\n        \"\"\"Text representation of searx result.\"\"\"\n        return self._data\n    @property\n    def results(self) -> Any:\n        \"\"\"Silence mypy for accessing this field.\n        :meta private:\n        \"\"\"\n        return self.get(\"results\")\n    @property\n    def answers(self) -> Any:\n        \"\"\"Helper accessor on the json result.\"\"\"\n        return self.get(\"answers\")\n[docs]class SearxSearchWrapper(BaseModel):\n    \"\"\"Wrapper for Searx API.\n    To use you need to provide the searx host by passing the named parameter\n    ``searx_host`` or exporting the environment variable ``SEARX_HOST``.\n    In some situations you might want to disable SSL verification, for example\n    if you are running searx locally. You can do this by passing the named parameter\n    ``unsecure``. You can also pass the host url scheme as ``http`` to disable SSL.\n    Example:\n        .. code-block:: python\n            from langchain.utilities import SearxSearchWrapper\n            searx = SearxSearchWrapper(searx_host=\"http://localhost:8888\")\n    Example with SSL disabled:\n        .. code-block:: python\n            from langchain.utilities import SearxSearchWrapper", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}610{"id": "f6b5ad050170-4", "text": ".. code-block:: python\n            from langchain.utilities import SearxSearchWrapper\n            # note the unsecure parameter is not needed if you pass the url scheme as\n            # http\n            searx = SearxSearchWrapper(searx_host=\"http://localhost:8888\",\n                                                    unsecure=True)\n    \"\"\"\n    _result: SearxResults = PrivateAttr()\n    searx_host: str = \"\"\n    unsecure: bool = False\n    params: dict = Field(default_factory=_get_default_params)\n    headers: Optional[dict] = None\n    engines: Optional[List[str]] = []\n    categories: Optional[List[str]] = []\n    query_suffix: Optional[str] = \"\"\n    k: int = 10\n    aiosession: Optional[Any] = None\n    @validator(\"unsecure\")\n    def disable_ssl_warnings(cls, v: bool) -> bool:\n        \"\"\"Disable SSL warnings.\"\"\"\n        if v:\n            # requests.urllib3.disable_warnings()\n            try:\n                import urllib3\n                urllib3.disable_warnings()\n            except ImportError as e:\n                print(e)\n        return v\n    @root_validator()\n    def validate_params(cls, values: Dict) -> Dict:\n        \"\"\"Validate that custom searx params are merged with default ones.\"\"\"\n        user_params = values[\"params\"]\n        default = _get_default_params()\n        values[\"params\"] = {**default, **user_params}\n        engines = values.get(\"engines\")\n        if engines:\n            values[\"params\"][\"engines\"] = \",\".join(engines)\n        categories = values.get(\"categories\")\n        if categories:\n            values[\"params\"][\"categories\"] = \",\".join(categories)", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}611{"id": "f6b5ad050170-5", "text": "if categories:\n            values[\"params\"][\"categories\"] = \",\".join(categories)\n        searx_host = get_from_dict_or_env(values, \"searx_host\", \"SEARX_HOST\")\n        if not searx_host.startswith(\"http\"):\n            print(\n                f\"Warning: missing the url scheme on host \\\n                ! assuming secure https://{searx_host} \"\n            )\n            searx_host = \"https://\" + searx_host\n        elif searx_host.startswith(\"http://\"):\n            values[\"unsecure\"] = True\n            cls.disable_ssl_warnings(True)\n        values[\"searx_host\"] = searx_host\n        return values\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def _searx_api_query(self, params: dict) -> SearxResults:\n        \"\"\"Actual request to searx API.\"\"\"\n        raw_result = requests.get(\n            self.searx_host,\n            headers=self.headers,\n            params=params,\n            verify=not self.unsecure,\n        )\n        # test if http result is ok\n        if not raw_result.ok:\n            raise ValueError(\"Searx API returned an error: \", raw_result.text)\n        res = SearxResults(raw_result.text)\n        self._result = res\n        return res\n    async def _asearx_api_query(self, params: dict) -> SearxResults:\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.get(\n                    self.searx_host,\n                    headers=self.headers,\n                    params=params,\n                    ssl=(lambda: False if self.unsecure else None)(),\n                ) as response:\n                    if not response.ok:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}612{"id": "f6b5ad050170-6", "text": ") as response:\n                    if not response.ok:\n                        raise ValueError(\"Searx API returned an error: \", response.text)\n                    result = SearxResults(await response.text())\n                    self._result = result\n        else:\n            async with self.aiosession.get(\n                self.searx_host,\n                headers=self.headers,\n                params=params,\n                verify=not self.unsecure,\n            ) as response:\n                if not response.ok:\n                    raise ValueError(\"Searx API returned an error: \", response.text)\n                result = SearxResults(await response.text())\n                self._result = result\n        return result\n[docs]    def run(\n        self,\n        query: str,\n        engines: Optional[List[str]] = None,\n        categories: Optional[List[str]] = None,\n        query_suffix: Optional[str] = \"\",\n        **kwargs: Any,\n    ) -> str:\n        \"\"\"Run query through Searx API and parse results.\n        You can pass any other params to the searx query API.\n        Args:\n            query: The query to search for.\n            query_suffix: Extra suffix appended to the query.\n            engines: List of engines to use for the query.\n            categories: List of categories to use for the query.\n            **kwargs: extra parameters to pass to the searx API.\n        Returns:\n            str: The result of the query.\n        Raises:\n            ValueError: If an error occured with the query.\n        Example:\n            This will make a query to the qwant engine:\n            .. code-block:: python\n                from langchain.utilities import SearxSearchWrapper\n                searx = SearxSearchWrapper(searx_host=\"http://my.searx.host\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}613{"id": "f6b5ad050170-7", "text": "searx.run(\"what is the weather in France ?\", engine=\"qwant\")\n                # the same result can be achieved using the `!` syntax of searx\n                # to select the engine using `query_suffix`\n                searx.run(\"what is the weather in France ?\", query_suffix=\"!qwant\")\n        \"\"\"\n        _params = {\n            \"q\": query,\n        }\n        params = {**self.params, **_params, **kwargs}\n        if self.query_suffix and len(self.query_suffix) > 0:\n            params[\"q\"] += \" \" + self.query_suffix\n        if isinstance(query_suffix, str) and len(query_suffix) > 0:\n            params[\"q\"] += \" \" + query_suffix\n        if isinstance(engines, list) and len(engines) > 0:\n            params[\"engines\"] = \",\".join(engines)\n        if isinstance(categories, list) and len(categories) > 0:\n            params[\"categories\"] = \",\".join(categories)\n        res = self._searx_api_query(params)\n        if len(res.answers) > 0:\n            toret = res.answers[0]\n        # only return the content of the results list\n        elif len(res.results) > 0:\n            toret = \"\\n\\n\".join([r.get(\"content\", \"\") for r in res.results[: self.k]])\n        else:\n            toret = \"No good search result found\"\n        return toret\n[docs]    async def arun(\n        self,\n        query: str,\n        engines: Optional[List[str]] = None,\n        query_suffix: Optional[str] = \"\",\n        **kwargs: Any,\n    ) -> str:\n        \"\"\"Asynchronously version of `run`.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}614{"id": "f6b5ad050170-8", "text": ") -> str:\n        \"\"\"Asynchronously version of `run`.\"\"\"\n        _params = {\n            \"q\": query,\n        }\n        params = {**self.params, **_params, **kwargs}\n        if self.query_suffix and len(self.query_suffix) > 0:\n            params[\"q\"] += \" \" + self.query_suffix\n        if isinstance(query_suffix, str) and len(query_suffix) > 0:\n            params[\"q\"] += \" \" + query_suffix\n        if isinstance(engines, list) and len(engines) > 0:\n            params[\"engines\"] = \",\".join(engines)\n        res = await self._asearx_api_query(params)\n        if len(res.answers) > 0:\n            toret = res.answers[0]\n        # only return the content of the results list\n        elif len(res.results) > 0:\n            toret = \"\\n\\n\".join([r.get(\"content\", \"\") for r in res.results[: self.k]])\n        else:\n            toret = \"No good search result found\"\n        return toret\n[docs]    def results(\n        self,\n        query: str,\n        num_results: int,\n        engines: Optional[List[str]] = None,\n        categories: Optional[List[str]] = None,\n        query_suffix: Optional[str] = \"\",\n        **kwargs: Any,\n    ) -> List[Dict]:\n        \"\"\"Run query through Searx API and returns the results with metadata.\n        Args:\n            query: The query to search for.\n            query_suffix: Extra suffix appended to the query.\n            num_results: Limit the number of results to return.\n            engines: List of engines to use for the query.\n            categories: List of categories to use for the query.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}615{"id": "f6b5ad050170-9", "text": "categories: List of categories to use for the query.\n            **kwargs: extra parameters to pass to the searx API.\n        Returns:\n            Dict with the following keys:\n            {\n                snippet:  The description of the result.\n                title:  The title of the result.\n                link: The link to the result.\n                engines: The engines used for the result.\n                category: Searx category of the result.\n            }\n        \"\"\"\n        _params = {\n            \"q\": query,\n        }\n        params = {**self.params, **_params, **kwargs}\n        if self.query_suffix and len(self.query_suffix) > 0:\n            params[\"q\"] += \" \" + self.query_suffix\n        if isinstance(query_suffix, str) and len(query_suffix) > 0:\n            params[\"q\"] += \" \" + query_suffix\n        if isinstance(engines, list) and len(engines) > 0:\n            params[\"engines\"] = \",\".join(engines)\n        if isinstance(categories, list) and len(categories) > 0:\n            params[\"categories\"] = \",\".join(categories)\n        results = self._searx_api_query(params).results[:num_results]\n        if len(results) == 0:\n            return [{\"Result\": \"No good Search Result was found\"}]\n        return [\n            {\n                \"snippet\": result.get(\"content\", \"\"),\n                \"title\": result[\"title\"],\n                \"link\": result[\"url\"],\n                \"engines\": result[\"engines\"],\n                \"category\": result[\"category\"],\n            }\n            for result in results\n        ]\n[docs]    async def aresults(\n        self,\n        query: str,\n        num_results: int,", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}616{"id": "f6b5ad050170-10", "text": "self,\n        query: str,\n        num_results: int,\n        engines: Optional[List[str]] = None,\n        query_suffix: Optional[str] = \"\",\n        **kwargs: Any,\n    ) -> List[Dict]:\n        \"\"\"Asynchronously query with json results.\n        Uses aiohttp. See `results` for more info.\n        \"\"\"\n        _params = {\n            \"q\": query,\n        }\n        params = {**self.params, **_params, **kwargs}\n        if self.query_suffix and len(self.query_suffix) > 0:\n            params[\"q\"] += \" \" + self.query_suffix\n        if isinstance(query_suffix, str) and len(query_suffix) > 0:\n            params[\"q\"] += \" \" + query_suffix\n        if isinstance(engines, list) and len(engines) > 0:\n            params[\"engines\"] = \",\".join(engines)\n        results = (await self._asearx_api_query(params)).results[:num_results]\n        if len(results) == 0:\n            return [{\"Result\": \"No good Search Result was found\"}]\n        return [\n            {\n                \"snippet\": result.get(\"content\", \"\"),\n                \"title\": result[\"title\"],\n                \"link\": result[\"url\"],\n                \"engines\": result[\"engines\"],\n                \"category\": result[\"category\"],\n            }\n            for result in results\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/searx_search.html"}617{"id": "9a93ade63c1e-0", "text": "Source code for langchain.utilities.google_search\n\"\"\"Util that calls Google Search.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\n[docs]class GoogleSearchAPIWrapper(BaseModel):\n    \"\"\"Wrapper for Google Search API.\n    Adapted from: Instructions adapted from https://stackoverflow.com/questions/\n    37083058/\n    programmatically-searching-google-in-python-using-custom-search\n    TODO: DOCS for using it\n    1. Install google-api-python-client\n    - If you don't already have a Google account, sign up.\n    - If you have never created a Google APIs Console project,\n    read the Managing Projects page and create a project in the Google API Console.\n    - Install the library using pip install google-api-python-client\n    The current version of the library is 2.70.0 at this time\n    2. To create an API key:\n    - Navigate to the APIs & Services\u2192Credentials panel in Cloud Console.\n    - Select Create credentials, then select API key from the drop-down menu.\n    - The API key created dialog box displays your newly created key.\n    - You now have an API_KEY\n    3. Setup Custom Search Engine so you can search the entire web\n    - Create a custom search engine in this link.\n    - In Sites to search, add any valid URL (i.e. www.stackoverflow.com).\n    - That\u2019s all you have to fill up, the rest doesn\u2019t matter.\n    In the left-side menu, click Edit search engine \u2192 {your search engine name}\n    \u2192 Setup Set Search the entire web to ON. Remove the URL you added from\n     the list of Sites to search.\n    - Under Search engine ID you\u2019ll find the search-engine-ID.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_search.html"}618{"id": "9a93ade63c1e-1", "text": "- Under Search engine ID you\u2019ll find the search-engine-ID.\n    4. Enable the Custom Search API\n    - Navigate to the APIs & Services\u2192Dashboard panel in Cloud Console.\n    - Click Enable APIs and Services.\n    - Search for Custom Search API and click on it.\n    - Click Enable.\n    URL for it: https://console.cloud.google.com/apis/library/customsearch.googleapis\n    .com\n    \"\"\"\n    search_engine: Any  #: :meta private:\n    google_api_key: Optional[str] = None\n    google_cse_id: Optional[str] = None\n    k: int = 10\n    siterestrict: bool = False\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def _google_search_results(self, search_term: str, **kwargs: Any) -> List[dict]:\n        cse = self.search_engine.cse()\n        if self.siterestrict:\n            cse = cse.siterestrict()\n        res = cse.list(q=search_term, cx=self.google_cse_id, **kwargs).execute()\n        return res.get(\"items\", [])\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        google_api_key = get_from_dict_or_env(\n            values, \"google_api_key\", \"GOOGLE_API_KEY\"\n        )\n        values[\"google_api_key\"] = google_api_key\n        google_cse_id = get_from_dict_or_env(values, \"google_cse_id\", \"GOOGLE_CSE_ID\")\n        values[\"google_cse_id\"] = google_cse_id\n        try:\n            from googleapiclient.discovery import build\n        except ImportError:\n            raise ImportError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_search.html"}619{"id": "9a93ade63c1e-2", "text": "from googleapiclient.discovery import build\n        except ImportError:\n            raise ImportError(\n                \"google-api-python-client is not installed. \"\n                \"Please install it with `pip install google-api-python-client`\"\n            )\n        service = build(\"customsearch\", \"v1\", developerKey=google_api_key)\n        values[\"search_engine\"] = service\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run query through GoogleSearch and parse result.\"\"\"\n        snippets = []\n        results = self._google_search_results(query, num=self.k)\n        if len(results) == 0:\n            return \"No good Google Search Result was found\"\n        for result in results:\n            if \"snippet\" in result:\n                snippets.append(result[\"snippet\"])\n        return \" \".join(snippets)\n[docs]    def results(self, query: str, num_results: int) -> List[Dict]:\n        \"\"\"Run query through GoogleSearch and return metadata.\n        Args:\n            query: The query to search for.\n            num_results: The number of results to return.\n        Returns:\n            A list of dictionaries with the following keys:\n                snippet - The description of the result.\n                title - The title of the result.\n                link - The link to the result.\n        \"\"\"\n        metadata_results = []\n        results = self._google_search_results(query, num=num_results)\n        if len(results) == 0:\n            return [{\"Result\": \"No good Google Search Result was found\"}]\n        for result in results:\n            metadata_result = {\n                \"title\": result[\"title\"],\n                \"link\": result[\"link\"],\n            }\n            if \"snippet\" in result:\n                metadata_result[\"snippet\"] = result[\"snippet\"]", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_search.html"}620{"id": "9a93ade63c1e-3", "text": "if \"snippet\" in result:\n                metadata_result[\"snippet\"] = result[\"snippet\"]\n            metadata_results.append(metadata_result)\n        return metadata_results\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_search.html"}621{"id": "467341b6206f-0", "text": "Source code for langchain.utilities.wikipedia\n\"\"\"Util that calls Wikipedia.\"\"\"\nimport logging\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.schema import Document\nlogger = logging.getLogger(__name__)\nWIKIPEDIA_MAX_QUERY_LENGTH = 300\n[docs]class WikipediaAPIWrapper(BaseModel):\n    \"\"\"Wrapper around WikipediaAPI.\n    To use, you should have the ``wikipedia`` python package installed.\n    This wrapper will use the Wikipedia API to conduct searches and\n    fetch page summaries. By default, it will return the page summaries\n    of the top-k results.\n    It limits the Document content by doc_content_chars_max.\n    \"\"\"\n    wiki_client: Any  #: :meta private:\n    top_k_results: int = 3\n    lang: str = \"en\"\n    load_all_available_meta: bool = False\n    doc_content_chars_max: int = 4000\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that the python package exists in environment.\"\"\"\n        try:\n            import wikipedia\n            wikipedia.set_lang(values[\"lang\"])\n            values[\"wiki_client\"] = wikipedia\n        except ImportError:\n            raise ImportError(\n                \"Could not import wikipedia python package. \"\n                \"Please install it with `pip install wikipedia`.\"\n            )\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run Wikipedia search and get page summaries.\"\"\"\n        page_titles = self.wiki_client.search(query[:WIKIPEDIA_MAX_QUERY_LENGTH])\n        summaries = []\n        for page_title in page_titles[: self.top_k_results]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/wikipedia.html"}622{"id": "467341b6206f-1", "text": "summaries = []\n        for page_title in page_titles[: self.top_k_results]:\n            if wiki_page := self._fetch_page(page_title):\n                if summary := self._formatted_page_summary(page_title, wiki_page):\n                    summaries.append(summary)\n        if not summaries:\n            return \"No good Wikipedia Search Result was found\"\n        return \"\\n\\n\".join(summaries)[: self.doc_content_chars_max]\n    @staticmethod\n    def _formatted_page_summary(page_title: str, wiki_page: Any) -> Optional[str]:\n        return f\"Page: {page_title}\\nSummary: {wiki_page.summary}\"\n    def _page_to_document(self, page_title: str, wiki_page: Any) -> Document:\n        main_meta = {\n            \"title\": page_title,\n            \"summary\": wiki_page.summary,\n            \"source\": wiki_page.url,\n        }\n        add_meta = (\n            {\n                \"categories\": wiki_page.categories,\n                \"page_url\": wiki_page.url,\n                \"image_urls\": wiki_page.images,\n                \"related_titles\": wiki_page.links,\n                \"parent_id\": wiki_page.parent_id,\n                \"references\": wiki_page.references,\n                \"revision_id\": wiki_page.revision_id,\n                \"sections\": wiki_page.sections,\n            }\n            if self.load_all_available_meta\n            else {}\n        )\n        doc = Document(\n            page_content=wiki_page.content[: self.doc_content_chars_max],\n            metadata={\n                **main_meta,\n                **add_meta,\n            },\n        )\n        return doc\n    def _fetch_page(self, page: str) -> Optional[str]:\n        try:\n            return self.wiki_client.page(title=page, auto_suggest=False)\n        except (\n            self.wiki_client.exceptions.PageError,", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/wikipedia.html"}623{"id": "467341b6206f-2", "text": "except (\n            self.wiki_client.exceptions.PageError,\n            self.wiki_client.exceptions.DisambiguationError,\n        ):\n            return None\n[docs]    def load(self, query: str) -> List[Document]:\n        \"\"\"\n        Run Wikipedia search and get the article text plus the meta information.\n        See\n        Returns: a list of documents.\n        \"\"\"\n        page_titles = self.wiki_client.search(query[:WIKIPEDIA_MAX_QUERY_LENGTH])\n        docs = []\n        for page_title in page_titles[: self.top_k_results]:\n            if wiki_page := self._fetch_page(page_title):\n                if doc := self._page_to_document(page_title, wiki_page):\n                    docs.append(doc)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/wikipedia.html"}624{"id": "460c719e5711-0", "text": "Source code for langchain.utilities.awslambda\n\"\"\"Util that calls Lambda.\"\"\"\nimport json\nfrom typing import Any, Dict, Optional\nfrom pydantic import BaseModel, Extra, root_validator\n[docs]class LambdaWrapper(BaseModel):\n    \"\"\"Wrapper for AWS Lambda SDK.\n    Docs for using:\n    1. pip install boto3\n    2. Create a lambda function using the AWS Console or CLI\n    3. Run `aws configure` and enter your AWS credentials\n    \"\"\"\n    lambda_client: Any  #: :meta private:\n    function_name: Optional[str] = None\n    awslambda_tool_name: Optional[str] = None\n    awslambda_tool_description: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that python package exists in environment.\"\"\"\n        try:\n            import boto3\n        except ImportError:\n            raise ImportError(\n                \"boto3 is not installed. Please install it with `pip install boto3`\"\n            )\n        values[\"lambda_client\"] = boto3.client(\"lambda\")\n        values[\"function_name\"] = values[\"function_name\"]\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Invoke Lambda function and parse result.\"\"\"\n        res = self.lambda_client.invoke(\n            FunctionName=self.function_name,\n            InvocationType=\"RequestResponse\",\n            Payload=json.dumps({\"body\": query}),\n        )\n        try:\n            payload_stream = res[\"Payload\"]\n            payload_string = payload_stream.read().decode(\"utf-8\")\n            answer = json.loads(payload_string)[\"body\"]\n        except StopIteration:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/awslambda.html"}625{"id": "460c719e5711-1", "text": "answer = json.loads(payload_string)[\"body\"]\n        except StopIteration:\n            return \"Failed to parse response from Lambda\"\n        if answer is None or answer == \"\":\n            # We don't want to return the assumption alone if answer is empty\n            return \"Request failed.\"\n        else:\n            return f\"Result: {answer}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/awslambda.html"}626{"id": "5fff2bc12059-0", "text": "Source code for langchain.utilities.google_serper\n\"\"\"Util that calls Google Search using the Serper.dev API.\"\"\"\nfrom typing import Any, Dict, List, Optional\nimport aiohttp\nimport requests\nfrom pydantic.class_validators import root_validator\nfrom pydantic.main import BaseModel\nfrom typing_extensions import Literal\nfrom langchain.utils import get_from_dict_or_env\n[docs]class GoogleSerperAPIWrapper(BaseModel):\n    \"\"\"Wrapper around the Serper.dev Google Search API.\n    You can create a free API key at https://serper.dev.\n    To use, you should have the environment variable ``SERPER_API_KEY``\n    set with your API key, or pass `serper_api_key` as a named parameter\n    to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain import GoogleSerperAPIWrapper\n            google_serper = GoogleSerperAPIWrapper()\n    \"\"\"\n    k: int = 10\n    gl: str = \"us\"\n    hl: str = \"en\"\n    # \"places\" and \"images\" is available from Serper but not implemented in the\n    # parser of run(). They can be used in results()\n    type: Literal[\"news\", \"search\", \"places\", \"images\"] = \"search\"\n    result_key_for_type = {\n        \"news\": \"news\",\n        \"places\": \"places\",\n        \"images\": \"images\",\n        \"search\": \"organic\",\n    }\n    tbs: Optional[str] = None\n    serper_api_key: Optional[str] = None\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @root_validator()", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_serper.html"}627{"id": "5fff2bc12059-1", "text": "arbitrary_types_allowed = True\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key exists in environment.\"\"\"\n        serper_api_key = get_from_dict_or_env(\n            values, \"serper_api_key\", \"SERPER_API_KEY\"\n        )\n        values[\"serper_api_key\"] = serper_api_key\n        return values\n[docs]    def results(self, query: str, **kwargs: Any) -> Dict:\n        \"\"\"Run query through GoogleSearch.\"\"\"\n        return self._google_serper_api_results(\n            query,\n            gl=self.gl,\n            hl=self.hl,\n            num=self.k,\n            tbs=self.tbs,\n            search_type=self.type,\n            **kwargs,\n        )\n[docs]    def run(self, query: str, **kwargs: Any) -> str:\n        \"\"\"Run query through GoogleSearch and parse result.\"\"\"\n        results = self._google_serper_api_results(\n            query,\n            gl=self.gl,\n            hl=self.hl,\n            num=self.k,\n            tbs=self.tbs,\n            search_type=self.type,\n            **kwargs,\n        )\n        return self._parse_results(results)\n[docs]    async def aresults(self, query: str, **kwargs: Any) -> Dict:\n        \"\"\"Run query through GoogleSearch.\"\"\"\n        results = await self._async_google_serper_search_results(\n            query,\n            gl=self.gl,\n            hl=self.hl,\n            num=self.k,\n            search_type=self.type,\n            tbs=self.tbs,\n            **kwargs,\n        )\n        return results\n[docs]    async def arun(self, query: str, **kwargs: Any) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_serper.html"}628{"id": "5fff2bc12059-2", "text": "\"\"\"Run query through GoogleSearch and parse result async.\"\"\"\n        results = await self._async_google_serper_search_results(\n            query,\n            gl=self.gl,\n            hl=self.hl,\n            num=self.k,\n            search_type=self.type,\n            tbs=self.tbs,\n            **kwargs,\n        )\n        return self._parse_results(results)\n    def _parse_snippets(self, results: dict) -> List[str]:\n        snippets = []\n        if results.get(\"answerBox\"):\n            answer_box = results.get(\"answerBox\", {})\n            if answer_box.get(\"answer\"):\n                return [answer_box.get(\"answer\")]\n            elif answer_box.get(\"snippet\"):\n                return [answer_box.get(\"snippet\").replace(\"\\n\", \" \")]\n            elif answer_box.get(\"snippetHighlighted\"):\n                return answer_box.get(\"snippetHighlighted\")\n        if results.get(\"knowledgeGraph\"):\n            kg = results.get(\"knowledgeGraph\", {})\n            title = kg.get(\"title\")\n            entity_type = kg.get(\"type\")\n            if entity_type:\n                snippets.append(f\"{title}: {entity_type}.\")\n            description = kg.get(\"description\")\n            if description:\n                snippets.append(description)\n            for attribute, value in kg.get(\"attributes\", {}).items():\n                snippets.append(f\"{title} {attribute}: {value}.\")\n        for result in results[self.result_key_for_type[self.type]][: self.k]:\n            if \"snippet\" in result:\n                snippets.append(result[\"snippet\"])\n            for attribute, value in result.get(\"attributes\", {}).items():\n                snippets.append(f\"{attribute}: {value}.\")\n        if len(snippets) == 0:\n            return [\"No good Google Search Result was found\"]\n        return snippets\n    def _parse_results(self, results: dict) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_serper.html"}629{"id": "5fff2bc12059-3", "text": "return snippets\n    def _parse_results(self, results: dict) -> str:\n        return \" \".join(self._parse_snippets(results))\n    def _google_serper_api_results(\n        self, search_term: str, search_type: str = \"search\", **kwargs: Any\n    ) -> dict:\n        headers = {\n            \"X-API-KEY\": self.serper_api_key or \"\",\n            \"Content-Type\": \"application/json\",\n        }\n        params = {\n            \"q\": search_term,\n            **{key: value for key, value in kwargs.items() if value is not None},\n        }\n        response = requests.post(\n            f\"https://google.serper.dev/{search_type}\", headers=headers, params=params\n        )\n        response.raise_for_status()\n        search_results = response.json()\n        return search_results\n    async def _async_google_serper_search_results(\n        self, search_term: str, search_type: str = \"search\", **kwargs: Any\n    ) -> dict:\n        headers = {\n            \"X-API-KEY\": self.serper_api_key or \"\",\n            \"Content-Type\": \"application/json\",\n        }\n        url = f\"https://google.serper.dev/{search_type}\"\n        params = {\n            \"q\": search_term,\n            **{key: value for key, value in kwargs.items() if value is not None},\n        }\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.post(\n                    url, params=params, headers=headers, raise_for_status=False\n                ) as response:\n                    search_results = await response.json()\n        else:\n            async with self.aiosession.post(\n                url, params=params, headers=headers, raise_for_status=True", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_serper.html"}630{"id": "5fff2bc12059-4", "text": "url, params=params, headers=headers, raise_for_status=True\n            ) as response:\n                search_results = await response.json()\n        return search_results\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/google_serper.html"}631{"id": "f60f3b36a9de-0", "text": "Source code for langchain.utilities.wolfram_alpha\n\"\"\"Util that calls WolframAlpha.\"\"\"\nfrom typing import Any, Dict, Optional\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\n[docs]class WolframAlphaAPIWrapper(BaseModel):\n    \"\"\"Wrapper for Wolfram Alpha.\n    Docs for using:\n    1. Go to wolfram alpha and sign up for a developer account\n    2. Create an app and get your APP ID\n    3. Save your APP ID into WOLFRAM_ALPHA_APPID env variable\n    4. pip install wolframalpha\n    \"\"\"\n    wolfram_client: Any  #: :meta private:\n    wolfram_alpha_appid: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        wolfram_alpha_appid = get_from_dict_or_env(\n            values, \"wolfram_alpha_appid\", \"WOLFRAM_ALPHA_APPID\"\n        )\n        values[\"wolfram_alpha_appid\"] = wolfram_alpha_appid\n        try:\n            import wolframalpha\n        except ImportError:\n            raise ImportError(\n                \"wolframalpha is not installed. \"\n                \"Please install it with `pip install wolframalpha`\"\n            )\n        client = wolframalpha.Client(wolfram_alpha_appid)\n        values[\"wolfram_client\"] = client\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run query through WolframAlpha and parse result.\"\"\"\n        res = self.wolfram_client.query(query)", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/wolfram_alpha.html"}632{"id": "f60f3b36a9de-1", "text": "res = self.wolfram_client.query(query)\n        try:\n            assumption = next(res.pods).text\n            answer = next(res.results).text\n        except StopIteration:\n            return \"Wolfram Alpha wasn't able to answer it\"\n        if answer is None or answer == \"\":\n            # We don't want to return the assumption alone if answer is empty\n            return \"No good Wolfram Alpha Result was found\"\n        else:\n            return f\"Assumption: {assumption} \\nAnswer: {answer}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/wolfram_alpha.html"}633{"id": "95563a2d99d6-0", "text": "Source code for langchain.utilities.bing_search\n\"\"\"Util that calls Bing Search.\nIn order to set this up, follow instructions at:\nhttps://levelup.gitconnected.com/api-tutorial-how-to-use-bing-web-search-api-in-python-4165d5592a7e\n\"\"\"\nfrom typing import Dict, List\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\n[docs]class BingSearchAPIWrapper(BaseModel):\n    \"\"\"Wrapper for Bing Search API.\n    In order to set this up, follow instructions at:\n    https://levelup.gitconnected.com/api-tutorial-how-to-use-bing-web-search-api-in-python-4165d5592a7e\n    \"\"\"\n    bing_subscription_key: str\n    bing_search_url: str\n    k: int = 10\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def _bing_search_results(self, search_term: str, count: int) -> List[dict]:\n        headers = {\"Ocp-Apim-Subscription-Key\": self.bing_subscription_key}\n        params = {\n            \"q\": search_term,\n            \"count\": count,\n            \"textDecorations\": True,\n            \"textFormat\": \"HTML\",\n        }\n        response = requests.get(\n            self.bing_search_url, headers=headers, params=params  # type: ignore\n        )\n        response.raise_for_status()\n        search_results = response.json()\n        return search_results[\"webPages\"][\"value\"]\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        bing_subscription_key = get_from_dict_or_env(", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bing_search.html"}634{"id": "95563a2d99d6-1", "text": "bing_subscription_key = get_from_dict_or_env(\n            values, \"bing_subscription_key\", \"BING_SUBSCRIPTION_KEY\"\n        )\n        values[\"bing_subscription_key\"] = bing_subscription_key\n        bing_search_url = get_from_dict_or_env(\n            values,\n            \"bing_search_url\",\n            \"BING_SEARCH_URL\",\n            # default=\"https://api.bing.microsoft.com/v7.0/search\",\n        )\n        values[\"bing_search_url\"] = bing_search_url\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"Run query through BingSearch and parse result.\"\"\"\n        snippets = []\n        results = self._bing_search_results(query, count=self.k)\n        if len(results) == 0:\n            return \"No good Bing Search Result was found\"\n        for result in results:\n            snippets.append(result[\"snippet\"])\n        return \" \".join(snippets)\n[docs]    def results(self, query: str, num_results: int) -> List[Dict]:\n        \"\"\"Run query through BingSearch and return metadata.\n        Args:\n            query: The query to search for.\n            num_results: The number of results to return.\n        Returns:\n            A list of dictionaries with the following keys:\n                snippet - The description of the result.\n                title - The title of the result.\n                link - The link to the result.\n        \"\"\"\n        metadata_results = []\n        results = self._bing_search_results(query, count=num_results)\n        if len(results) == 0:\n            return [{\"Result\": \"No good Bing Search Result was found\"}]\n        for result in results:\n            metadata_result = {\n                \"snippet\": result[\"snippet\"],\n                \"title\": result[\"name\"],", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bing_search.html"}635{"id": "95563a2d99d6-2", "text": "\"snippet\": result[\"snippet\"],\n                \"title\": result[\"name\"],\n                \"link\": result[\"url\"],\n            }\n            metadata_results.append(metadata_result)\n        return metadata_results\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/bing_search.html"}636{"id": "64377c3bb793-0", "text": "Source code for langchain.utilities.powerbi\n\"\"\"Wrapper around a Power BI endpoint.\"\"\"\nfrom __future__ import annotations\nimport asyncio\nimport logging\nimport os\nfrom typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Union\nimport aiohttp\nimport requests\nfrom aiohttp import ServerTimeoutError\nfrom pydantic import BaseModel, Field, root_validator, validator\nfrom requests.exceptions import Timeout\n_LOGGER = logging.getLogger(__name__)\nBASE_URL = os.getenv(\"POWERBI_BASE_URL\", \"https://api.powerbi.com/v1.0/myorg\")\nif TYPE_CHECKING:\n    from azure.core.credentials import TokenCredential\n[docs]class PowerBIDataset(BaseModel):\n    \"\"\"Create PowerBI engine from dataset ID and credential or token.\n    Use either the credential or a supplied token to authenticate.\n    If both are supplied the credential is used to generate a token.\n    The impersonated_user_name is the UPN of a user to be impersonated.\n    If the model is not RLS enabled, this will be ignored.\n    \"\"\"\n    dataset_id: str\n    table_names: List[str]\n    group_id: Optional[str] = None\n    credential: Optional[TokenCredential] = None\n    token: Optional[str] = None\n    impersonated_user_name: Optional[str] = None\n    sample_rows_in_table_info: int = Field(default=1, gt=0, le=10)\n    schemas: Dict[str, str] = Field(default_factory=dict)\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @validator(\"table_names\", allow_reuse=True)\n    def fix_table_names(cls, table_names: List[str]) -> List[str]:\n        \"\"\"Fix the table names.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}637{"id": "64377c3bb793-1", "text": "\"\"\"Fix the table names.\"\"\"\n        return [fix_table_name(table) for table in table_names]\n    @root_validator(pre=True, allow_reuse=True)\n    def token_or_credential_present(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Validate that at least one of token and credentials is present.\"\"\"\n        if \"token\" in values or \"credential\" in values:\n            return values\n        raise ValueError(\"Please provide either a credential or a token.\")\n    @property\n    def request_url(self) -> str:\n        \"\"\"Get the request url.\"\"\"\n        if self.group_id:\n            return f\"{BASE_URL}/groups/{self.group_id}/datasets/{self.dataset_id}/executeQueries\"  # noqa: E501 # pylint: disable=C0301\n        return f\"{BASE_URL}/datasets/{self.dataset_id}/executeQueries\"  # noqa: E501 # pylint: disable=C0301\n    @property\n    def headers(self) -> Dict[str, str]:\n        \"\"\"Get the token.\"\"\"\n        if self.token:\n            return {\n                \"Content-Type\": \"application/json\",\n                \"Authorization\": \"Bearer \" + self.token,\n            }\n        from azure.core.exceptions import (\n            ClientAuthenticationError,  # pylint: disable=import-outside-toplevel\n        )\n        if self.credential:\n            try:\n                token = self.credential.get_token(\n                    \"https://analysis.windows.net/powerbi/api/.default\"\n                ).token\n                return {\n                    \"Content-Type\": \"application/json\",\n                    \"Authorization\": \"Bearer \" + token,\n                }\n            except Exception as exc:  # pylint: disable=broad-exception-caught\n                raise ClientAuthenticationError(\n                    \"Could not get a token from the supplied credentials.\"\n                ) from exc", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}638{"id": "64377c3bb793-2", "text": "\"Could not get a token from the supplied credentials.\"\n                ) from exc\n        raise ClientAuthenticationError(\"No credential or token supplied.\")\n[docs]    def get_table_names(self) -> Iterable[str]:\n        \"\"\"Get names of tables available.\"\"\"\n        return self.table_names\n[docs]    def get_schemas(self) -> str:\n        \"\"\"Get the available schema's.\"\"\"\n        if self.schemas:\n            return \", \".join([f\"{key}: {value}\" for key, value in self.schemas.items()])\n        return \"No known schema's yet. Use the schema_powerbi tool first.\"\n    @property\n    def table_info(self) -> str:\n        \"\"\"Information about all tables in the database.\"\"\"\n        return self.get_table_info()\n    def _get_tables_to_query(\n        self, table_names: Optional[Union[List[str], str]] = None\n    ) -> Optional[List[str]]:\n        \"\"\"Get the tables names that need to be queried, after checking they exist.\"\"\"\n        if table_names is not None:\n            if (\n                isinstance(table_names, list)\n                and len(table_names) > 0\n                and table_names[0] != \"\"\n            ):\n                fixed_tables = [fix_table_name(table) for table in table_names]\n                non_existing_tables = [\n                    table for table in fixed_tables if table not in self.table_names\n                ]\n                if non_existing_tables:\n                    _LOGGER.warning(\n                        \"Table(s) %s not found in dataset.\",\n                        \", \".join(non_existing_tables),\n                    )\n                tables = [\n                    table for table in fixed_tables if table not in non_existing_tables\n                ]\n                return tables if tables else None\n            if isinstance(table_names, str) and table_names != \"\":\n                if table_names not in self.table_names:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}639{"id": "64377c3bb793-3", "text": "if table_names not in self.table_names:\n                    _LOGGER.warning(\"Table %s not found in dataset.\", table_names)\n                    return None\n                return [fix_table_name(table_names)]\n        return self.table_names\n    def _get_tables_todo(self, tables_todo: List[str]) -> List[str]:\n        \"\"\"Get the tables that still need to be queried.\"\"\"\n        return [table for table in tables_todo if table not in self.schemas]\n    def _get_schema_for_tables(self, table_names: List[str]) -> str:\n        \"\"\"Create a string of the table schemas for the supplied tables.\"\"\"\n        schemas = [\n            schema for table, schema in self.schemas.items() if table in table_names\n        ]\n        return \", \".join(schemas)\n[docs]    def get_table_info(\n        self, table_names: Optional[Union[List[str], str]] = None\n    ) -> str:\n        \"\"\"Get information about specified tables.\"\"\"\n        tables_requested = self._get_tables_to_query(table_names)\n        if tables_requested is None:\n            return \"No (valid) tables requested.\"\n        tables_todo = self._get_tables_todo(tables_requested)\n        for table in tables_todo:\n            self._get_schema(table)\n        return self._get_schema_for_tables(tables_requested)\n[docs]    async def aget_table_info(\n        self, table_names: Optional[Union[List[str], str]] = None\n    ) -> str:\n        \"\"\"Get information about specified tables.\"\"\"\n        tables_requested = self._get_tables_to_query(table_names)\n        if tables_requested is None:\n            return \"No (valid) tables requested.\"\n        tables_todo = self._get_tables_todo(tables_requested)\n        await asyncio.gather(*[self._aget_schema(table) for table in tables_todo])", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}640{"id": "64377c3bb793-4", "text": "await asyncio.gather(*[self._aget_schema(table) for table in tables_todo])\n        return self._get_schema_for_tables(tables_requested)\n    def _get_schema(self, table: str) -> None:\n        \"\"\"Get the schema for a table.\"\"\"\n        try:\n            result = self.run(\n                f\"EVALUATE TOPN({self.sample_rows_in_table_info}, {table})\"\n            )\n            self.schemas[table] = json_to_md(result[\"results\"][0][\"tables\"][0][\"rows\"])\n        except Timeout:\n            _LOGGER.warning(\"Timeout while getting table info for %s\", table)\n            self.schemas[table] = \"unknown\"\n        except Exception as exc:  # pylint: disable=broad-exception-caught\n            _LOGGER.warning(\"Error while getting table info for %s: %s\", table, exc)\n            self.schemas[table] = \"unknown\"\n    async def _aget_schema(self, table: str) -> None:\n        \"\"\"Get the schema for a table.\"\"\"\n        try:\n            result = await self.arun(\n                f\"EVALUATE TOPN({self.sample_rows_in_table_info}, {table})\"\n            )\n            self.schemas[table] = json_to_md(result[\"results\"][0][\"tables\"][0][\"rows\"])\n        except ServerTimeoutError:\n            _LOGGER.warning(\"Timeout while getting table info for %s\", table)\n            self.schemas[table] = \"unknown\"\n        except Exception as exc:  # pylint: disable=broad-exception-caught\n            _LOGGER.warning(\"Error while getting table info for %s: %s\", table, exc)\n            self.schemas[table] = \"unknown\"\n    def _create_json_content(self, command: str) -> dict[str, Any]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}641{"id": "64377c3bb793-5", "text": "def _create_json_content(self, command: str) -> dict[str, Any]:\n        \"\"\"Create the json content for the request.\"\"\"\n        return {\n            \"queries\": [{\"query\": rf\"{command}\"}],\n            \"impersonatedUserName\": self.impersonated_user_name,\n            \"serializerSettings\": {\"includeNulls\": True},\n        }\n[docs]    def run(self, command: str) -> Any:\n        \"\"\"Execute a DAX command and return a json representing the results.\"\"\"\n        _LOGGER.debug(\"Running command: %s\", command)\n        result = requests.post(\n            self.request_url,\n            json=self._create_json_content(command),\n            headers=self.headers,\n            timeout=10,\n        )\n        return result.json()\n[docs]    async def arun(self, command: str) -> Any:\n        \"\"\"Execute a DAX command and return the result asynchronously.\"\"\"\n        _LOGGER.debug(\"Running command: %s\", command)\n        if self.aiosession:\n            async with self.aiosession.post(\n                self.request_url,\n                headers=self.headers,\n                json=self._create_json_content(command),\n                timeout=10,\n            ) as response:\n                response_json = await response.json()\n                return response_json\n        async with aiohttp.ClientSession() as session:\n            async with session.post(\n                self.request_url,\n                headers=self.headers,\n                json=self._create_json_content(command),\n                timeout=10,\n            ) as response:\n                response_json = await response.json()\n                return response_json\ndef json_to_md(\n    json_contents: List[Dict[str, Union[str, int, float]]],\n    table_name: Optional[str] = None,\n) -> str:\n    \"\"\"Converts a JSON object to a markdown table.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}642{"id": "64377c3bb793-6", "text": ") -> str:\n    \"\"\"Converts a JSON object to a markdown table.\"\"\"\n    output_md = \"\"\n    headers = json_contents[0].keys()\n    for header in headers:\n        header.replace(\"[\", \".\").replace(\"]\", \"\")\n        if table_name:\n            header.replace(f\"{table_name}.\", \"\")\n        output_md += f\"| {header} \"\n    output_md += \"|\\n\"\n    for row in json_contents:\n        for value in row.values():\n            output_md += f\"| {value} \"\n        output_md += \"|\\n\"\n    return output_md\ndef fix_table_name(table: str) -> str:\n    \"\"\"Add single quotes around table names that contain spaces.\"\"\"\n    if \" \" in table and not table.startswith(\"'\") and not table.endswith(\"'\"):\n        return f\"'{table}'\"\n    return table\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/powerbi.html"}643{"id": "b6c20e01b7c3-0", "text": "Source code for langchain.utilities.openweathermap\n\"\"\"Util that calls OpenWeatherMap using PyOWM.\"\"\"\nfrom typing import Any, Dict, Optional\nfrom pydantic import Extra, root_validator\nfrom langchain.tools.base import BaseModel\nfrom langchain.utils import get_from_dict_or_env\n[docs]class OpenWeatherMapAPIWrapper(BaseModel):\n    \"\"\"Wrapper for OpenWeatherMap API using PyOWM.\n    Docs for using:\n    1. Go to OpenWeatherMap and sign up for an API key\n    2. Save your API KEY into OPENWEATHERMAP_API_KEY env variable\n    3. pip install pyowm\n    \"\"\"\n    owm: Any\n    openweathermap_api_key: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key exists in environment.\"\"\"\n        openweathermap_api_key = get_from_dict_or_env(\n            values, \"openweathermap_api_key\", \"OPENWEATHERMAP_API_KEY\"\n        )\n        try:\n            import pyowm\n        except ImportError:\n            raise ImportError(\n                \"pyowm is not installed. Please install it with `pip install pyowm`\"\n            )\n        owm = pyowm.OWM(openweathermap_api_key)\n        values[\"owm\"] = owm\n        return values\n    def _format_weather_info(self, location: str, w: Any) -> str:\n        detailed_status = w.detailed_status\n        wind = w.wind()\n        humidity = w.humidity\n        temperature = w.temperature(\"celsius\")\n        rain = w.rain\n        heat_index = w.heat_index\n        clouds = w.clouds", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/openweathermap.html"}644{"id": "b6c20e01b7c3-1", "text": "heat_index = w.heat_index\n        clouds = w.clouds\n        return (\n            f\"In {location}, the current weather is as follows:\\n\"\n            f\"Detailed status: {detailed_status}\\n\"\n            f\"Wind speed: {wind['speed']} m/s, direction: {wind['deg']}\u00b0\\n\"\n            f\"Humidity: {humidity}%\\n\"\n            f\"Temperature: \\n\"\n            f\"  - Current: {temperature['temp']}\u00b0C\\n\"\n            f\"  - High: {temperature['temp_max']}\u00b0C\\n\"\n            f\"  - Low: {temperature['temp_min']}\u00b0C\\n\"\n            f\"  - Feels like: {temperature['feels_like']}\u00b0C\\n\"\n            f\"Rain: {rain}\\n\"\n            f\"Heat index: {heat_index}\\n\"\n            f\"Cloud cover: {clouds}%\"\n        )\n[docs]    def run(self, location: str) -> str:\n        \"\"\"Get the current weather information for a specified location.\"\"\"\n        mgr = self.owm.weather_manager()\n        observation = mgr.weather_at_place(location)\n        w = observation.weather\n        return self._format_weather_info(location, w)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/openweathermap.html"}645{"id": "b298705f4e34-0", "text": "Source code for langchain.utilities.metaphor_search\n\"\"\"Util that calls Metaphor Search API.\nIn order to set this up, follow instructions at:\n\"\"\"\nimport json\nfrom typing import Dict, List\nimport aiohttp\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.utils import get_from_dict_or_env\nMETAPHOR_API_URL = \"https://api.metaphor.systems\"\n[docs]class MetaphorSearchAPIWrapper(BaseModel):\n    \"\"\"Wrapper for Metaphor Search API.\"\"\"\n    metaphor_api_key: str\n    k: int = 10\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def _metaphor_search_results(self, query: str, num_results: int) -> List[dict]:\n        headers = {\"X-Api-Key\": self.metaphor_api_key}\n        params = {\"numResults\": num_results, \"query\": query}\n        response = requests.post(\n            # type: ignore\n            f\"{METAPHOR_API_URL}/search\",\n            headers=headers,\n            json=params,\n        )\n        response.raise_for_status()\n        search_results = response.json()\n        print(search_results)\n        return search_results[\"results\"]\n    @root_validator(pre=True)\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and endpoint exists in environment.\"\"\"\n        metaphor_api_key = get_from_dict_or_env(\n            values, \"metaphor_api_key\", \"METAPHOR_API_KEY\"\n        )\n        values[\"metaphor_api_key\"] = metaphor_api_key\n        return values\n[docs]    def results(self, query: str, num_results: int) -> List[Dict]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/metaphor_search.html"}646{"id": "b298705f4e34-1", "text": "\"\"\"Run query through Metaphor Search and return metadata.\n        Args:\n            query: The query to search for.\n            num_results: The number of results to return.\n        Returns:\n            A list of dictionaries with the following keys:\n                title - The title of the\n                url - The url\n                author - Author of the content, if applicable. Otherwise, None.\n                date_created - Estimated date created,\n                    in YYYY-MM-DD format. Otherwise, None.\n        \"\"\"\n        raw_search_results = self._metaphor_search_results(\n            query, num_results=num_results\n        )\n        return self._clean_results(raw_search_results)\n[docs]    async def results_async(self, query: str, num_results: int) -> List[Dict]:\n        \"\"\"Get results from the Metaphor Search API asynchronously.\"\"\"\n        # Function to perform the API call\n        async def fetch() -> str:\n            headers = {\"X-Api-Key\": self.metaphor_api_key}\n            params = {\"numResults\": num_results, \"query\": query}\n            async with aiohttp.ClientSession() as session:\n                async with session.post(\n                    f\"{METAPHOR_API_URL}/search\", json=params, headers=headers\n                ) as res:\n                    if res.status == 200:\n                        data = await res.text()\n                        return data\n                    else:\n                        raise Exception(f\"Error {res.status}: {res.reason}\")\n        results_json_str = await fetch()\n        results_json = json.loads(results_json_str)\n        return self._clean_results(results_json[\"results\"])\n    def _clean_results(self, raw_search_results: List[Dict]) -> List[Dict]:\n        cleaned_results = []\n        for result in raw_search_results:\n            cleaned_results.append(\n                {", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/metaphor_search.html"}647{"id": "b298705f4e34-2", "text": "for result in raw_search_results:\n            cleaned_results.append(\n                {\n                    \"title\": result[\"title\"],\n                    \"url\": result[\"url\"],\n                    \"author\": result[\"author\"],\n                    \"date_created\": result[\"dateCreated\"],\n                }\n            )\n        return cleaned_results\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/metaphor_search.html"}648{"id": "921a4674d2f5-0", "text": "Source code for langchain.utilities.serpapi\n\"\"\"Chain that calls SerpAPI.\nHeavily borrowed from https://github.com/ofirpress/self-ask\n\"\"\"\nimport os\nimport sys\nfrom typing import Any, Dict, Optional, Tuple\nimport aiohttp\nfrom pydantic import BaseModel, Extra, Field, root_validator\nfrom langchain.utils import get_from_dict_or_env\nclass HiddenPrints:\n    \"\"\"Context manager to hide prints.\"\"\"\n    def __enter__(self) -> None:\n        \"\"\"Open file to pipe stdout to.\"\"\"\n        self._original_stdout = sys.stdout\n        sys.stdout = open(os.devnull, \"w\")\n    def __exit__(self, *_: Any) -> None:\n        \"\"\"Close file that stdout was piped to.\"\"\"\n        sys.stdout.close()\n        sys.stdout = self._original_stdout\n[docs]class SerpAPIWrapper(BaseModel):\n    \"\"\"Wrapper around SerpAPI.\n    To use, you should have the ``google-search-results`` python package installed,\n    and the environment variable ``SERPAPI_API_KEY`` set with your API key, or pass\n    `serpapi_api_key` as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain import SerpAPIWrapper\n            serpapi = SerpAPIWrapper()\n    \"\"\"\n    search_engine: Any  #: :meta private:\n    params: dict = Field(\n        default={\n            \"engine\": \"google\",\n            \"google_domain\": \"google.com\",\n            \"gl\": \"us\",\n            \"hl\": \"en\",\n        }\n    )\n    serpapi_api_key: Optional[str] = None\n    aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/serpapi.html"}649{"id": "921a4674d2f5-1", "text": "aiosession: Optional[aiohttp.ClientSession] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        serpapi_api_key = get_from_dict_or_env(\n            values, \"serpapi_api_key\", \"SERPAPI_API_KEY\"\n        )\n        values[\"serpapi_api_key\"] = serpapi_api_key\n        try:\n            from serpapi import GoogleSearch\n            values[\"search_engine\"] = GoogleSearch\n        except ImportError:\n            raise ValueError(\n                \"Could not import serpapi python package. \"\n                \"Please install it with `pip install google-search-results`.\"\n            )\n        return values\n[docs]    async def arun(self, query: str, **kwargs: Any) -> str:\n        \"\"\"Run query through SerpAPI and parse result async.\"\"\"\n        return self._process_response(await self.aresults(query))\n[docs]    def run(self, query: str, **kwargs: Any) -> str:\n        \"\"\"Run query through SerpAPI and parse result.\"\"\"\n        return self._process_response(self.results(query))\n[docs]    def results(self, query: str) -> dict:\n        \"\"\"Run query through SerpAPI and return the raw result.\"\"\"\n        params = self.get_params(query)\n        with HiddenPrints():\n            search = self.search_engine(params)\n            res = search.get_dict()\n        return res\n[docs]    async def aresults(self, query: str) -> dict:\n        \"\"\"Use aiohttp to run query through SerpAPI and return the results async.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/serpapi.html"}650{"id": "921a4674d2f5-2", "text": "\"\"\"Use aiohttp to run query through SerpAPI and return the results async.\"\"\"\n        def construct_url_and_params() -> Tuple[str, Dict[str, str]]:\n            params = self.get_params(query)\n            params[\"source\"] = \"python\"\n            if self.serpapi_api_key:\n                params[\"serp_api_key\"] = self.serpapi_api_key\n            params[\"output\"] = \"json\"\n            url = \"https://serpapi.com/search\"\n            return url, params\n        url, params = construct_url_and_params()\n        if not self.aiosession:\n            async with aiohttp.ClientSession() as session:\n                async with session.get(url, params=params) as response:\n                    res = await response.json()\n        else:\n            async with self.aiosession.get(url, params=params) as response:\n                res = await response.json()\n        return res\n[docs]    def get_params(self, query: str) -> Dict[str, str]:\n        \"\"\"Get parameters for SerpAPI.\"\"\"\n        _params = {\n            \"api_key\": self.serpapi_api_key,\n            \"q\": query,\n        }\n        params = {**self.params, **_params}\n        return params\n    @staticmethod\n    def _process_response(res: dict) -> str:\n        \"\"\"Process response from SerpAPI.\"\"\"\n        if \"error\" in res.keys():\n            raise ValueError(f\"Got error from SerpAPI: {res['error']}\")\n        if \"answer_box\" in res.keys() and \"answer\" in res[\"answer_box\"].keys():\n            toret = res[\"answer_box\"][\"answer\"]\n        elif \"answer_box\" in res.keys() and \"snippet\" in res[\"answer_box\"].keys():\n            toret = res[\"answer_box\"][\"snippet\"]", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/serpapi.html"}651{"id": "921a4674d2f5-3", "text": "toret = res[\"answer_box\"][\"snippet\"]\n        elif (\n            \"answer_box\" in res.keys()\n            and \"snippet_highlighted_words\" in res[\"answer_box\"].keys()\n        ):\n            toret = res[\"answer_box\"][\"snippet_highlighted_words\"][0]\n        elif (\n            \"sports_results\" in res.keys()\n            and \"game_spotlight\" in res[\"sports_results\"].keys()\n        ):\n            toret = res[\"sports_results\"][\"game_spotlight\"]\n        elif (\n            \"shopping_results\" in res.keys()\n            and \"title\" in res[\"shopping_results\"][0].keys()\n        ):\n            toret = res[\"shopping_results\"][:3]\n        elif (\n            \"knowledge_graph\" in res.keys()\n            and \"description\" in res[\"knowledge_graph\"].keys()\n        ):\n            toret = res[\"knowledge_graph\"][\"description\"]\n        elif \"snippet\" in res[\"organic_results\"][0].keys():\n            toret = res[\"organic_results\"][0][\"snippet\"]\n        elif \"link\" in res[\"organic_results\"][0].keys():\n            toret = res[\"organic_results\"][0][\"link\"]\n        else:\n            toret = \"No good search result found\"\n        return toret\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/serpapi.html"}652{"id": "9c0e129e090f-0", "text": "Source code for langchain.utilities.arxiv\n\"\"\"Util that calls Arxiv.\"\"\"\nimport logging\nimport os\nfrom typing import Any, Dict, List\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.schema import Document\nlogger = logging.getLogger(__name__)\n[docs]class ArxivAPIWrapper(BaseModel):\n    \"\"\"Wrapper around ArxivAPI.\n    To use, you should have the ``arxiv`` python package installed.\n    https://lukasschwab.me/arxiv.py/index.html\n    This wrapper will use the Arxiv API to conduct searches and\n    fetch document summaries. By default, it will return the document summaries\n    of the top-k results.\n    It limits the Document content by doc_content_chars_max.\n    Set doc_content_chars_max=None if you don't want to limit the content size.\n    Parameters:\n        top_k_results: number of the top-scored document used for the arxiv tool\n        ARXIV_MAX_QUERY_LENGTH: the cut limit on the query used for the arxiv tool.\n        load_max_docs: a limit to the number of loaded documents\n        load_all_available_meta:\n          if True: the `metadata` of the loaded Documents gets all available meta info\n            (see https://lukasschwab.me/arxiv.py/index.html#Result),\n          if False: the `metadata` gets only the most informative fields.\n    \"\"\"\n    arxiv_client: Any  #: :meta private:\n    arxiv_exceptions: Any  # :meta private:\n    top_k_results: int = 3\n    ARXIV_MAX_QUERY_LENGTH = 300\n    load_max_docs: int = 100\n    load_all_available_meta: bool = False\n    doc_content_chars_max: int = 4000\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/arxiv.html"}653{"id": "9c0e129e090f-1", "text": "class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that the python package exists in environment.\"\"\"\n        try:\n            import arxiv\n            values[\"arxiv_search\"] = arxiv.Search\n            values[\"arxiv_exceptions\"] = (\n                arxiv.ArxivError,\n                arxiv.UnexpectedEmptyPageError,\n                arxiv.HTTPError,\n            )\n            values[\"arxiv_result\"] = arxiv.Result\n        except ImportError:\n            raise ImportError(\n                \"Could not import arxiv python package. \"\n                \"Please install it with `pip install arxiv`.\"\n            )\n        return values\n[docs]    def run(self, query: str) -> str:\n        \"\"\"\n        Run Arxiv search and get the article meta information.\n        See https://lukasschwab.me/arxiv.py/index.html#Search\n        See https://lukasschwab.me/arxiv.py/index.html#Result\n        It uses only the most informative fields of article meta information.\n        \"\"\"\n        try:\n            results = self.arxiv_search(  # type: ignore\n                query[: self.ARXIV_MAX_QUERY_LENGTH], max_results=self.top_k_results\n            ).results()\n        except self.arxiv_exceptions as ex:\n            return f\"Arxiv exception: {ex}\"\n        docs = [\n            f\"Published: {result.updated.date()}\\nTitle: {result.title}\\n\"\n            f\"Authors: {', '.join(a.name for a in result.authors)}\\n\"\n            f\"Summary: {result.summary}\"\n            for result in results\n        ]\n        if docs:", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/arxiv.html"}654{"id": "9c0e129e090f-2", "text": "for result in results\n        ]\n        if docs:\n            return \"\\n\\n\".join(docs)[: self.doc_content_chars_max]\n        else:\n            return \"No good Arxiv Result was found\"\n[docs]    def load(self, query: str) -> List[Document]:\n        \"\"\"\n        Run Arxiv search and get the article texts plus the article meta information.\n        See https://lukasschwab.me/arxiv.py/index.html#Search\n        Returns: a list of documents with the document.page_content in text format\n        \"\"\"\n        try:\n            import fitz\n        except ImportError:\n            raise ImportError(\n                \"PyMuPDF package not found, please install it with \"\n                \"`pip install pymupdf`\"\n            )\n        try:\n            results = self.arxiv_search(  # type: ignore\n                query[: self.ARXIV_MAX_QUERY_LENGTH], max_results=self.load_max_docs\n            ).results()\n        except self.arxiv_exceptions as ex:\n            logger.debug(\"Error on arxiv: %s\", ex)\n            return []\n        docs: List[Document] = []\n        for result in results:\n            try:\n                doc_file_name: str = result.download_pdf()\n                with fitz.open(doc_file_name) as doc_file:\n                    text: str = \"\".join(page.get_text() for page in doc_file)\n            except FileNotFoundError as f_ex:\n                logger.debug(f_ex)\n                continue\n            if self.load_all_available_meta:\n                extra_metadata = {\n                    \"entry_id\": result.entry_id,\n                    \"published_first_time\": str(result.published.date()),\n                    \"comment\": result.comment,\n                    \"journal_ref\": result.journal_ref,\n                    \"doi\": result.doi,\n                    \"primary_category\": result.primary_category,", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/arxiv.html"}655{"id": "9c0e129e090f-3", "text": "\"doi\": result.doi,\n                    \"primary_category\": result.primary_category,\n                    \"categories\": result.categories,\n                    \"links\": [link.href for link in result.links],\n                }\n            else:\n                extra_metadata = {}\n            metadata = {\n                \"Published\": str(result.updated.date()),\n                \"Title\": result.title,\n                \"Authors\": \", \".join(a.name for a in result.authors),\n                \"Summary\": result.summary,\n                **extra_metadata,\n            }\n            doc = Document(\n                page_content=text[: self.doc_content_chars_max], metadata=metadata\n            )\n            docs.append(doc)\n            os.remove(doc_file_name)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/utilities/arxiv.html"}656{"id": "c0493d7648ec-0", "text": "Source code for langchain.vectorstores.opensearch_vector_search\n\"\"\"Wrapper around OpenSearch vector database.\"\"\"\nfrom __future__ import annotations\nimport uuid\nfrom typing import Any, Dict, Iterable, List, Optional, Tuple\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nfrom langchain.vectorstores.base import VectorStore\nIMPORT_OPENSEARCH_PY_ERROR = (\n    \"Could not import OpenSearch. Please install it with `pip install opensearch-py`.\"\n)\nSCRIPT_SCORING_SEARCH = \"script_scoring\"\nPAINLESS_SCRIPTING_SEARCH = \"painless_scripting\"\nMATCH_ALL_QUERY = {\"match_all\": {}}  # type: Dict\ndef _import_opensearch() -> Any:\n    \"\"\"Import OpenSearch if available, otherwise raise error.\"\"\"\n    try:\n        from opensearchpy import OpenSearch\n    except ImportError:\n        raise ValueError(IMPORT_OPENSEARCH_PY_ERROR)\n    return OpenSearch\ndef _import_bulk() -> Any:\n    \"\"\"Import bulk if available, otherwise raise error.\"\"\"\n    try:\n        from opensearchpy.helpers import bulk\n    except ImportError:\n        raise ValueError(IMPORT_OPENSEARCH_PY_ERROR)\n    return bulk\ndef _import_not_found_error() -> Any:\n    \"\"\"Import not found error if available, otherwise raise error.\"\"\"\n    try:\n        from opensearchpy.exceptions import NotFoundError\n    except ImportError:\n        raise ValueError(IMPORT_OPENSEARCH_PY_ERROR)\n    return NotFoundError\ndef _get_opensearch_client(opensearch_url: str, **kwargs: Any) -> Any:\n    \"\"\"Get OpenSearch client from the opensearch_url, otherwise raise error.\"\"\"\n    try:\n        opensearch = _import_opensearch()", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}657{"id": "c0493d7648ec-1", "text": "try:\n        opensearch = _import_opensearch()\n        client = opensearch(opensearch_url, **kwargs)\n    except ValueError as e:\n        raise ValueError(\n            f\"OpenSearch client string provided is not in proper format. \"\n            f\"Got error: {e} \"\n        )\n    return client\ndef _validate_embeddings_and_bulk_size(embeddings_length: int, bulk_size: int) -> None:\n    \"\"\"Validate Embeddings Length and Bulk Size.\"\"\"\n    if embeddings_length == 0:\n        raise RuntimeError(\"Embeddings size is zero\")\n    if bulk_size < embeddings_length:\n        raise RuntimeError(\n            f\"The embeddings count, {embeddings_length} is more than the \"\n            f\"[bulk_size], {bulk_size}. Increase the value of [bulk_size].\"\n        )\ndef _bulk_ingest_embeddings(\n    client: Any,\n    index_name: str,\n    embeddings: List[List[float]],\n    texts: Iterable[str],\n    metadatas: Optional[List[dict]] = None,\n    vector_field: str = \"vector_field\",\n    text_field: str = \"text\",\n    mapping: Dict = {},\n) -> List[str]:\n    \"\"\"Bulk Ingest Embeddings into given index.\"\"\"\n    bulk = _import_bulk()\n    not_found_error = _import_not_found_error()\n    requests = []\n    ids = []\n    mapping = mapping\n    try:\n        client.indices.get(index=index_name)\n    except not_found_error:\n        client.indices.create(index=index_name, body=mapping)\n    for i, text in enumerate(texts):\n        metadata = metadatas[i] if metadatas else {}\n        _id = str(uuid.uuid4())\n        request = {\n            \"_op_type\": \"index\",", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}658{"id": "c0493d7648ec-2", "text": "request = {\n            \"_op_type\": \"index\",\n            \"_index\": index_name,\n            vector_field: embeddings[i],\n            text_field: text,\n            \"metadata\": metadata,\n            \"_id\": _id,\n        }\n        requests.append(request)\n        ids.append(_id)\n    bulk(client, requests)\n    client.indices.refresh(index=index_name)\n    return ids\ndef _default_scripting_text_mapping(\n    dim: int,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Painless Scripting or Script Scoring,the default mapping to create index.\"\"\"\n    return {\n        \"mappings\": {\n            \"properties\": {\n                vector_field: {\"type\": \"knn_vector\", \"dimension\": dim},\n            }\n        }\n    }\ndef _default_text_mapping(\n    dim: int,\n    engine: str = \"nmslib\",\n    space_type: str = \"l2\",\n    ef_search: int = 512,\n    ef_construction: int = 512,\n    m: int = 16,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Approximate k-NN Search, this is the default mapping to create index.\"\"\"\n    return {\n        \"settings\": {\"index\": {\"knn\": True, \"knn.algo_param.ef_search\": ef_search}},\n        \"mappings\": {\n            \"properties\": {\n                vector_field: {\n                    \"type\": \"knn_vector\",\n                    \"dimension\": dim,\n                    \"method\": {\n                        \"name\": \"hnsw\",\n                        \"space_type\": space_type,\n                        \"engine\": engine,\n                        \"parameters\": {\"ef_construction\": ef_construction, \"m\": m},", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}659{"id": "c0493d7648ec-3", "text": "\"parameters\": {\"ef_construction\": ef_construction, \"m\": m},\n                    },\n                }\n            }\n        },\n    }\ndef _default_approximate_search_query(\n    query_vector: List[float],\n    k: int = 4,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Approximate k-NN Search, this is the default query.\"\"\"\n    return {\n        \"size\": k,\n        \"query\": {\"knn\": {vector_field: {\"vector\": query_vector, \"k\": k}}},\n    }\ndef _approximate_search_query_with_boolean_filter(\n    query_vector: List[float],\n    boolean_filter: Dict,\n    k: int = 4,\n    vector_field: str = \"vector_field\",\n    subquery_clause: str = \"must\",\n) -> Dict:\n    \"\"\"For Approximate k-NN Search, with Boolean Filter.\"\"\"\n    return {\n        \"size\": k,\n        \"query\": {\n            \"bool\": {\n                \"filter\": boolean_filter,\n                subquery_clause: [\n                    {\"knn\": {vector_field: {\"vector\": query_vector, \"k\": k}}}\n                ],\n            }\n        },\n    }\ndef _approximate_search_query_with_lucene_filter(\n    query_vector: List[float],\n    lucene_filter: Dict,\n    k: int = 4,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Approximate k-NN Search, with Lucene Filter.\"\"\"\n    search_query = _default_approximate_search_query(\n        query_vector, k=k, vector_field=vector_field\n    )\n    search_query[\"query\"][\"knn\"][vector_field][\"filter\"] = lucene_filter\n    return search_query", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}660{"id": "c0493d7648ec-4", "text": "return search_query\ndef _default_script_query(\n    query_vector: List[float],\n    space_type: str = \"l2\",\n    pre_filter: Dict = MATCH_ALL_QUERY,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Script Scoring Search, this is the default query.\"\"\"\n    return {\n        \"query\": {\n            \"script_score\": {\n                \"query\": pre_filter,\n                \"script\": {\n                    \"source\": \"knn_score\",\n                    \"lang\": \"knn\",\n                    \"params\": {\n                        \"field\": vector_field,\n                        \"query_value\": query_vector,\n                        \"space_type\": space_type,\n                    },\n                },\n            }\n        }\n    }\ndef __get_painless_scripting_source(\n    space_type: str, query_vector: List[float], vector_field: str = \"vector_field\"\n) -> str:\n    \"\"\"For Painless Scripting, it returns the script source based on space type.\"\"\"\n    source_value = (\n        \"(1.0 + \"\n        + space_type\n        + \"(\"\n        + str(query_vector)\n        + \", doc['\"\n        + vector_field\n        + \"']))\"\n    )\n    if space_type == \"cosineSimilarity\":\n        return source_value\n    else:\n        return \"1/\" + source_value\ndef _default_painless_scripting_query(\n    query_vector: List[float],\n    space_type: str = \"l2Squared\",\n    pre_filter: Dict = MATCH_ALL_QUERY,\n    vector_field: str = \"vector_field\",\n) -> Dict:\n    \"\"\"For Painless Scripting Search, this is the default query.\"\"\"\n    source = __get_painless_scripting_source(space_type, query_vector)\n    return {", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}661{"id": "c0493d7648ec-5", "text": "source = __get_painless_scripting_source(space_type, query_vector)\n    return {\n        \"query\": {\n            \"script_score\": {\n                \"query\": pre_filter,\n                \"script\": {\n                    \"source\": source,\n                    \"params\": {\n                        \"field\": vector_field,\n                        \"query_value\": query_vector,\n                    },\n                },\n            }\n        }\n    }\ndef _get_kwargs_value(kwargs: Any, key: str, default_value: Any) -> Any:\n    \"\"\"Get the value of the key if present. Else get the default_value.\"\"\"\n    if key in kwargs:\n        return kwargs.get(key)\n    return default_value\n[docs]class OpenSearchVectorSearch(VectorStore):\n    \"\"\"Wrapper around OpenSearch as a vector database.\n    Example:\n        .. code-block:: python\n            from langchain import OpenSearchVectorSearch\n            opensearch_vector_search = OpenSearchVectorSearch(\n                \"http://localhost:9200\",\n                \"embeddings\",\n                embedding_function\n            )\n    \"\"\"\n    def __init__(\n        self,\n        opensearch_url: str,\n        index_name: str,\n        embedding_function: Embeddings,\n        **kwargs: Any,\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        self.embedding_function = embedding_function\n        self.index_name = index_name\n        self.client = _get_opensearch_client(opensearch_url, **kwargs)\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        bulk_size: int = 500,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}662{"id": "c0493d7648ec-6", "text": "\"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            bulk_size: Bulk API request count; Default: 500\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        Optional Args:\n            vector_field: Document field embeddings are stored in. Defaults to\n            \"vector_field\".\n            text_field: Document field the text of the document is stored in. Defaults\n            to \"text\".\n        \"\"\"\n        embeddings = self.embedding_function.embed_documents(list(texts))\n        _validate_embeddings_and_bulk_size(len(embeddings), bulk_size)\n        text_field = _get_kwargs_value(kwargs, \"text_field\", \"text\")\n        dim = len(embeddings[0])\n        engine = _get_kwargs_value(kwargs, \"engine\", \"nmslib\")\n        space_type = _get_kwargs_value(kwargs, \"space_type\", \"l2\")\n        ef_search = _get_kwargs_value(kwargs, \"ef_search\", 512)\n        ef_construction = _get_kwargs_value(kwargs, \"ef_construction\", 512)\n        m = _get_kwargs_value(kwargs, \"m\", 16)\n        vector_field = _get_kwargs_value(kwargs, \"vector_field\", \"vector_field\")\n        mapping = _default_text_mapping(\n            dim, engine, space_type, ef_search, ef_construction, m, vector_field\n        )\n        return _bulk_ingest_embeddings(\n            self.client,\n            self.index_name,\n            embeddings,\n            texts,\n            metadatas,\n            vector_field,\n            text_field,\n            mapping,\n        )\n[docs]    def similarity_search(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}663{"id": "c0493d7648ec-7", "text": "text_field,\n            mapping,\n        )\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        By default supports Approximate Search.\n        Also supports Script Scoring and Painless Scripting.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        Optional Args:\n            vector_field: Document field embeddings are stored in. Defaults to\n            \"vector_field\".\n            text_field: Document field the text of the document is stored in. Defaults\n            to \"text\".\n            metadata_field: Document field that metadata is stored in. Defaults to\n            \"metadata\".\n            Can be set to a special value \"*\" to include the entire document.\n        Optional Args for Approximate Search:\n            search_type: \"approximate_search\"; default: \"approximate_search\"\n            boolean_filter: A Boolean filter consists of a Boolean query that\n            contains a k-NN query and a filter.\n            subquery_clause: Query clause on the knn vector field; default: \"must\"\n            lucene_filter: the Lucene algorithm decides whether to perform an exact\n            k-NN search with pre-filtering or an approximate search with modified\n            post-filtering.\n        Optional Args for Script Scoring Search:\n            search_type: \"script_scoring\"; default: \"approximate_search\"\n            space_type: \"l2\", \"l1\", \"linf\", \"cosinesimil\", \"innerproduct\",\n            \"hammingbit\"; default: \"l2\"\n            pre_filter: script_score query to pre-filter documents before identifying", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}664{"id": "c0493d7648ec-8", "text": "pre_filter: script_score query to pre-filter documents before identifying\n            nearest neighbors; default: {\"match_all\": {}}\n        Optional Args for Painless Scripting Search:\n            search_type: \"painless_scripting\"; default: \"approximate_search\"\n            space_type: \"l2Squared\", \"l1Norm\", \"cosineSimilarity\"; default: \"l2Squared\"\n            pre_filter: script_score query to pre-filter documents before identifying\n            nearest neighbors; default: {\"match_all\": {}}\n        \"\"\"\n        docs_with_scores = self.similarity_search_with_score(query, k, **kwargs)\n        return [doc[0] for doc in docs_with_scores]\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and it's scores most similar to query.\n        By default supports Approximate Search.\n        Also supports Script Scoring and Painless Scripting.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents along with its scores most similar to the query.\n        Optional Args:\n            same as `similarity_search`\n        \"\"\"\n        embedding = self.embedding_function.embed_query(query)\n        search_type = _get_kwargs_value(kwargs, \"search_type\", \"approximate_search\")\n        text_field = _get_kwargs_value(kwargs, \"text_field\", \"text\")\n        metadata_field = _get_kwargs_value(kwargs, \"metadata_field\", \"metadata\")\n        vector_field = _get_kwargs_value(kwargs, \"vector_field\", \"vector_field\")\n        if search_type == \"approximate_search\":", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}665{"id": "c0493d7648ec-9", "text": "if search_type == \"approximate_search\":\n            boolean_filter = _get_kwargs_value(kwargs, \"boolean_filter\", {})\n            subquery_clause = _get_kwargs_value(kwargs, \"subquery_clause\", \"must\")\n            lucene_filter = _get_kwargs_value(kwargs, \"lucene_filter\", {})\n            if boolean_filter != {} and lucene_filter != {}:\n                raise ValueError(\n                    \"Both `boolean_filter` and `lucene_filter` are provided which \"\n                    \"is invalid\"\n                )\n            if boolean_filter != {}:\n                search_query = _approximate_search_query_with_boolean_filter(\n                    embedding,\n                    boolean_filter,\n                    k=k,\n                    vector_field=vector_field,\n                    subquery_clause=subquery_clause,\n                )\n            elif lucene_filter != {}:\n                search_query = _approximate_search_query_with_lucene_filter(\n                    embedding, lucene_filter, k=k, vector_field=vector_field\n                )\n            else:\n                search_query = _default_approximate_search_query(\n                    embedding, k=k, vector_field=vector_field\n                )\n        elif search_type == SCRIPT_SCORING_SEARCH:\n            space_type = _get_kwargs_value(kwargs, \"space_type\", \"l2\")\n            pre_filter = _get_kwargs_value(kwargs, \"pre_filter\", MATCH_ALL_QUERY)\n            search_query = _default_script_query(\n                embedding, space_type, pre_filter, vector_field\n            )\n        elif search_type == PAINLESS_SCRIPTING_SEARCH:\n            space_type = _get_kwargs_value(kwargs, \"space_type\", \"l2Squared\")\n            pre_filter = _get_kwargs_value(kwargs, \"pre_filter\", MATCH_ALL_QUERY)\n            search_query = _default_painless_scripting_query(\n                embedding, space_type, pre_filter, vector_field\n            )\n        else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}666{"id": "c0493d7648ec-10", "text": "embedding, space_type, pre_filter, vector_field\n            )\n        else:\n            raise ValueError(\"Invalid `search_type` provided as an argument\")\n        response = self.client.search(index=self.index_name, body=search_query)\n        hits = [hit for hit in response[\"hits\"][\"hits\"][:k]]\n        documents_with_scores = [\n            (\n                Document(\n                    page_content=hit[\"_source\"][text_field],\n                    metadata=hit[\"_source\"]\n                    if metadata_field == \"*\" or metadata_field not in hit[\"_source\"]\n                    else hit[\"_source\"][metadata_field],\n                ),\n                hit[\"_score\"],\n            )\n            for hit in hits\n        ]\n        return documents_with_scores\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        bulk_size: int = 500,\n        **kwargs: Any,\n    ) -> OpenSearchVectorSearch:\n        \"\"\"Construct OpenSearchVectorSearch wrapper from raw documents.\n        Example:\n            .. code-block:: python\n                from langchain import OpenSearchVectorSearch\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                opensearch_vector_search = OpenSearchVectorSearch.from_texts(\n                    texts,\n                    embeddings,\n                    opensearch_url=\"http://localhost:9200\"\n                )\n        OpenSearch by default supports Approximate Search powered by nmslib, faiss\n        and lucene engines recommended for large datasets. Also supports brute force\n        search through Script Scoring and Painless Scripting.\n        Optional Args:\n            vector_field: Document field embeddings are stored in. Defaults to\n            \"vector_field\".", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}667{"id": "c0493d7648ec-11", "text": "vector_field: Document field embeddings are stored in. Defaults to\n            \"vector_field\".\n            text_field: Document field the text of the document is stored in. Defaults\n            to \"text\".\n        Optional Keyword Args for Approximate Search:\n            engine: \"nmslib\", \"faiss\", \"lucene\"; default: \"nmslib\"\n            space_type: \"l2\", \"l1\", \"cosinesimil\", \"linf\", \"innerproduct\"; default: \"l2\"\n            ef_search: Size of the dynamic list used during k-NN searches. Higher values\n            lead to more accurate but slower searches; default: 512\n            ef_construction: Size of the dynamic list used during k-NN graph creation.\n            Higher values lead to more accurate graph but slower indexing speed;\n            default: 512\n            m: Number of bidirectional links created for each new element. Large impact\n            on memory consumption. Between 2 and 100; default: 16\n        Keyword Args for Script Scoring or Painless Scripting:\n            is_appx_search: False\n        \"\"\"\n        opensearch_url = get_from_dict_or_env(\n            kwargs, \"opensearch_url\", \"OPENSEARCH_URL\"\n        )\n        # List of arguments that needs to be removed from kwargs\n        # before passing kwargs to get opensearch client\n        keys_list = [\n            \"opensearch_url\",\n            \"index_name\",\n            \"is_appx_search\",\n            \"vector_field\",\n            \"text_field\",\n            \"engine\",\n            \"space_type\",\n            \"ef_search\",\n            \"ef_construction\",\n            \"m\",\n        ]\n        embeddings = embedding.embed_documents(texts)\n        _validate_embeddings_and_bulk_size(len(embeddings), bulk_size)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}668{"id": "c0493d7648ec-12", "text": "_validate_embeddings_and_bulk_size(len(embeddings), bulk_size)\n        dim = len(embeddings[0])\n        # Get the index name from either from kwargs or ENV Variable\n        # before falling back to random generation\n        index_name = get_from_dict_or_env(\n            kwargs, \"index_name\", \"OPENSEARCH_INDEX_NAME\", default=uuid.uuid4().hex\n        )\n        is_appx_search = _get_kwargs_value(kwargs, \"is_appx_search\", True)\n        vector_field = _get_kwargs_value(kwargs, \"vector_field\", \"vector_field\")\n        text_field = _get_kwargs_value(kwargs, \"text_field\", \"text\")\n        if is_appx_search:\n            engine = _get_kwargs_value(kwargs, \"engine\", \"nmslib\")\n            space_type = _get_kwargs_value(kwargs, \"space_type\", \"l2\")\n            ef_search = _get_kwargs_value(kwargs, \"ef_search\", 512)\n            ef_construction = _get_kwargs_value(kwargs, \"ef_construction\", 512)\n            m = _get_kwargs_value(kwargs, \"m\", 16)\n            mapping = _default_text_mapping(\n                dim, engine, space_type, ef_search, ef_construction, m, vector_field\n            )\n        else:\n            mapping = _default_scripting_text_mapping(dim)\n        [kwargs.pop(key, None) for key in keys_list]\n        client = _get_opensearch_client(opensearch_url, **kwargs)\n        _bulk_ingest_embeddings(\n            client,\n            index_name,\n            embeddings,\n            texts,\n            metadatas,\n            vector_field,\n            text_field,\n            mapping,\n        )\n        return cls(opensearch_url, index_name, embedding, **kwargs)\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}669{"id": "c0493d7648ec-13", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/opensearch_vector_search.html"}670{"id": "1e3993815549-0", "text": "Source code for langchain.vectorstores.faiss\n\"\"\"Wrapper around FAISS vector database.\"\"\"\nfrom __future__ import annotations\nimport math\nimport os\nimport pickle\nimport uuid\nfrom pathlib import Path\nfrom typing import Any, Callable, Dict, Iterable, List, Optional, Tuple\nimport numpy as np\nfrom langchain.docstore.base import AddableMixin, Docstore\nfrom langchain.docstore.document import Document\nfrom langchain.docstore.in_memory import InMemoryDocstore\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\ndef dependable_faiss_import(no_avx2: Optional[bool] = None) -> Any:\n    \"\"\"\n    Import faiss if available, otherwise raise error.\n    If FAISS_NO_AVX2 environment variable is set, it will be considered\n    to load FAISS with no AVX2 optimization.\n    Args:\n        no_avx2: Load FAISS strictly with no AVX2 optimization\n            so that the vectorstore is portable and compatible with other devices.\n    \"\"\"\n    if no_avx2 is None and \"FAISS_NO_AVX2\" in os.environ:\n        no_avx2 = bool(os.getenv(\"FAISS_NO_AVX2\"))\n    try:\n        if no_avx2:\n            from faiss import swigfaiss as faiss\n        else:\n            import faiss\n    except ImportError:\n        raise ValueError(\n            \"Could not import faiss python package. \"\n            \"Please install it with `pip install faiss` \"\n            \"or `pip install faiss-cpu` (depending on Python version).\"\n        )\n    return faiss\ndef _default_relevance_score_fn(score: float) -> float:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}671{"id": "1e3993815549-1", "text": "return faiss\ndef _default_relevance_score_fn(score: float) -> float:\n    \"\"\"Return a similarity score on a scale [0, 1].\"\"\"\n    # The 'correct' relevance function\n    # may differ depending on a few things, including:\n    # - the distance / similarity metric used by the VectorStore\n    # - the scale of your embeddings (OpenAI's are unit normed. Many others are not!)\n    # - embedding dimensionality\n    # - etc.\n    # This function converts the euclidean norm of normalized embeddings\n    # (0 is most similar, sqrt(2) most dissimilar)\n    # to a similarity function (0 to 1)\n    return 1.0 - score / math.sqrt(2)\n[docs]class FAISS(VectorStore):\n    \"\"\"Wrapper around FAISS vector database.\n    To use, you should have the ``faiss`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain import FAISS\n            faiss = FAISS(embedding_function, index, docstore, index_to_docstore_id)\n    \"\"\"\n    def __init__(\n        self,\n        embedding_function: Callable,\n        index: Any,\n        docstore: Docstore,\n        index_to_docstore_id: Dict[int, str],\n        relevance_score_fn: Optional[\n            Callable[[float], float]\n        ] = _default_relevance_score_fn,\n        normalize_L2: bool = False,\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        self.embedding_function = embedding_function\n        self.index = index\n        self.docstore = docstore\n        self.index_to_docstore_id = index_to_docstore_id\n        self.relevance_score_fn = relevance_score_fn\n        self._normalize_L2 = normalize_L2", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}672{"id": "1e3993815549-2", "text": "self._normalize_L2 = normalize_L2\n    def __add(\n        self,\n        texts: Iterable[str],\n        embeddings: Iterable[List[float]],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        if not isinstance(self.docstore, AddableMixin):\n            raise ValueError(\n                \"If trying to add texts, the underlying docstore should support \"\n                f\"adding items, which {self.docstore} does not\"\n            )\n        documents = []\n        for i, text in enumerate(texts):\n            metadata = metadatas[i] if metadatas else {}\n            documents.append(Document(page_content=text, metadata=metadata))\n        if ids is None:\n            ids = [str(uuid.uuid4()) for _ in texts]\n        # Add to the index, the index_to_id mapping, and the docstore.\n        starting_len = len(self.index_to_docstore_id)\n        faiss = dependable_faiss_import()\n        vector = np.array(embeddings, dtype=np.float32)\n        if self._normalize_L2:\n            faiss.normalize_L2(vector)\n        self.index.add(vector)\n        # Get list of index, id, and docs.\n        full_info = [(starting_len + i, ids[i], doc) for i, doc in enumerate(documents)]\n        # Add information to docstore and index.\n        self.docstore.add({_id: doc for _, _id, doc in full_info})\n        index_to_id = {index: _id for index, _id, _ in full_info}\n        self.index_to_docstore_id.update(index_to_id)\n        return [_id for _, _id, _ in full_info]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}673{"id": "1e3993815549-3", "text": "return [_id for _, _id, _ in full_info]\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            ids: Optional list of unique IDs.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        if not isinstance(self.docstore, AddableMixin):\n            raise ValueError(\n                \"If trying to add texts, the underlying docstore should support \"\n                f\"adding items, which {self.docstore} does not\"\n            )\n        # Embed and create the documents.\n        embeddings = [self.embedding_function(text) for text in texts]\n        return self.__add(texts, embeddings, metadatas=metadatas, ids=ids, **kwargs)\n[docs]    def add_embeddings(\n        self,\n        text_embeddings: Iterable[Tuple[str, List[float]]],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            text_embeddings: Iterable pairs of string and embedding to\n                add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            ids: Optional list of unique IDs.\n        Returns:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}674{"id": "1e3993815549-4", "text": "ids: Optional list of unique IDs.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        if not isinstance(self.docstore, AddableMixin):\n            raise ValueError(\n                \"If trying to add texts, the underlying docstore should support \"\n                f\"adding items, which {self.docstore} does not\"\n            )\n        # Embed and create the documents.\n        texts, embeddings = zip(*text_embeddings)\n        return self.__add(texts, embeddings, metadatas=metadatas, ids=ids, **kwargs)\n[docs]    def similarity_search_with_score_by_vector(\n        self, embedding: List[float], k: int = 4\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            embedding: Embedding vector to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        faiss = dependable_faiss_import()\n        vector = np.array([embedding], dtype=np.float32)\n        if self._normalize_L2:\n            faiss.normalize_L2(vector)\n        scores, indices = self.index.search(vector, k)\n        docs = []\n        for j, i in enumerate(indices[0]):\n            if i == -1:\n                # This happens when not enough docs are returned.\n                continue\n            _id = self.index_to_docstore_id[i]\n            doc = self.docstore.search(_id)\n            if not isinstance(doc, Document):\n                raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n            docs.append((doc, scores[0][j]))\n        return docs", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}675{"id": "1e3993815549-5", "text": "docs.append((doc, scores[0][j]))\n        return docs\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        embedding = self.embedding_function(query)\n        docs = self.similarity_search_with_score_by_vector(embedding, k)\n        return docs\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the embedding.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score_by_vector(embedding, k)\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}676{"id": "1e3993815549-6", "text": "\"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k)\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        _, indices = self.index.search(np.array([embedding], dtype=np.float32), fetch_k)\n        # -1 happens when not enough docs are returned.\n        embeddings = [self.index.reconstruct(int(i)) for i in indices[0] if i != -1]\n        mmr_selected = maximal_marginal_relevance(\n            np.array([embedding], dtype=np.float32),\n            embeddings,\n            k=k,\n            lambda_mult=lambda_mult,\n        )\n        selected_indices = [indices[0][i] for i in mmr_selected]\n        docs = []\n        for i in selected_indices:\n            if i == -1:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}677{"id": "1e3993815549-7", "text": "docs = []\n        for i in selected_indices:\n            if i == -1:\n                # This happens when not enough docs are returned.\n                continue\n            _id = self.index_to_docstore_id[i]\n            doc = self.docstore.search(_id)\n            if not isinstance(doc, Document):\n                raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n            docs.append(doc)\n        return docs\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        embedding = self.embedding_function(query)\n        docs = self.max_marginal_relevance_search_by_vector(\n            embedding, k, fetch_k, lambda_mult=lambda_mult\n        )\n        return docs\n[docs]    def merge_from(self, target: FAISS) -> None:\n        \"\"\"Merge another FAISS object with the current one.\n        Add the target FAISS to the current one.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}678{"id": "1e3993815549-8", "text": "Add the target FAISS to the current one.\n        Args:\n            target: FAISS object you wish to merge into the current one\n        Returns:\n            None.\n        \"\"\"\n        if not isinstance(self.docstore, AddableMixin):\n            raise ValueError(\"Cannot merge with this type of docstore\")\n        # Numerical index for target docs are incremental on existing ones\n        starting_len = len(self.index_to_docstore_id)\n        # Merge two IndexFlatL2\n        self.index.merge_from(target.index)\n        # Get id and docs from target FAISS object\n        full_info = []\n        for i, target_id in target.index_to_docstore_id.items():\n            doc = target.docstore.search(target_id)\n            if not isinstance(doc, Document):\n                raise ValueError(\"Document should be returned\")\n            full_info.append((starting_len + i, target_id, doc))\n        # Add information to docstore and index_to_docstore_id.\n        self.docstore.add({_id: doc for _, _id, doc in full_info})\n        index_to_id = {index: _id for index, _id, _ in full_info}\n        self.index_to_docstore_id.update(index_to_id)\n    @classmethod\n    def __from(\n        cls,\n        texts: List[str],\n        embeddings: List[List[float]],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        normalize_L2: bool = False,\n        **kwargs: Any,\n    ) -> FAISS:\n        faiss = dependable_faiss_import()\n        index = faiss.IndexFlatL2(len(embeddings[0]))\n        vector = np.array(embeddings, dtype=np.float32)\n        if normalize_L2:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}679{"id": "1e3993815549-9", "text": "vector = np.array(embeddings, dtype=np.float32)\n        if normalize_L2:\n            faiss.normalize_L2(vector)\n        index.add(vector)\n        documents = []\n        if ids is None:\n            ids = [str(uuid.uuid4()) for _ in texts]\n        for i, text in enumerate(texts):\n            metadata = metadatas[i] if metadatas else {}\n            documents.append(Document(page_content=text, metadata=metadata))\n        index_to_id = dict(enumerate(ids))\n        docstore = InMemoryDocstore(dict(zip(index_to_id.values(), documents)))\n        return cls(\n            embedding.embed_query,\n            index,\n            docstore,\n            index_to_id,\n            normalize_L2=normalize_L2,\n            **kwargs,\n        )\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> FAISS:\n        \"\"\"Construct FAISS wrapper from raw documents.\n        This is a user friendly interface that:\n            1. Embeds documents.\n            2. Creates an in memory docstore\n            3. Initializes the FAISS database\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import FAISS\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                faiss = FAISS.from_texts(texts, embeddings)\n        \"\"\"\n        embeddings = embedding.embed_documents(texts)\n        return cls.__from(\n            texts,\n            embeddings,\n            embedding,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}680{"id": "1e3993815549-10", "text": "return cls.__from(\n            texts,\n            embeddings,\n            embedding,\n            metadatas=metadatas,\n            ids=ids,\n            **kwargs,\n        )\n[docs]    @classmethod\n    def from_embeddings(\n        cls,\n        text_embeddings: List[Tuple[str, List[float]]],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> FAISS:\n        \"\"\"Construct FAISS wrapper from raw documents.\n        This is a user friendly interface that:\n            1. Embeds documents.\n            2. Creates an in memory docstore\n            3. Initializes the FAISS database\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import FAISS\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                text_embeddings = embeddings.embed_documents(texts)\n                text_embedding_pairs = list(zip(texts, text_embeddings))\n                faiss = FAISS.from_embeddings(text_embedding_pairs, embeddings)\n        \"\"\"\n        texts = [t[0] for t in text_embeddings]\n        embeddings = [t[1] for t in text_embeddings]\n        return cls.__from(\n            texts,\n            embeddings,\n            embedding,\n            metadatas=metadatas,\n            ids=ids,\n            **kwargs,\n        )\n[docs]    def save_local(self, folder_path: str, index_name: str = \"index\") -> None:\n        \"\"\"Save FAISS index, docstore, and index_to_docstore_id to disk.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}681{"id": "1e3993815549-11", "text": "Args:\n            folder_path: folder path to save index, docstore,\n                and index_to_docstore_id to.\n            index_name: for saving with a specific index file name\n        \"\"\"\n        path = Path(folder_path)\n        path.mkdir(exist_ok=True, parents=True)\n        # save index separately since it is not picklable\n        faiss = dependable_faiss_import()\n        faiss.write_index(\n            self.index, str(path / \"{index_name}.faiss\".format(index_name=index_name))\n        )\n        # save docstore and index_to_docstore_id\n        with open(path / \"{index_name}.pkl\".format(index_name=index_name), \"wb\") as f:\n            pickle.dump((self.docstore, self.index_to_docstore_id), f)\n[docs]    @classmethod\n    def load_local(\n        cls, folder_path: str, embeddings: Embeddings, index_name: str = \"index\"\n    ) -> FAISS:\n        \"\"\"Load FAISS index, docstore, and index_to_docstore_id to disk.\n        Args:\n            folder_path: folder path to load index, docstore,\n                and index_to_docstore_id from.\n            embeddings: Embeddings to use when generating queries\n            index_name: for saving with a specific index file name\n        \"\"\"\n        path = Path(folder_path)\n        # load index separately since it is not picklable\n        faiss = dependable_faiss_import()\n        index = faiss.read_index(\n            str(path / \"{index_name}.faiss\".format(index_name=index_name))\n        )\n        # load docstore and index_to_docstore_id\n        with open(path / \"{index_name}.pkl\".format(index_name=index_name), \"rb\") as f:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}682{"id": "1e3993815549-12", "text": "docstore, index_to_docstore_id = pickle.load(f)\n        return cls(embeddings.embed_query, index, docstore, index_to_docstore_id)\n    def _similarity_search_with_relevance_scores(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and their similarity scores on a scale from 0 to 1.\"\"\"\n        if self.relevance_score_fn is None:\n            raise ValueError(\n                \"normalize_score_fn must be provided to\"\n                \" FAISS constructor to normalize scores\"\n            )\n        docs_and_scores = self.similarity_search_with_score(query, k=k)\n        return [(doc, self.relevance_score_fn(score)) for doc, score in docs_and_scores]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/faiss.html"}683{"id": "9358d8948c4c-0", "text": "Source code for langchain.vectorstores.atlas\n\"\"\"Wrapper around Atlas by Nomic.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport uuid\nfrom typing import Any, Iterable, List, Optional, Type\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nlogger = logging.getLogger(__name__)\n[docs]class AtlasDB(VectorStore):\n    \"\"\"Wrapper around Atlas: Nomic's neural database and rhizomatic instrument.\n    To use, you should have the ``nomic`` python package installed.\n    Example:\n        .. code-block:: python\n                from langchain.vectorstores import AtlasDB\n                from langchain.embeddings.openai import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                vectorstore = AtlasDB(\"my_project\", embeddings.embed_query)\n    \"\"\"\n    _ATLAS_DEFAULT_ID_FIELD = \"atlas_id\"\n    def __init__(\n        self,\n        name: str,\n        embedding_function: Optional[Embeddings] = None,\n        api_key: Optional[str] = None,\n        description: str = \"A description for your project\",\n        is_public: bool = True,\n        reset_project_if_exists: bool = False,\n    ) -> None:\n        \"\"\"\n        Initialize the Atlas Client\n        Args:\n            name (str): The name of your project. If the project already exists,\n                it will be loaded.\n            embedding_function (Optional[Callable]): An optional function used for\n                embedding your data. If None, data will be embedded with\n                Nomic's embed model.\n            api_key (str): Your nomic API key\n            description (str): A description for your project.\n            is_public (bool): Whether your project is publicly accessible.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}684{"id": "9358d8948c4c-1", "text": "is_public (bool): Whether your project is publicly accessible.\n                True by default.\n            reset_project_if_exists (bool): Whether to reset this project if it\n                already exists. Default False.\n                Generally userful during development and testing.\n        \"\"\"\n        try:\n            import nomic\n            from nomic import AtlasProject\n        except ImportError:\n            raise ValueError(\n                \"Could not import nomic python package. \"\n                \"Please install it with `pip install nomic`.\"\n            )\n        if api_key is None:\n            raise ValueError(\"No API key provided. Sign up at atlas.nomic.ai!\")\n        nomic.login(api_key)\n        self._embedding_function = embedding_function\n        modality = \"text\"\n        if self._embedding_function is not None:\n            modality = \"embedding\"\n        # Check if the project exists, create it if not\n        self.project = AtlasProject(\n            name=name,\n            description=description,\n            modality=modality,\n            is_public=is_public,\n            reset_project_if_exists=reset_project_if_exists,\n            unique_id_field=AtlasDB._ATLAS_DEFAULT_ID_FIELD,\n        )\n        self.project._latest_project_state()\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        refresh: bool = True,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts (Iterable[str]): Texts to add to the vectorstore.\n            metadatas (Optional[List[dict]], optional): Optional list of metadatas.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}685{"id": "9358d8948c4c-2", "text": "metadatas (Optional[List[dict]], optional): Optional list of metadatas.\n            ids (Optional[List[str]]): An optional list of ids.\n            refresh(bool): Whether or not to refresh indices with the updated data.\n                Default True.\n        Returns:\n            List[str]: List of IDs of the added texts.\n        \"\"\"\n        if (\n            metadatas is not None\n            and len(metadatas) > 0\n            and \"text\" in metadatas[0].keys()\n        ):\n            raise ValueError(\"Cannot accept key text in metadata!\")\n        texts = list(texts)\n        if ids is None:\n            ids = [str(uuid.uuid1()) for _ in texts]\n        # Embedding upload case\n        if self._embedding_function is not None:\n            _embeddings = self._embedding_function.embed_documents(texts)\n            embeddings = np.stack(_embeddings)\n            if metadatas is None:\n                data = [\n                    {AtlasDB._ATLAS_DEFAULT_ID_FIELD: ids[i], \"text\": texts[i]}\n                    for i, _ in enumerate(texts)\n                ]\n            else:\n                for i in range(len(metadatas)):\n                    metadatas[i][AtlasDB._ATLAS_DEFAULT_ID_FIELD] = ids[i]\n                    metadatas[i][\"text\"] = texts[i]\n                data = metadatas\n            self.project._validate_map_data_inputs(\n                [], id_field=AtlasDB._ATLAS_DEFAULT_ID_FIELD, data=data\n            )\n            with self.project.wait_for_project_lock():\n                self.project.add_embeddings(embeddings=embeddings, data=data)\n        # Text upload case\n        else:\n            if metadatas is None:\n                data = [", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}686{"id": "9358d8948c4c-3", "text": "else:\n            if metadatas is None:\n                data = [\n                    {\"text\": text, AtlasDB._ATLAS_DEFAULT_ID_FIELD: ids[i]}\n                    for i, text in enumerate(texts)\n                ]\n            else:\n                for i, text in enumerate(texts):\n                    metadatas[i][\"text\"] = texts\n                    metadatas[i][AtlasDB._ATLAS_DEFAULT_ID_FIELD] = ids[i]\n                data = metadatas\n            self.project._validate_map_data_inputs(\n                [], id_field=AtlasDB._ATLAS_DEFAULT_ID_FIELD, data=data\n            )\n            with self.project.wait_for_project_lock():\n                self.project.add_text(data)\n        if refresh:\n            if len(self.project.indices) > 0:\n                with self.project.wait_for_project_lock():\n                    self.project.rebuild_maps()\n        return ids\n[docs]    def create_index(self, **kwargs: Any) -> Any:\n        \"\"\"Creates an index in your project.\n        See\n        https://docs.nomic.ai/atlas_api.html#nomic.project.AtlasProject.create_index\n        for full detail.\n        \"\"\"\n        with self.project.wait_for_project_lock():\n            return self.project.create_index(**kwargs)\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Run similarity search with AtlasDB\n        Args:\n            query (str): Query text to search for.\n            k (int): Number of results to return. Defaults to 4.\n        Returns:\n            List[Document]: List of documents most similar to the query text.\n        \"\"\"\n        if self._embedding_function is None:\n            raise NotImplementedError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}687{"id": "9358d8948c4c-4", "text": "\"\"\"\n        if self._embedding_function is None:\n            raise NotImplementedError(\n                \"AtlasDB requires an embedding_function for text similarity search!\"\n            )\n        _embedding = self._embedding_function.embed_documents([query])[0]\n        embedding = np.array(_embedding).reshape(1, -1)\n        with self.project.wait_for_project_lock():\n            neighbors, _ = self.project.projections[0].vector_search(\n                queries=embedding, k=k\n            )\n            datas = self.project.get_data(ids=neighbors[0])\n        docs = [\n            Document(page_content=datas[i][\"text\"], metadata=datas[i])\n            for i, neighbor in enumerate(neighbors)\n        ]\n        return docs\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[AtlasDB],\n        texts: List[str],\n        embedding: Optional[Embeddings] = None,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        name: Optional[str] = None,\n        api_key: Optional[str] = None,\n        description: str = \"A description for your project\",\n        is_public: bool = True,\n        reset_project_if_exists: bool = False,\n        index_kwargs: Optional[dict] = None,\n        **kwargs: Any,\n    ) -> AtlasDB:\n        \"\"\"Create an AtlasDB vectorstore from a raw documents.\n        Args:\n            texts (List[str]): The list of texts to ingest.\n            name (str): Name of the project to create.\n            api_key (str): Your nomic API key,\n            embedding (Optional[Embeddings]): Embedding function. Defaults to None.\n            metadatas (Optional[List[dict]]): List of metadatas. Defaults to None.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}688{"id": "9358d8948c4c-5", "text": "ids (Optional[List[str]]): Optional list of document IDs. If None,\n                ids will be auto created\n            description (str): A description for your project.\n            is_public (bool): Whether your project is publicly accessible.\n                True by default.\n            reset_project_if_exists (bool): Whether to reset this project if it\n                already exists. Default False.\n                Generally userful during development and testing.\n            index_kwargs (Optional[dict]): Dict of kwargs for index creation.\n                See https://docs.nomic.ai/atlas_api.html\n        Returns:\n            AtlasDB: Nomic's neural database and finest rhizomatic instrument\n        \"\"\"\n        if name is None or api_key is None:\n            raise ValueError(\"`name` and `api_key` cannot be None.\")\n        # Inject relevant kwargs\n        all_index_kwargs = {\"name\": name + \"_index\", \"indexed_field\": \"text\"}\n        if index_kwargs is not None:\n            for k, v in index_kwargs.items():\n                all_index_kwargs[k] = v\n        # Build project\n        atlasDB = cls(\n            name,\n            embedding_function=embedding,\n            api_key=api_key,\n            description=\"A description for your project\",\n            is_public=is_public,\n            reset_project_if_exists=reset_project_if_exists,\n        )\n        with atlasDB.project.wait_for_project_lock():\n            atlasDB.add_texts(texts=texts, metadatas=metadatas, ids=ids)\n            atlasDB.create_index(**all_index_kwargs)\n        return atlasDB\n[docs]    @classmethod\n    def from_documents(\n        cls: Type[AtlasDB],\n        documents: List[Document],\n        embedding: Optional[Embeddings] = None,\n        ids: Optional[List[str]] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}689{"id": "9358d8948c4c-6", "text": "ids: Optional[List[str]] = None,\n        name: Optional[str] = None,\n        api_key: Optional[str] = None,\n        persist_directory: Optional[str] = None,\n        description: str = \"A description for your project\",\n        is_public: bool = True,\n        reset_project_if_exists: bool = False,\n        index_kwargs: Optional[dict] = None,\n        **kwargs: Any,\n    ) -> AtlasDB:\n        \"\"\"Create an AtlasDB vectorstore from a list of documents.\n        Args:\n            name (str): Name of the collection to create.\n            api_key (str): Your nomic API key,\n            documents (List[Document]): List of documents to add to the vectorstore.\n            embedding (Optional[Embeddings]): Embedding function. Defaults to None.\n            ids (Optional[List[str]]): Optional list of document IDs. If None,\n                ids will be auto created\n            description (str): A description for your project.\n            is_public (bool): Whether your project is publicly accessible.\n                True by default.\n            reset_project_if_exists (bool): Whether to reset this project if\n                it already exists. Default False.\n                Generally userful during development and testing.\n            index_kwargs (Optional[dict]): Dict of kwargs for index creation.\n                See https://docs.nomic.ai/atlas_api.html\n        Returns:\n            AtlasDB: Nomic's neural database and finest rhizomatic instrument\n        \"\"\"\n        if name is None or api_key is None:\n            raise ValueError(\"`name` and `api_key` cannot be None.\")\n        texts = [doc.page_content for doc in documents]\n        metadatas = [doc.metadata for doc in documents]\n        return cls.from_texts(\n            name=name,\n            api_key=api_key,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}690{"id": "9358d8948c4c-7", "text": "return cls.from_texts(\n            name=name,\n            api_key=api_key,\n            texts=texts,\n            embedding=embedding,\n            metadatas=metadatas,\n            ids=ids,\n            description=description,\n            is_public=is_public,\n            reset_project_if_exists=reset_project_if_exists,\n            index_kwargs=index_kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/atlas.html"}691{"id": "00a02e02c1f5-0", "text": "Source code for langchain.vectorstores.sklearn\n\"\"\" Wrapper around scikit-learn NearestNeighbors implementation.\nThe vector store can be persisted in json, bson or parquet format.\n\"\"\"\nimport importlib\nimport json\nimport math\nimport os\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Dict, Iterable, List, Literal, Optional, Tuple, Type\nfrom uuid import uuid4\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\ndef guard_import(\n    module_name: str, *, pip_name: Optional[str] = None, package: Optional[str] = None\n) -> Any:\n    \"\"\"Dynamically imports a module and raises a helpful exception if the module is not\n    installed.\"\"\"\n    try:\n        module = importlib.import_module(module_name, package)\n    except ImportError:\n        raise ImportError(\n            f\"Could not import {module_name} python package. \"\n            f\"Please install it with `pip install {pip_name or module_name}`.\"\n        )\n    return module\nclass BaseSerializer(ABC):\n    \"\"\"Abstract base class for saving and loading data.\"\"\"\n    def __init__(self, persist_path: str) -> None:\n        self.persist_path = persist_path\n    @classmethod\n    @abstractmethod\n    def extension(cls) -> str:\n        \"\"\"The file extension suggested by this serializer (without dot).\"\"\"\n    @abstractmethod\n    def save(self, data: Any) -> None:\n        \"\"\"Saves the data to the persist_path\"\"\"\n    @abstractmethod\n    def load(self) -> Any:\n        \"\"\"Loads the data from the persist_path\"\"\"\nclass JsonSerializer(BaseSerializer):\n    \"\"\"Serializes data in json using the json package from python standard library.\"\"\"\n    @classmethod", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}692{"id": "00a02e02c1f5-1", "text": "\"\"\"Serializes data in json using the json package from python standard library.\"\"\"\n    @classmethod\n    def extension(cls) -> str:\n        return \"json\"\n    def save(self, data: Any) -> None:\n        with open(self.persist_path, \"w\") as fp:\n            json.dump(data, fp)\n    def load(self) -> Any:\n        with open(self.persist_path, \"r\") as fp:\n            return json.load(fp)\nclass BsonSerializer(BaseSerializer):\n    \"\"\"Serializes data in binary json using the bson python package.\"\"\"\n    def __init__(self, persist_path: str) -> None:\n        super().__init__(persist_path)\n        self.bson = guard_import(\"bson\")\n    @classmethod\n    def extension(cls) -> str:\n        return \"bson\"\n    def save(self, data: Any) -> None:\n        with open(self.persist_path, \"wb\") as fp:\n            fp.write(self.bson.dumps(data))\n    def load(self) -> Any:\n        with open(self.persist_path, \"rb\") as fp:\n            return self.bson.loads(fp.read())\nclass ParquetSerializer(BaseSerializer):\n    \"\"\"Serializes data in Apache Parquet format using the pyarrow package.\"\"\"\n    def __init__(self, persist_path: str) -> None:\n        super().__init__(persist_path)\n        self.pd = guard_import(\"pandas\")\n        self.pa = guard_import(\"pyarrow\")\n        self.pq = guard_import(\"pyarrow.parquet\")\n    @classmethod\n    def extension(cls) -> str:\n        return \"parquet\"\n    def save(self, data: Any) -> None:\n        df = self.pd.DataFrame(data)\n        table = self.pa.Table.from_pandas(df)\n        if os.path.exists(self.persist_path):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}693{"id": "00a02e02c1f5-2", "text": "if os.path.exists(self.persist_path):\n            backup_path = str(self.persist_path) + \"-backup\"\n            os.rename(self.persist_path, backup_path)\n            try:\n                self.pq.write_table(table, self.persist_path)\n            except Exception as exc:\n                os.rename(backup_path, self.persist_path)\n                raise exc\n            else:\n                os.remove(backup_path)\n        else:\n            self.pq.write_table(table, self.persist_path)\n    def load(self) -> Any:\n        table = self.pq.read_table(self.persist_path)\n        df = table.to_pandas()\n        return {col: series.tolist() for col, series in df.items()}\nSERIALIZER_MAP: Dict[str, Type[BaseSerializer]] = {\n    \"json\": JsonSerializer,\n    \"bson\": BsonSerializer,\n    \"parquet\": ParquetSerializer,\n}\nclass SKLearnVectorStoreException(RuntimeError):\n    pass\n[docs]class SKLearnVectorStore(VectorStore):\n    \"\"\"A simple in-memory vector store based on the scikit-learn library\n    NearestNeighbors implementation.\"\"\"\n    def __init__(\n        self,\n        embedding: Embeddings,\n        *,\n        persist_path: Optional[str] = None,\n        serializer: Literal[\"json\", \"bson\", \"parquet\"] = \"json\",\n        metric: str = \"cosine\",\n        **kwargs: Any,\n    ) -> None:\n        np = guard_import(\"numpy\")\n        sklearn_neighbors = guard_import(\"sklearn.neighbors\", pip_name=\"scikit-learn\")\n        # non-persistent properties\n        self._np = np\n        self._neighbors = sklearn_neighbors.NearestNeighbors(metric=metric, **kwargs)\n        self._neighbors_fitted = False\n        self._embedding_function = embedding", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}694{"id": "00a02e02c1f5-3", "text": "self._neighbors_fitted = False\n        self._embedding_function = embedding\n        self._persist_path = persist_path\n        self._serializer: Optional[BaseSerializer] = None\n        if self._persist_path is not None:\n            serializer_cls = SERIALIZER_MAP[serializer]\n            self._serializer = serializer_cls(persist_path=self._persist_path)\n        # data properties\n        self._embeddings: List[List[float]] = []\n        self._texts: List[str] = []\n        self._metadatas: List[dict] = []\n        self._ids: List[str] = []\n        # cache properties\n        self._embeddings_np: Any = np.asarray([])\n        if self._persist_path is not None and os.path.isfile(self._persist_path):\n            self._load()\n[docs]    def persist(self) -> None:\n        if self._serializer is None:\n            raise SKLearnVectorStoreException(\n                \"You must specify a persist_path on creation to persist the \"\n                \"collection.\"\n            )\n        data = {\n            \"ids\": self._ids,\n            \"texts\": self._texts,\n            \"metadatas\": self._metadatas,\n            \"embeddings\": self._embeddings,\n        }\n        self._serializer.save(data)\n    def _load(self) -> None:\n        if self._serializer is None:\n            raise SKLearnVectorStoreException(\n                \"You must specify a persist_path on creation to load the \" \"collection.\"\n            )\n        data = self._serializer.load()\n        self._embeddings = data[\"embeddings\"]\n        self._texts = data[\"texts\"]\n        self._metadatas = data[\"metadatas\"]\n        self._ids = data[\"ids\"]\n        self._update_neighbors()", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}695{"id": "00a02e02c1f5-4", "text": "self._ids = data[\"ids\"]\n        self._update_neighbors()\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        _texts = list(texts)\n        _ids = ids or [str(uuid4()) for _ in _texts]\n        self._texts.extend(_texts)\n        self._embeddings.extend(self._embedding_function.embed_documents(_texts))\n        self._metadatas.extend(metadatas or ([{}] * len(_texts)))\n        self._ids.extend(_ids)\n        self._update_neighbors()\n        return _ids\n    def _update_neighbors(self) -> None:\n        if len(self._embeddings) == 0:\n            raise SKLearnVectorStoreException(\n                \"No data was added to SKLearnVectorStore.\"\n            )\n        self._embeddings_np = self._np.asarray(self._embeddings)\n        self._neighbors.fit(self._embeddings_np)\n        self._neighbors_fitted = True\n[docs]    def similarity_search_with_score(\n        self, query: str, *, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        if not self._neighbors_fitted:\n            raise SKLearnVectorStoreException(\n                \"No data was added to SKLearnVectorStore.\"\n            )\n        query_embedding = self._embedding_function.embed_query(query)\n        neigh_dists, neigh_idxs = self._neighbors.kneighbors(\n            [query_embedding], n_neighbors=k\n        )\n        res = []\n        for idx, dist in zip(neigh_idxs[0], neigh_dists[0]):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}696{"id": "00a02e02c1f5-5", "text": "for idx, dist in zip(neigh_idxs[0], neigh_dists[0]):\n            _idx = int(idx)\n            metadata = {\"id\": self._ids[_idx], **self._metadatas[_idx]}\n            doc = Document(page_content=self._texts[_idx], metadata=metadata)\n            res.append((doc, dist))\n        return res\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        docs_scores = self.similarity_search_with_score(query, k=k, **kwargs)\n        return [doc for doc, _ in docs_scores]\n    def _similarity_search_with_relevance_scores(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        docs_dists = self.similarity_search_with_score(query=query, k=k, **kwargs)\n        docs, dists = zip(*docs_dists)\n        scores = [1 / math.exp(dist) for dist in dists]\n        return list(zip(list(docs), scores))\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        persist_path: Optional[str] = None,\n        **kwargs: Any,\n    ) -> \"SKLearnVectorStore\":\n        vs = SKLearnVectorStore(embedding, persist_path=persist_path, **kwargs)\n        vs.add_texts(texts, metadatas=metadatas, ids=ids)\n        return vs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}697{"id": "00a02e02c1f5-6", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/sklearn.html"}698{"id": "658aeb7f3185-0", "text": "Source code for langchain.vectorstores.vectara\n\"\"\"Wrapper around Vectara vector database.\"\"\"\nfrom __future__ import annotations\nimport json\nimport logging\nimport os\nfrom hashlib import md5\nfrom typing import Any, Iterable, List, Optional, Tuple, Type\nimport requests\nfrom pydantic import Field\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.schema import Document\nfrom langchain.vectorstores.base import VectorStore, VectorStoreRetriever\n[docs]class Vectara(VectorStore):\n    \"\"\"Implementation of Vector Store using Vectara (https://vectara.com).\n    Example:\n        .. code-block:: python\n            from langchain.vectorstores import Vectara\n            vectorstore = Vectara(\n                vectara_customer_id=vectara_customer_id,\n                vectara_corpus_id=vectara_corpus_id,\n                vectara_api_key=vectara_api_key\n            )\n    \"\"\"\n    def __init__(\n        self,\n        vectara_customer_id: Optional[str] = None,\n        vectara_corpus_id: Optional[str] = None,\n        vectara_api_key: Optional[str] = None,\n    ):\n        \"\"\"Initialize with Vectara API.\"\"\"\n        self._vectara_customer_id = vectara_customer_id or os.environ.get(\n            \"VECTARA_CUSTOMER_ID\"\n        )\n        self._vectara_corpus_id = vectara_corpus_id or os.environ.get(\n            \"VECTARA_CORPUS_ID\"\n        )\n        self._vectara_api_key = vectara_api_key or os.environ.get(\"VECTARA_API_KEY\")\n        if (\n            self._vectara_customer_id is None\n            or self._vectara_corpus_id is None\n            or self._vectara_api_key is None\n        ):\n            logging.warning(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}699{"id": "658aeb7f3185-1", "text": "or self._vectara_api_key is None\n        ):\n            logging.warning(\n                \"Cant find Vectara credentials, customer_id or corpus_id in \"\n                \"environment.\"\n            )\n        else:\n            logging.debug(f\"Using corpus id {self._vectara_corpus_id}\")\n        self._session = requests.Session()  # to reuse connections\n    def _get_post_headers(self) -> dict:\n        \"\"\"Returns headers that should be attached to each post request.\"\"\"\n        return {\n            \"x-api-key\": self._vectara_api_key,\n            \"customer-id\": self._vectara_customer_id,\n            \"Content-Type\": \"application/json\",\n        }\n    def _delete_doc(self, doc_id: str) -> bool:\n        \"\"\"\n        Delete a document from the Vectara corpus.\n        Args:\n            url (str): URL of the page to delete.\n            doc_id (str): ID of the document to delete.\n        Returns:\n            bool: True if deletion was successful, False otherwise.\n        \"\"\"\n        body = {\n            \"customer_id\": self._vectara_customer_id,\n            \"corpus_id\": self._vectara_corpus_id,\n            \"document_id\": doc_id,\n        }\n        response = self._session.post(\n            \"https://api.vectara.io/v1/delete-doc\",\n            data=json.dumps(body),\n            verify=True,\n            headers=self._get_post_headers(),\n        )\n        if response.status_code != 200:\n            logging.error(\n                f\"Delete request failed for doc_id = {doc_id} with status code \"\n                f\"{response.status_code}, reason {response.reason}, text \"\n                f\"{response.text}\"\n            )\n            return False\n        return True", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}700{"id": "658aeb7f3185-2", "text": "f\"{response.text}\"\n            )\n            return False\n        return True\n    def _index_doc(self, doc_id: str, text: str, metadata: dict) -> bool:\n        request: dict[str, Any] = {}\n        request[\"customer_id\"] = self._vectara_customer_id\n        request[\"corpus_id\"] = self._vectara_corpus_id\n        request[\"document\"] = {\n            \"document_id\": doc_id,\n            \"metadataJson\": json.dumps(metadata),\n            \"section\": [{\"text\": text, \"metadataJson\": json.dumps(metadata)}],\n        }\n        response = self._session.post(\n            headers=self._get_post_headers(),\n            url=\"https://api.vectara.io/v1/index\",\n            data=json.dumps(request),\n            timeout=30,\n            verify=True,\n        )\n        status_code = response.status_code\n        result = response.json()\n        status_str = result[\"status\"][\"code\"] if \"status\" in result else None\n        if status_code == 409 or (status_str and status_str == \"ALREADY_EXISTS\"):\n            return False\n        else:\n            return True\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        ids = [md5(text.encode(\"utf-8\")).hexdigest() for text in texts]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}701{"id": "658aeb7f3185-3", "text": "ids = [md5(text.encode(\"utf-8\")).hexdigest() for text in texts]\n        for i, doc in enumerate(texts):\n            doc_id = ids[i]\n            metadata = metadatas[i] if metadatas else {}\n            succeeded = self._index_doc(doc_id, doc, metadata)\n            if not succeeded:\n                self._delete_doc(doc_id)\n                self._index_doc(doc_id, doc, metadata)\n        return ids\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = 5,\n        alpha: float = 0.025,\n        filter: Optional[str] = None,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return Vectara documents most similar to query, along with scores.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 5.\n            alpha: parameter for hybrid search (called \"lambda\" in Vectara\n                documentation).\n            filter: Dictionary of argument(s) to filter on metadata. For example a\n                filter can be \"doc.rating > 3.0 and part.lang = 'deu'\"} see\n                https://docs.vectara.com/docs/search-apis/sql/filter-overview\n                for more details.\n        Returns:\n            List of Documents most similar to the query and score for each.\n        \"\"\"\n        response = self._session.post(\n            headers=self._get_post_headers(),\n            url=\"https://api.vectara.io/v1/query\",\n            data=json.dumps(\n                {\n                    \"query\": [\n                        {\n                            \"query\": query,\n                            \"start\": 0,\n                            \"num_results\": k,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}702{"id": "658aeb7f3185-4", "text": "\"start\": 0,\n                            \"num_results\": k,\n                            \"context_config\": {\n                                \"sentences_before\": 3,\n                                \"sentences_after\": 3,\n                            },\n                            \"corpus_key\": [\n                                {\n                                    \"customer_id\": self._vectara_customer_id,\n                                    \"corpus_id\": self._vectara_corpus_id,\n                                    \"metadataFilter\": filter,\n                                    \"lexical_interpolation_config\": {\"lambda\": alpha},\n                                }\n                            ],\n                        }\n                    ]\n                }\n            ),\n            timeout=10,\n        )\n        if response.status_code != 200:\n            logging.error(\n                \"Query failed %s\",\n                f\"(code {response.status_code}, reason {response.reason}, details \"\n                f\"{response.text})\",\n            )\n            return []\n        result = response.json()\n        responses = result[\"responseSet\"][0][\"response\"]\n        vectara_default_metadata = [\"lang\", \"len\", \"offset\"]\n        docs = [\n            (\n                Document(\n                    page_content=x[\"text\"],\n                    metadata={\n                        m[\"name\"]: m[\"value\"]\n                        for m in x[\"metadata\"]\n                        if m[\"name\"] not in vectara_default_metadata\n                    },\n                ),\n                x[\"score\"],\n            )\n            for x in responses\n        ]\n        return docs\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 5,\n        alpha: float = 0.025,\n        filter: Optional[str] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return Vectara documents most similar to query, along with scores.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}703{"id": "658aeb7f3185-5", "text": "\"\"\"Return Vectara documents most similar to query, along with scores.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 5.\n            filter: Dictionary of argument(s) to filter on metadata. For example a\n                filter can be \"doc.rating > 3.0 and part.lang = 'deu'\"} see\n                https://docs.vectara.com/docs/search-apis/sql/filter-overview for more\n                details.\n        Returns:\n            List of Documents most similar to the query\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(\n            query, k=k, alpha=alpha, filter=filter, **kwargs\n        )\n        return [doc for doc, _ in docs_and_scores]\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Vectara],\n        texts: List[str],\n        embedding: Optional[Embeddings] = None,\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> Vectara:\n        \"\"\"Construct Vectara wrapper from raw documents.\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import Vectara\n                vectara = Vectara.from_texts(\n                    texts,\n                    vectara_customer_id=customer_id,\n                    vectara_corpus_id=corpus_id,\n                    vectara_api_key=api_key,\n                )\n        \"\"\"\n        # Note: Vectara generates its own embeddings, so we ignore the provided\n        # embeddings (required by interface)\n        vectara = cls(**kwargs)\n        vectara.add_texts(texts, metadatas)\n        return vectara", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}704{"id": "658aeb7f3185-6", "text": "vectara.add_texts(texts, metadatas)\n        return vectara\n[docs]    def as_retriever(self, **kwargs: Any) -> VectaraRetriever:\n        return VectaraRetriever(vectorstore=self, **kwargs)\nclass VectaraRetriever(VectorStoreRetriever):\n    vectorstore: Vectara\n    search_kwargs: dict = Field(default_factory=lambda: {\"alpha\": 0.025, \"k\": 5})\n    \"\"\"Search params.\n        k: Number of Documents to return. Defaults to 5.\n        alpha: parameter for hybrid search (called \"lambda\" in Vectara\n            documentation).\n        filter: Dictionary of argument(s) to filter on metadata. For example a\n            filter can be \"doc.rating > 3.0 and part.lang = 'deu'\"} see\n            https://docs.vectara.com/docs/search-apis/sql/filter-overview\n            for more details.\n    \"\"\"\n    def add_texts(\n        self, texts: List[str], metadatas: Optional[List[dict]] = None\n    ) -> None:\n        \"\"\"Add text to the Vectara vectorstore.\n        Args:\n            texts (List[str]): The text\n            metadatas (List[dict]): Metadata dicts, must line up with existing store\n        \"\"\"\n        self.vectorstore.add_texts(texts, metadatas)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/vectara.html"}705{"id": "1988082ebfab-0", "text": "Source code for langchain.vectorstores.zilliz\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, List, Optional\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.milvus import Milvus\nlogger = logging.getLogger(__name__)\n[docs]class Zilliz(Milvus):\n    def _create_index(self) -> None:\n        \"\"\"Create a index on the collection\"\"\"\n        from pymilvus import Collection, MilvusException\n        if isinstance(self.col, Collection) and self._get_index() is None:\n            try:\n                # If no index params, use a default AutoIndex based one\n                if self.index_params is None:\n                    self.index_params = {\n                        \"metric_type\": \"L2\",\n                        \"index_type\": \"AUTOINDEX\",\n                        \"params\": {},\n                    }\n                try:\n                    self.col.create_index(\n                        self._vector_field,\n                        index_params=self.index_params,\n                        using=self.alias,\n                    )\n                # If default did not work, most likely Milvus self-hosted\n                except MilvusException:\n                    # Use HNSW based index\n                    self.index_params = {\n                        \"metric_type\": \"L2\",\n                        \"index_type\": \"HNSW\",\n                        \"params\": {\"M\": 8, \"efConstruction\": 64},\n                    }\n                    self.col.create_index(\n                        self._vector_field,\n                        index_params=self.index_params,\n                        using=self.alias,\n                    )\n                logger.debug(\n                    \"Successfully created an index on collection: %s\",\n                    self.collection_name,\n                )\n            except MilvusException as e:\n                logger.error(\n                    \"Failed to create an index on collection: %s\", self.collection_name", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/zilliz.html"}706{"id": "1988082ebfab-1", "text": "\"Failed to create an index on collection: %s\", self.collection_name\n                )\n                raise e\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        collection_name: str = \"LangChainCollection\",\n        connection_args: dict[str, Any] = {},\n        consistency_level: str = \"Session\",\n        index_params: Optional[dict] = None,\n        search_params: Optional[dict] = None,\n        drop_old: bool = False,\n        **kwargs: Any,\n    ) -> Zilliz:\n        \"\"\"Create a Zilliz collection, indexes it with HNSW, and insert data.\n        Args:\n            texts (List[str]): Text data.\n            embedding (Embeddings): Embedding function.\n            metadatas (Optional[List[dict]]): Metadata for each text if it exists.\n                Defaults to None.\n            collection_name (str, optional): Collection name to use. Defaults to\n                \"LangChainCollection\".\n            connection_args (dict[str, Any], optional): Connection args to use. Defaults\n                to DEFAULT_MILVUS_CONNECTION.\n            consistency_level (str, optional): Which consistency level to use. Defaults\n                to \"Session\".\n            index_params (Optional[dict], optional): Which index_params to use.\n                Defaults to None.\n            search_params (Optional[dict], optional): Which search params to use.\n                Defaults to None.\n            drop_old (Optional[bool], optional): Whether to drop the collection with\n                that name if it exists. Defaults to False.\n        Returns:\n            Zilliz: Zilliz Vector Store\n        \"\"\"\n        vector_db = cls(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/zilliz.html"}707{"id": "1988082ebfab-2", "text": "Zilliz: Zilliz Vector Store\n        \"\"\"\n        vector_db = cls(\n            embedding_function=embedding,\n            collection_name=collection_name,\n            connection_args=connection_args,\n            consistency_level=consistency_level,\n            index_params=index_params,\n            search_params=search_params,\n            drop_old=drop_old,\n            **kwargs,\n        )\n        vector_db.add_texts(texts=texts, metadatas=metadatas)\n        return vector_db\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/zilliz.html"}708{"id": "076c198ce78a-0", "text": "Source code for langchain.vectorstores.base\n\"\"\"Interface for vector stores.\"\"\"\nfrom __future__ import annotations\nimport asyncio\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom functools import partial\nfrom typing import Any, Dict, Iterable, List, Optional, Tuple, Type, TypeVar\nfrom pydantic import BaseModel, Field, root_validator\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.schema import BaseRetriever\nVST = TypeVar(\"VST\", bound=\"VectorStore\")\n[docs]class VectorStore(ABC):\n    \"\"\"Interface for vector stores.\"\"\"\n[docs]    @abstractmethod\n    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            kwargs: vectorstore specific parameters\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n[docs]    async def aadd_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\"\"\"\n        raise NotImplementedError\n[docs]    def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]:\n        \"\"\"Run more documents through the embeddings and add to the vectorstore.\n        Args:\n            documents (List[Document]: Documents to add to the vectorstore.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}709{"id": "076c198ce78a-1", "text": "Args:\n            documents (List[Document]: Documents to add to the vectorstore.\n        Returns:\n            List[str]: List of IDs of the added texts.\n        \"\"\"\n        # TODO: Handle the case where the user doesn't provide ids on the Collection\n        texts = [doc.page_content for doc in documents]\n        metadatas = [doc.metadata for doc in documents]\n        return self.add_texts(texts, metadatas, **kwargs)\n[docs]    async def aadd_documents(\n        self, documents: List[Document], **kwargs: Any\n    ) -> List[str]:\n        \"\"\"Run more documents through the embeddings and add to the vectorstore.\n        Args:\n            documents (List[Document]: Documents to add to the vectorstore.\n        Returns:\n            List[str]: List of IDs of the added texts.\n        \"\"\"\n        texts = [doc.page_content for doc in documents]\n        metadatas = [doc.metadata for doc in documents]\n        return await self.aadd_texts(texts, metadatas, **kwargs)\n[docs]    def search(self, query: str, search_type: str, **kwargs: Any) -> List[Document]:\n        \"\"\"Return docs most similar to query using specified search type.\"\"\"\n        if search_type == \"similarity\":\n            return self.similarity_search(query, **kwargs)\n        elif search_type == \"mmr\":\n            return self.max_marginal_relevance_search(query, **kwargs)\n        else:\n            raise ValueError(\n                f\"search_type of {search_type} not allowed. Expected \"\n                \"search_type to be 'similarity' or 'mmr'.\"\n            )\n[docs]    async def asearch(\n        self, query: str, search_type: str, **kwargs: Any", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}710{"id": "076c198ce78a-2", "text": "self, query: str, search_type: str, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query using specified search type.\"\"\"\n        if search_type == \"similarity\":\n            return await self.asimilarity_search(query, **kwargs)\n        elif search_type == \"mmr\":\n            return await self.amax_marginal_relevance_search(query, **kwargs)\n        else:\n            raise ValueError(\n                f\"search_type of {search_type} not allowed. Expected \"\n                \"search_type to be 'similarity' or 'mmr'.\"\n            )\n[docs]    @abstractmethod\n    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\"\"\"\n[docs]    def similarity_search_with_relevance_scores(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and relevance scores in the range [0, 1].\n        0 is dissimilar, 1 is most similar.\n        Args:\n            query: input text\n            k: Number of Documents to return. Defaults to 4.\n            **kwargs: kwargs to be passed to similarity search. Should include:\n                score_threshold: Optional, a floating point value between 0 to 1 to\n                    filter the resulting set of retrieved docs\n        Returns:\n            List of Tuples of (doc, similarity_score)\n        \"\"\"\n        docs_and_similarities = self._similarity_search_with_relevance_scores(\n            query, k=k, **kwargs\n        )\n        if any(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}711{"id": "076c198ce78a-3", "text": "query, k=k, **kwargs\n        )\n        if any(\n            similarity < 0.0 or similarity > 1.0\n            for _, similarity in docs_and_similarities\n        ):\n            warnings.warn(\n                \"Relevance scores must be between\"\n                f\" 0 and 1, got {docs_and_similarities}\"\n            )\n        score_threshold = kwargs.get(\"score_threshold\")\n        if score_threshold is not None:\n            docs_and_similarities = [\n                (doc, similarity)\n                for doc, similarity in docs_and_similarities\n                if similarity >= score_threshold\n            ]\n            if len(docs_and_similarities) == 0:\n                warnings.warn(\n                    f\"No relevant docs were retrieved using the relevance score\\\n                          threshold {score_threshold}\"\n                )\n        return docs_and_similarities\n    def _similarity_search_with_relevance_scores(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and relevance scores, normalized on a scale from 0 to 1.\n        0 is dissimilar, 1 is most similar.\n        \"\"\"\n        raise NotImplementedError\n[docs]    async def asimilarity_search_with_relevance_scores(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\"\"\"\n        # This is a temporary workaround to make the similarity search\n        # asynchronous. The proper solution is to make the similarity search\n        # asynchronous in the vector store implementations.\n        func = partial(self.similarity_search_with_relevance_scores, query, k, **kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}712{"id": "076c198ce78a-4", "text": "func = partial(self.similarity_search_with_relevance_scores, query, k, **kwargs)\n        return await asyncio.get_event_loop().run_in_executor(None, func)\n[docs]    async def asimilarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\"\"\"\n        # This is a temporary workaround to make the similarity search\n        # asynchronous. The proper solution is to make the similarity search\n        # asynchronous in the vector store implementations.\n        func = partial(self.similarity_search, query, k, **kwargs)\n        return await asyncio.get_event_loop().run_in_executor(None, func)\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query vector.\n        \"\"\"\n        raise NotImplementedError\n[docs]    async def asimilarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\"\"\"\n        # This is a temporary workaround to make the similarity search\n        # asynchronous. The proper solution is to make the similarity search\n        # asynchronous in the vector store implementations.\n        func = partial(self.similarity_search_by_vector, embedding, k, **kwargs)\n        return await asyncio.get_event_loop().run_in_executor(None, func)\n[docs]    def max_marginal_relevance_search(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}713{"id": "076c198ce78a-5", "text": "[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        raise NotImplementedError\n[docs]    async def amax_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\"\"\"\n        # This is a temporary workaround to make the similarity search\n        # asynchronous. The proper solution is to make the similarity search\n        # asynchronous in the vector store implementations.\n        func = partial(\n            self.max_marginal_relevance_search, query, k, fetch_k, lambda_mult, **kwargs\n        )\n        return await asyncio.get_event_loop().run_in_executor(None, func)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}714{"id": "076c198ce78a-6", "text": ")\n        return await asyncio.get_event_loop().run_in_executor(None, func)\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        raise NotImplementedError\n[docs]    async def amax_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\"\"\"\n        raise NotImplementedError\n[docs]    @classmethod\n    def from_documents(\n        cls: Type[VST],\n        documents: List[Document],\n        embedding: Embeddings,\n        **kwargs: Any,\n    ) -> VST:\n        \"\"\"Return VectorStore initialized from documents and embeddings.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}715{"id": "076c198ce78a-7", "text": ") -> VST:\n        \"\"\"Return VectorStore initialized from documents and embeddings.\"\"\"\n        texts = [d.page_content for d in documents]\n        metadatas = [d.metadata for d in documents]\n        return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)\n[docs]    @classmethod\n    async def afrom_documents(\n        cls: Type[VST],\n        documents: List[Document],\n        embedding: Embeddings,\n        **kwargs: Any,\n    ) -> VST:\n        \"\"\"Return VectorStore initialized from documents and embeddings.\"\"\"\n        texts = [d.page_content for d in documents]\n        metadatas = [d.metadata for d in documents]\n        return await cls.afrom_texts(texts, embedding, metadatas=metadatas, **kwargs)\n[docs]    @classmethod\n    @abstractmethod\n    def from_texts(\n        cls: Type[VST],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> VST:\n        \"\"\"Return VectorStore initialized from texts and embeddings.\"\"\"\n[docs]    @classmethod\n    async def afrom_texts(\n        cls: Type[VST],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> VST:\n        \"\"\"Return VectorStore initialized from texts and embeddings.\"\"\"\n        raise NotImplementedError\n[docs]    def as_retriever(self, **kwargs: Any) -> VectorStoreRetriever:\n        return VectorStoreRetriever(vectorstore=self, **kwargs)\nclass VectorStoreRetriever(BaseRetriever, BaseModel):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}716{"id": "076c198ce78a-8", "text": "class VectorStoreRetriever(BaseRetriever, BaseModel):\n    vectorstore: VectorStore\n    search_type: str = \"similarity\"\n    search_kwargs: dict = Field(default_factory=dict)\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_search_type(cls, values: Dict) -> Dict:\n        \"\"\"Validate search type.\"\"\"\n        if \"search_type\" in values:\n            search_type = values[\"search_type\"]\n            if search_type not in (\"similarity\", \"similarity_score_threshold\", \"mmr\"):\n                raise ValueError(f\"search_type of {search_type} not allowed.\")\n            if search_type == \"similarity_score_threshold\":\n                score_threshold = values[\"search_kwargs\"].get(\"score_threshold\")\n                if (score_threshold is None) or (\n                    not isinstance(score_threshold, float)\n                ):\n                    raise ValueError(\n                        \"`score_threshold` is not specified with a float value(0~1) \"\n                        \"in `search_kwargs`.\"\n                    )\n        return values\n    def get_relevant_documents(self, query: str) -> List[Document]:\n        if self.search_type == \"similarity\":\n            docs = self.vectorstore.similarity_search(query, **self.search_kwargs)\n        elif self.search_type == \"similarity_score_threshold\":\n            docs_and_similarities = (\n                self.vectorstore.similarity_search_with_relevance_scores(\n                    query, **self.search_kwargs\n                )\n            )\n            docs = [doc for doc, _ in docs_and_similarities]\n        elif self.search_type == \"mmr\":\n            docs = self.vectorstore.max_marginal_relevance_search(\n                query, **self.search_kwargs\n            )\n        else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}717{"id": "076c198ce78a-9", "text": "query, **self.search_kwargs\n            )\n        else:\n            raise ValueError(f\"search_type of {self.search_type} not allowed.\")\n        return docs\n    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        if self.search_type == \"similarity\":\n            docs = await self.vectorstore.asimilarity_search(\n                query, **self.search_kwargs\n            )\n        elif self.search_type == \"similarity_score_threshold\":\n            docs_and_similarities = (\n                await self.vectorstore.asimilarity_search_with_relevance_scores(\n                    query, **self.search_kwargs\n                )\n            )\n            docs = [doc for doc, _ in docs_and_similarities]\n        elif self.search_type == \"mmr\":\n            docs = await self.vectorstore.amax_marginal_relevance_search(\n                query, **self.search_kwargs\n            )\n        else:\n            raise ValueError(f\"search_type of {self.search_type} not allowed.\")\n        return docs\n    def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        return self.vectorstore.add_documents(documents, **kwargs)\n    async def aadd_documents(\n        self, documents: List[Document], **kwargs: Any\n    ) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        return await self.vectorstore.aadd_documents(documents, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/base.html"}718{"id": "20346f865f71-0", "text": "Source code for langchain.vectorstores.lancedb\n\"\"\"Wrapper around LanceDB vector database\"\"\"\nfrom __future__ import annotations\nimport uuid\nfrom typing import Any, Iterable, List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\n[docs]class LanceDB(VectorStore):\n    \"\"\"Wrapper around LanceDB vector database.\n    To use, you should have ``lancedb`` python package installed.\n    Example:\n        .. code-block:: python\n            db = lancedb.connect('./lancedb')\n            table = db.open_table('my_table')\n            vectorstore = LanceDB(table, embedding_function)\n            vectorstore.add_texts(['text1', 'text2'])\n            result = vectorstore.similarity_search('text1')\n    \"\"\"\n    def __init__(\n        self,\n        connection: Any,\n        embedding: Embeddings,\n        vector_key: Optional[str] = \"vector\",\n        id_key: Optional[str] = \"id\",\n        text_key: Optional[str] = \"text\",\n    ):\n        \"\"\"Initialize with Lance DB connection\"\"\"\n        try:\n            import lancedb\n        except ImportError:\n            raise ValueError(\n                \"Could not import lancedb python package. \"\n                \"Please install it with `pip install lancedb`.\"\n            )\n        if not isinstance(connection, lancedb.db.LanceTable):\n            raise ValueError(\n                \"connection should be an instance of lancedb.db.LanceTable, \",\n                f\"got {type(connection)}\",\n            )\n        self._connection = connection\n        self._embedding = embedding\n        self._vector_key = vector_key\n        self._id_key = id_key\n        self._text_key = text_key", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/lancedb.html"}719{"id": "20346f865f71-1", "text": "self._id_key = id_key\n        self._text_key = text_key\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Turn texts into embedding and add it to the database\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            ids: Optional list of ids to associate with the texts.\n        Returns:\n            List of ids of the added texts.\n        \"\"\"\n        # Embed texts and create documents\n        docs = []\n        ids = ids or [str(uuid.uuid4()) for _ in texts]\n        embeddings = self._embedding.embed_documents(list(texts))\n        for idx, text in enumerate(texts):\n            embedding = embeddings[idx]\n            metadata = metadatas[idx] if metadatas else {}\n            docs.append(\n                {\n                    self._vector_key: embedding,\n                    self._id_key: ids[idx],\n                    self._text_key: text,\n                    **metadata,\n                }\n            )\n        self._connection.add(docs)\n        return ids\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return documents most similar to the query\n        Args:\n            query: String to query the vectorstore with.\n            k: Number of documents to return.\n        Returns:\n            List of documents most similar to the query.\n        \"\"\"\n        embedding = self._embedding.embed_query(query)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/lancedb.html"}720{"id": "20346f865f71-2", "text": "\"\"\"\n        embedding = self._embedding.embed_query(query)\n        docs = self._connection.search(embedding).limit(k).to_df()\n        return [\n            Document(\n                page_content=row[self._text_key],\n                metadata=row[docs.columns != self._text_key],\n            )\n            for _, row in docs.iterrows()\n        ]\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        connection: Any = None,\n        vector_key: Optional[str] = \"vector\",\n        id_key: Optional[str] = \"id\",\n        text_key: Optional[str] = \"text\",\n        **kwargs: Any,\n    ) -> LanceDB:\n        instance = LanceDB(\n            connection,\n            embedding,\n            vector_key,\n            id_key,\n            text_key,\n        )\n        instance.add_texts(texts, metadatas=metadatas, **kwargs)\n        return instance\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/lancedb.html"}721{"id": "2b5c46bced14-0", "text": "Source code for langchain.vectorstores.annoy\n\"\"\"Wrapper around Annoy vector database.\"\"\"\nfrom __future__ import annotations\nimport os\nimport pickle\nimport uuid\nfrom configparser import ConfigParser\nfrom pathlib import Path\nfrom typing import Any, Callable, Dict, Iterable, List, Optional, Tuple\nimport numpy as np\nfrom langchain.docstore.base import Docstore\nfrom langchain.docstore.document import Document\nfrom langchain.docstore.in_memory import InMemoryDocstore\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nINDEX_METRICS = frozenset([\"angular\", \"euclidean\", \"manhattan\", \"hamming\", \"dot\"])\nDEFAULT_METRIC = \"angular\"\ndef dependable_annoy_import() -> Any:\n    \"\"\"Import annoy if available, otherwise raise error.\"\"\"\n    try:\n        import annoy\n    except ImportError:\n        raise ValueError(\n            \"Could not import annoy python package. \"\n            \"Please install it with `pip install --user annoy` \"\n        )\n    return annoy\n[docs]class Annoy(VectorStore):\n    \"\"\"Wrapper around Annoy vector database.\n    To use, you should have the ``annoy`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain import Annoy\n            db = Annoy(embedding_function, index, docstore, index_to_docstore_id)\n    \"\"\"\n    def __init__(\n        self,\n        embedding_function: Callable,\n        index: Any,\n        metric: str,\n        docstore: Docstore,\n        index_to_docstore_id: Dict[int, str],\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        self.embedding_function = embedding_function", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}722{"id": "2b5c46bced14-1", "text": "):\n        \"\"\"Initialize with necessary components.\"\"\"\n        self.embedding_function = embedding_function\n        self.index = index\n        self.metric = metric\n        self.docstore = docstore\n        self.index_to_docstore_id = index_to_docstore_id\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        raise NotImplementedError(\n            \"Annoy does not allow to add new data once the index is build.\"\n        )\n[docs]    def process_index_results(\n        self, idxs: List[int], dists: List[float]\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Turns annoy results into a list of documents and scores.\n        Args:\n            idxs: List of indices of the documents in the index.\n            dists: List of distances of the documents in the index.\n        Returns:\n            List of Documents and scores.\n        \"\"\"\n        docs = []\n        for idx, dist in zip(idxs, dists):\n            _id = self.index_to_docstore_id[idx]\n            doc = self.docstore.search(_id)\n            if not isinstance(doc, Document):\n                raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n            docs.append((doc, dist))\n        return docs\n[docs]    def similarity_search_with_score_by_vector(\n        self, embedding: List[float], k: int = 4, search_k: int = -1\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}723{"id": "2b5c46bced14-2", "text": "Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        idxs, dists = self.index.get_nns_by_vector(\n            embedding, k, search_k=search_k, include_distances=True\n        )\n        return self.process_index_results(idxs, dists)\n[docs]    def similarity_search_with_score_by_index(\n        self, docstore_index: int, k: int = 4, search_k: int = -1\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        idxs, dists = self.index.get_nns_by_item(\n            docstore_index, k, search_k=search_k, include_distances=True\n        )\n        return self.process_index_results(idxs, dists)\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4, search_k: int = -1\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}724{"id": "2b5c46bced14-3", "text": "k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        embedding = self.embedding_function(query)\n        docs = self.similarity_search_with_score_by_vector(embedding, k, search_k)\n        return docs\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, search_k: int = -1, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the embedding.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score_by_vector(\n            embedding, k, search_k\n        )\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def similarity_search_by_index(\n        self, docstore_index: int, k: int = 4, search_k: int = -1, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to docstore_index.\n        Args:\n            docstore_index: Index of document in docstore\n            k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the embedding.\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}725{"id": "2b5c46bced14-4", "text": "Returns:\n            List of Documents most similar to the embedding.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score_by_index(\n            docstore_index, k, search_k\n        )\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, search_k: int = -1, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            search_k: inspect up to search_k nodes which defaults\n                to n_trees * n if not provided\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k, search_k)\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            k: Number of Documents to return. Defaults to 4.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}726{"id": "2b5c46bced14-5", "text": "of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        idxs = self.index.get_nns_by_vector(\n            embedding, fetch_k, search_k=-1, include_distances=False\n        )\n        embeddings = [self.index.get_item_vector(i) for i in idxs]\n        mmr_selected = maximal_marginal_relevance(\n            np.array([embedding], dtype=np.float32),\n            embeddings,\n            k=k,\n            lambda_mult=lambda_mult,\n        )\n        # ignore the -1's if not enough docs are returned/indexed\n        selected_indices = [idxs[i] for i in mmr_selected if i != -1]\n        docs = []\n        for i in selected_indices:\n            _id = self.index_to_docstore_id[i]\n            doc = self.docstore.search(_id)\n            if not isinstance(doc, Document):\n                raise ValueError(f\"Could not find document for id {_id}, got {doc}\")\n            docs.append(doc)\n        return docs\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}727{"id": "2b5c46bced14-6", "text": "k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        embedding = self.embedding_function(query)\n        docs = self.max_marginal_relevance_search_by_vector(\n            embedding, k, fetch_k, lambda_mult=lambda_mult\n        )\n        return docs\n    @classmethod\n    def __from(\n        cls,\n        texts: List[str],\n        embeddings: List[List[float]],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        metric: str = DEFAULT_METRIC,\n        trees: int = 100,\n        n_jobs: int = -1,\n        **kwargs: Any,\n    ) -> Annoy:\n        if metric not in INDEX_METRICS:\n            raise ValueError(\n                (\n                    f\"Unsupported distance metric: {metric}. \"\n                    f\"Expected one of {list(INDEX_METRICS)}\"\n                )\n            )\n        annoy = dependable_annoy_import()\n        if not embeddings:\n            raise ValueError(\"embeddings must be provided to build AnnoyIndex\")\n        f = len(embeddings[0])\n        index = annoy.AnnoyIndex(f, metric=metric)\n        for i, emb in enumerate(embeddings):\n            index.add_item(i, emb)\n        index.build(trees, n_jobs=n_jobs)\n        documents = []\n        for i, text in enumerate(texts):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}728{"id": "2b5c46bced14-7", "text": "documents = []\n        for i, text in enumerate(texts):\n            metadata = metadatas[i] if metadatas else {}\n            documents.append(Document(page_content=text, metadata=metadata))\n        index_to_id = {i: str(uuid.uuid4()) for i in range(len(documents))}\n        docstore = InMemoryDocstore(\n            {index_to_id[i]: doc for i, doc in enumerate(documents)}\n        )\n        return cls(embedding.embed_query, index, metric, docstore, index_to_id)\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        metric: str = DEFAULT_METRIC,\n        trees: int = 100,\n        n_jobs: int = -1,\n        **kwargs: Any,\n    ) -> Annoy:\n        \"\"\"Construct Annoy wrapper from raw documents.\n        Args:\n            texts: List of documents to index.\n            embedding: Embedding function to use.\n            metadatas: List of metadata dictionaries to associate with documents.\n            metric: Metric to use for indexing. Defaults to \"angular\".\n            trees: Number of trees to use for indexing. Defaults to 100.\n            n_jobs: Number of jobs to use for indexing. Defaults to -1.\n        This is a user friendly interface that:\n            1. Embeds documents.\n            2. Creates an in memory docstore\n            3. Initializes the Annoy database\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import Annoy\n                from langchain.embeddings import OpenAIEmbeddings", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}729{"id": "2b5c46bced14-8", "text": "from langchain import Annoy\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                index = Annoy.from_texts(texts, embeddings)\n        \"\"\"\n        embeddings = embedding.embed_documents(texts)\n        return cls.__from(\n            texts, embeddings, embedding, metadatas, metric, trees, n_jobs, **kwargs\n        )\n[docs]    @classmethod\n    def from_embeddings(\n        cls,\n        text_embeddings: List[Tuple[str, List[float]]],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        metric: str = DEFAULT_METRIC,\n        trees: int = 100,\n        n_jobs: int = -1,\n        **kwargs: Any,\n    ) -> Annoy:\n        \"\"\"Construct Annoy wrapper from embeddings.\n        Args:\n            text_embeddings: List of tuples of (text, embedding)\n            embedding: Embedding function to use.\n            metadatas: List of metadata dictionaries to associate with documents.\n            metric: Metric to use for indexing. Defaults to \"angular\".\n            trees: Number of trees to use for indexing. Defaults to 100.\n            n_jobs: Number of jobs to use for indexing. Defaults to -1\n        This is a user friendly interface that:\n            1. Creates an in memory docstore with provided embeddings\n            2. Initializes the Annoy database\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import Annoy\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                text_embeddings = embeddings.embed_documents(texts)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}730{"id": "2b5c46bced14-9", "text": "embeddings = OpenAIEmbeddings()\n                text_embeddings = embeddings.embed_documents(texts)\n                text_embedding_pairs = list(zip(texts, text_embeddings))\n                db = Annoy.from_embeddings(text_embedding_pairs, embeddings)\n        \"\"\"\n        texts = [t[0] for t in text_embeddings]\n        embeddings = [t[1] for t in text_embeddings]\n        return cls.__from(\n            texts, embeddings, embedding, metadatas, metric, trees, n_jobs, **kwargs\n        )\n[docs]    def save_local(self, folder_path: str, prefault: bool = False) -> None:\n        \"\"\"Save Annoy index, docstore, and index_to_docstore_id to disk.\n        Args:\n            folder_path: folder path to save index, docstore,\n                and index_to_docstore_id to.\n            prefault: Whether to pre-load the index into memory.\n        \"\"\"\n        path = Path(folder_path)\n        os.makedirs(path, exist_ok=True)\n        # save index, index config, docstore and index_to_docstore_id\n        config_object = ConfigParser()\n        config_object[\"ANNOY\"] = {\n            \"f\": self.index.f,\n            \"metric\": self.metric,\n        }\n        self.index.save(str(path / \"index.annoy\"), prefault=prefault)\n        with open(path / \"index.pkl\", \"wb\") as file:\n            pickle.dump((self.docstore, self.index_to_docstore_id, config_object), file)\n[docs]    @classmethod\n    def load_local(\n        cls,\n        folder_path: str,\n        embeddings: Embeddings,\n    ) -> Annoy:\n        \"\"\"Load Annoy index, docstore, and index_to_docstore_id to disk.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}731{"id": "2b5c46bced14-10", "text": "Args:\n            folder_path: folder path to load index, docstore,\n                and index_to_docstore_id from.\n            embeddings: Embeddings to use when generating queries.\n        \"\"\"\n        path = Path(folder_path)\n        # load index separately since it is not picklable\n        annoy = dependable_annoy_import()\n        # load docstore and index_to_docstore_id\n        with open(path / \"index.pkl\", \"rb\") as file:\n            docstore, index_to_docstore_id, config_object = pickle.load(file)\n        f = int(config_object[\"ANNOY\"][\"f\"])\n        metric = config_object[\"ANNOY\"][\"metric\"]\n        index = annoy.AnnoyIndex(f, metric=metric)\n        index.load(str(path / \"index.annoy\"))\n        return cls(\n            embeddings.embed_query, index, metric, docstore, index_to_docstore_id\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/annoy.html"}732{"id": "28616f5d585f-0", "text": "Source code for langchain.vectorstores.redis\n\"\"\"Wrapper around Redis vector database.\"\"\"\nfrom __future__ import annotations\nimport json\nimport logging\nimport uuid\nfrom typing import (\n    TYPE_CHECKING,\n    Any,\n    Callable,\n    Dict,\n    Iterable,\n    List,\n    Literal,\n    Mapping,\n    Optional,\n    Tuple,\n    Type,\n)\nimport numpy as np\nfrom pydantic import BaseModel, root_validator\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nfrom langchain.vectorstores.base import VectorStore, VectorStoreRetriever\nlogger = logging.getLogger(__name__)\nif TYPE_CHECKING:\n    from redis.client import Redis as RedisType\n    from redis.commands.search.query import Query\n# required modules\nREDIS_REQUIRED_MODULES = [\n    {\"name\": \"search\", \"ver\": 20400},\n    {\"name\": \"searchlight\", \"ver\": 20400},\n]\n# distance mmetrics\nREDIS_DISTANCE_METRICS = Literal[\"COSINE\", \"IP\", \"L2\"]\ndef _check_redis_module_exist(client: RedisType, required_modules: List[dict]) -> None:\n    \"\"\"Check if the correct Redis modules are installed.\"\"\"\n    installed_modules = client.module_list()\n    installed_modules = {\n        module[b\"name\"].decode(\"utf-8\"): module for module in installed_modules\n    }\n    for module in required_modules:\n        if module[\"name\"] in installed_modules and int(\n            installed_modules[module[\"name\"]][b\"ver\"]\n        ) >= int(module[\"ver\"]):\n            return\n    # otherwise raise error\n    error_message = (\n        \"Redis cannot be used as a vector database without RediSearch >=2.4\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}733{"id": "28616f5d585f-1", "text": "\"Redis cannot be used as a vector database without RediSearch >=2.4\"\n        \"Please head to https://redis.io/docs/stack/search/quick_start/\"\n        \"to know more about installing the RediSearch module within Redis Stack.\"\n    )\n    logging.error(error_message)\n    raise ValueError(error_message)\ndef _check_index_exists(client: RedisType, index_name: str) -> bool:\n    \"\"\"Check if Redis index exists.\"\"\"\n    try:\n        client.ft(index_name).info()\n    except:  # noqa: E722\n        logger.info(\"Index does not exist\")\n        return False\n    logger.info(\"Index already exists\")\n    return True\ndef _redis_key(prefix: str) -> str:\n    \"\"\"Redis key schema for a given prefix.\"\"\"\n    return f\"{prefix}:{uuid.uuid4().hex}\"\ndef _redis_prefix(index_name: str) -> str:\n    \"\"\"Redis key prefix for a given index.\"\"\"\n    return f\"doc:{index_name}\"\ndef _default_relevance_score(val: float) -> float:\n    return 1 - val\n[docs]class Redis(VectorStore):\n    \"\"\"Wrapper around Redis vector database.\n    To use, you should have the ``redis`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.vectorstores import Redis\n            from langchain.embeddings import OpenAIEmbeddings\n            embeddings = OpenAIEmbeddings()\n            vectorstore = Redis(\n                redis_url=\"redis://username:password@localhost:6379\"\n                index_name=\"my-index\",\n                embedding_function=embeddings.embed_query,\n            )\n    \"\"\"\n    def __init__(\n        self,\n        redis_url: str,\n        index_name: str,\n        embedding_function: Callable,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}734{"id": "28616f5d585f-2", "text": "redis_url: str,\n        index_name: str,\n        embedding_function: Callable,\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        vector_key: str = \"content_vector\",\n        relevance_score_fn: Optional[\n            Callable[[float], float]\n        ] = _default_relevance_score,\n        **kwargs: Any,\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        try:\n            import redis\n        except ImportError:\n            raise ValueError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis>=4.1.0`.\"\n            )\n        self.embedding_function = embedding_function\n        self.index_name = index_name\n        try:\n            # connect to redis from url\n            redis_client = redis.from_url(redis_url, **kwargs)\n            # check if redis has redisearch module installed\n            _check_redis_module_exist(redis_client, REDIS_REQUIRED_MODULES)\n        except ValueError as e:\n            raise ValueError(f\"Redis failed to connect: {e}\")\n        self.client = redis_client\n        self.content_key = content_key\n        self.metadata_key = metadata_key\n        self.vector_key = vector_key\n        self.relevance_score_fn = relevance_score_fn\n    def _create_index(\n        self, dim: int = 1536, distance_metric: REDIS_DISTANCE_METRICS = \"COSINE\"\n    ) -> None:\n        try:\n            from redis.commands.search.field import TextField, VectorField\n            from redis.commands.search.indexDefinition import IndexDefinition, IndexType\n        except ImportError:\n            raise ValueError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        # Check if index exists", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}735{"id": "28616f5d585f-3", "text": ")\n        # Check if index exists\n        if not _check_index_exists(self.client, self.index_name):\n            # Define schema\n            schema = (\n                TextField(name=self.content_key),\n                TextField(name=self.metadata_key),\n                VectorField(\n                    self.vector_key,\n                    \"FLAT\",\n                    {\n                        \"TYPE\": \"FLOAT32\",\n                        \"DIM\": dim,\n                        \"DISTANCE_METRIC\": distance_metric,\n                    },\n                ),\n            )\n            prefix = _redis_prefix(self.index_name)\n            # Create Redis Index\n            self.client.ft(self.index_name).create_index(\n                fields=schema,\n                definition=IndexDefinition(prefix=[prefix], index_type=IndexType.HASH),\n            )\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        embeddings: Optional[List[List[float]]] = None,\n        keys: Optional[List[str]] = None,\n        batch_size: int = 1000,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Add more texts to the vectorstore.\n        Args:\n            texts (Iterable[str]): Iterable of strings/text to add to the vectorstore.\n            metadatas (Optional[List[dict]], optional): Optional list of metadatas.\n                Defaults to None.\n            embeddings (Optional[List[List[float]]], optional): Optional pre-generated\n                embeddings. Defaults to None.\n            keys (Optional[List[str]], optional): Optional key values to use as ids.\n                Defaults to None.\n            batch_size (int, optional): Batch size to use for writes. Defaults to 1000.\n        Returns:\n            List[str]: List of ids added to the vectorstore\n        \"\"\"\n        ids = []", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}736{"id": "28616f5d585f-4", "text": "List[str]: List of ids added to the vectorstore\n        \"\"\"\n        ids = []\n        prefix = _redis_prefix(self.index_name)\n        # Write data to redis\n        pipeline = self.client.pipeline(transaction=False)\n        for i, text in enumerate(texts):\n            # Use provided values by default or fallback\n            key = keys[i] if keys else _redis_key(prefix)\n            metadata = metadatas[i] if metadatas else {}\n            embedding = embeddings[i] if embeddings else self.embedding_function(text)\n            pipeline.hset(\n                key,\n                mapping={\n                    self.content_key: text,\n                    self.vector_key: np.array(embedding, dtype=np.float32).tobytes(),\n                    self.metadata_key: json.dumps(metadata),\n                },\n            )\n            ids.append(key)\n            # Write batch\n            if i % batch_size == 0:\n                pipeline.execute()\n        # Cleanup final batch\n        pipeline.execute()\n        return ids\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"\n        Returns the most similar indexed documents to the query text.\n        Args:\n            query (str): The query text for which to find similar documents.\n            k (int): The number of documents to return. Default is 4.\n        Returns:\n            List[Document]: A list of documents that are most similar to the query text.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k=k)\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def similarity_search_limit_score(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}737{"id": "28616f5d585f-5", "text": "[docs]    def similarity_search_limit_score(\n        self, query: str, k: int = 4, score_threshold: float = 0.2, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"\n        Returns the most similar indexed documents to the query text within the\n        score_threshold range.\n        Args:\n            query (str): The query text for which to find similar documents.\n            k (int): The number of documents to return. Default is 4.\n            score_threshold (float): The minimum matching score required for a document\n            to be considered a match. Defaults to 0.2.\n            Because the similarity calculation algorithm is based on cosine similarity,\n            the smaller the angle, the higher the similarity.\n        Returns:\n            List[Document]: A list of documents that are most similar to the query text,\n            including the match score for each document.\n        Note:\n            If there are no documents that satisfy the score_threshold value,\n            an empty list is returned.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k=k)\n        return [doc for doc, score in docs_and_scores if score < score_threshold]\n    def _prepare_query(self, k: int) -> Query:\n        try:\n            from redis.commands.search.query import Query\n        except ImportError:\n            raise ValueError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        # Prepare the Query\n        hybrid_fields = \"*\"\n        base_query = (\n            f\"{hybrid_fields}=>[KNN {k} @{self.vector_key} $vector AS vector_score]\"\n        )\n        return_fields = [self.metadata_key, self.content_key, \"vector_score\"]\n        return (", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}738{"id": "28616f5d585f-6", "text": "return (\n            Query(base_query)\n            .return_fields(*return_fields)\n            .sort_by(\"vector_score\")\n            .paging(0, k)\n            .dialect(2)\n        )\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        # Creates embedding vector from user query\n        embedding = self.embedding_function(query)\n        # Creates Redis query\n        redis_query = self._prepare_query(k)\n        params_dict: Mapping[str, str] = {\n            \"vector\": np.array(embedding)  # type: ignore\n            .astype(dtype=np.float32)\n            .tobytes()\n        }\n        # Perform vector search\n        results = self.client.ft(self.index_name).search(redis_query, params_dict)\n        # Prepare document results\n        docs = [\n            (\n                Document(\n                    page_content=result.content, metadata=json.loads(result.metadata)\n                ),\n                float(result.vector_score),\n            )\n            for result in results.docs\n        ]\n        return docs\n    def _similarity_search_with_relevance_scores(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and relevance scores, normalized on a scale from 0 to 1.\n        0 is dissimilar, 1 is most similar.\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}739{"id": "28616f5d585f-7", "text": "0 is dissimilar, 1 is most similar.\n        \"\"\"\n        if self.relevance_score_fn is None:\n            raise ValueError(\n                \"relevance_score_fn must be provided to\"\n                \" Redis constructor to normalize scores\"\n            )\n        docs_and_scores = self.similarity_search_with_score(query, k=k)\n        return [(doc, self.relevance_score_fn(score)) for doc, score in docs_and_scores]\n[docs]    @classmethod\n    def from_texts_return_keys(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        index_name: Optional[str] = None,\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        vector_key: str = \"content_vector\",\n        distance_metric: REDIS_DISTANCE_METRICS = \"COSINE\",\n        **kwargs: Any,\n    ) -> Tuple[Redis, List[str]]:\n        \"\"\"Create a Redis vectorstore from raw documents.\n        This is a user-friendly interface that:\n            1. Embeds documents.\n            2. Creates a new index for the embeddings in Redis.\n            3. Adds the documents to the newly created Redis index.\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain.vectorstores import Redis\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                redisearch = RediSearch.from_texts(\n                    texts,\n                    embeddings,\n                    redis_url=\"redis://username:password@localhost:6379\"\n                )\n        \"\"\"\n        redis_url = get_from_dict_or_env(kwargs, \"redis_url\", \"REDIS_URL\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}740{"id": "28616f5d585f-8", "text": "redis_url = get_from_dict_or_env(kwargs, \"redis_url\", \"REDIS_URL\")\n        if \"redis_url\" in kwargs:\n            kwargs.pop(\"redis_url\")\n        # Name of the search index if not given\n        if not index_name:\n            index_name = uuid.uuid4().hex\n        # Create instance\n        instance = cls(\n            redis_url,\n            index_name,\n            embedding.embed_query,\n            content_key=content_key,\n            metadata_key=metadata_key,\n            vector_key=vector_key,\n            **kwargs,\n        )\n        # Create embeddings over documents\n        embeddings = embedding.embed_documents(texts)\n        # Create the search index\n        instance._create_index(dim=len(embeddings[0]), distance_metric=distance_metric)\n        # Add data to Redis\n        keys = instance.add_texts(texts, metadatas, embeddings)\n        return instance, keys\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Redis],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        index_name: Optional[str] = None,\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        vector_key: str = \"content_vector\",\n        **kwargs: Any,\n    ) -> Redis:\n        \"\"\"Create a Redis vectorstore from raw documents.\n        This is a user-friendly interface that:\n            1. Embeds documents.\n            2. Creates a new index for the embeddings in Redis.\n            3. Adds the documents to the newly created Redis index.\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain.vectorstores import Redis", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}741{"id": "28616f5d585f-9", "text": "Example:\n            .. code-block:: python\n                from langchain.vectorstores import Redis\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                redisearch = RediSearch.from_texts(\n                    texts,\n                    embeddings,\n                    redis_url=\"redis://username:password@localhost:6379\"\n                )\n        \"\"\"\n        instance, _ = cls.from_texts_return_keys(\n            texts,\n            embedding,\n            metadatas=metadatas,\n            index_name=index_name,\n            content_key=content_key,\n            metadata_key=metadata_key,\n            vector_key=vector_key,\n            **kwargs,\n        )\n        return instance\n[docs]    @staticmethod\n    def drop_index(\n        index_name: str,\n        delete_documents: bool,\n        **kwargs: Any,\n    ) -> bool:\n        \"\"\"\n        Drop a Redis search index.\n        Args:\n            index_name (str): Name of the index to drop.\n            delete_documents (bool): Whether to drop the associated documents.\n        Returns:\n            bool: Whether or not the drop was successful.\n        \"\"\"\n        redis_url = get_from_dict_or_env(kwargs, \"redis_url\", \"REDIS_URL\")\n        try:\n            import redis\n        except ImportError:\n            raise ValueError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        try:\n            # We need to first remove redis_url from kwargs,\n            # otherwise passing it to Redis will result in an error.\n            if \"redis_url\" in kwargs:\n                kwargs.pop(\"redis_url\")\n            client = redis.from_url(url=redis_url, **kwargs)\n        except ValueError as e:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}742{"id": "28616f5d585f-10", "text": "except ValueError as e:\n            raise ValueError(f\"Your redis connected error: {e}\")\n        # Check if index exists\n        try:\n            client.ft(index_name).dropindex(delete_documents)\n            logger.info(\"Drop index\")\n            return True\n        except:  # noqa: E722\n            # Index not exist\n            return False\n[docs]    @classmethod\n    def from_existing_index(\n        cls,\n        embedding: Embeddings,\n        index_name: str,\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        vector_key: str = \"content_vector\",\n        **kwargs: Any,\n    ) -> Redis:\n        \"\"\"Connect to an existing Redis index.\"\"\"\n        redis_url = get_from_dict_or_env(kwargs, \"redis_url\", \"REDIS_URL\")\n        try:\n            import redis\n        except ImportError:\n            raise ValueError(\n                \"Could not import redis python package. \"\n                \"Please install it with `pip install redis`.\"\n            )\n        try:\n            # We need to first remove redis_url from kwargs,\n            # otherwise passing it to Redis will result in an error.\n            if \"redis_url\" in kwargs:\n                kwargs.pop(\"redis_url\")\n            client = redis.from_url(url=redis_url, **kwargs)\n            # check if redis has redisearch module installed\n            _check_redis_module_exist(client, REDIS_REQUIRED_MODULES)\n            # ensure that the index already exists\n            assert _check_index_exists(\n                client, index_name\n            ), f\"Index {index_name} does not exist\"\n        except Exception as e:\n            raise ValueError(f\"Redis failed to connect: {e}\")\n        return cls(\n            redis_url,\n            index_name,\n            embedding.embed_query,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}743{"id": "28616f5d585f-11", "text": "return cls(\n            redis_url,\n            index_name,\n            embedding.embed_query,\n            content_key=content_key,\n            metadata_key=metadata_key,\n            vector_key=vector_key,\n            **kwargs,\n        )\n[docs]    def as_retriever(self, **kwargs: Any) -> RedisVectorStoreRetriever:\n        return RedisVectorStoreRetriever(vectorstore=self, **kwargs)\nclass RedisVectorStoreRetriever(VectorStoreRetriever, BaseModel):\n    vectorstore: Redis\n    search_type: str = \"similarity\"\n    k: int = 4\n    score_threshold: float = 0.4\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        arbitrary_types_allowed = True\n    @root_validator()\n    def validate_search_type(cls, values: Dict) -> Dict:\n        \"\"\"Validate search type.\"\"\"\n        if \"search_type\" in values:\n            search_type = values[\"search_type\"]\n            if search_type not in (\"similarity\", \"similarity_limit\"):\n                raise ValueError(f\"search_type of {search_type} not allowed.\")\n        return values\n    def get_relevant_documents(self, query: str) -> List[Document]:\n        if self.search_type == \"similarity\":\n            docs = self.vectorstore.similarity_search(query, k=self.k)\n        elif self.search_type == \"similarity_limit\":\n            docs = self.vectorstore.similarity_search_limit_score(\n                query, k=self.k, score_threshold=self.score_threshold\n            )\n        else:\n            raise ValueError(f\"search_type of {self.search_type} not allowed.\")\n        return docs\n    async def aget_relevant_documents(self, query: str) -> List[Document]:\n        raise NotImplementedError(\"RedisVectorStoreRetriever does not support async\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}744{"id": "28616f5d585f-12", "text": "raise NotImplementedError(\"RedisVectorStoreRetriever does not support async\")\n    def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        return self.vectorstore.add_documents(documents, **kwargs)\n    async def aadd_documents(\n        self, documents: List[Document], **kwargs: Any\n    ) -> List[str]:\n        \"\"\"Add documents to vectorstore.\"\"\"\n        return await self.vectorstore.aadd_documents(documents, **kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/redis.html"}745{"id": "8e082a667bb3-0", "text": "Source code for langchain.vectorstores.supabase\nfrom __future__ import annotations\nfrom itertools import repeat\nfrom typing import (\n    TYPE_CHECKING,\n    Any,\n    Iterable,\n    List,\n    Optional,\n    Tuple,\n    Type,\n    Union,\n)\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nif TYPE_CHECKING:\n    import supabase\n[docs]class SupabaseVectorStore(VectorStore):\n    \"\"\"VectorStore for a Supabase postgres database. Assumes you have the `pgvector`\n    extension installed and a `match_documents` (or similar) function. For more details:\n    https://js.langchain.com/docs/modules/indexes/vector_stores/integrations/supabase\n    You can implement your own `match_documents` function in order to limit the search\n    space to a subset of documents based on your own authorization or business logic.\n    Note that the Supabase Python client does not yet support async operations.\n    If you'd like to use `max_marginal_relevance_search`, please review the instructions\n    below on modifying the `match_documents` function to return matched embeddings.\n    \"\"\"\n    _client: supabase.client.Client\n    # This is the embedding function. Don't confuse with the embedding vectors.\n    # We should perhaps rename the underlying Embedding base class to EmbeddingFunction\n    # or something\n    _embedding: Embeddings\n    table_name: str\n    query_name: str\n    def __init__(\n        self,\n        client: supabase.client.Client,\n        embedding: Embeddings,\n        table_name: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}746{"id": "8e082a667bb3-1", "text": "embedding: Embeddings,\n        table_name: str,\n        query_name: Union[str, None] = None,\n    ) -> None:\n        \"\"\"Initialize with supabase client.\"\"\"\n        try:\n            import supabase  # noqa: F401\n        except ImportError:\n            raise ValueError(\n                \"Could not import supabase python package. \"\n                \"Please install it with `pip install supabase`.\"\n            )\n        self._client = client\n        self._embedding: Embeddings = embedding\n        self.table_name = table_name or \"documents\"\n        self.query_name = query_name or \"match_documents\"\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict[Any, Any]]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        docs = self._texts_to_documents(texts, metadatas)\n        vectors = self._embedding.embed_documents(list(texts))\n        return self.add_vectors(vectors, docs)\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[\"SupabaseVectorStore\"],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        client: Optional[supabase.client.Client] = None,\n        table_name: Optional[str] = \"documents\",\n        query_name: Union[str, None] = \"match_documents\",\n        **kwargs: Any,\n    ) -> \"SupabaseVectorStore\":\n        \"\"\"Return VectorStore initialized from texts and embeddings.\"\"\"\n        if not client:\n            raise ValueError(\"Supabase client is required.\")\n        if not table_name:\n            raise ValueError(\"Supabase document table_name is required.\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}747{"id": "8e082a667bb3-2", "text": "if not table_name:\n            raise ValueError(\"Supabase document table_name is required.\")\n        embeddings = embedding.embed_documents(texts)\n        docs = cls._texts_to_documents(texts, metadatas)\n        _ids = cls._add_vectors(client, table_name, embeddings, docs)\n        return cls(\n            client=client,\n            embedding=embedding,\n            table_name=table_name,\n            query_name=query_name,\n        )\n[docs]    def add_vectors(\n        self, vectors: List[List[float]], documents: List[Document]\n    ) -> List[str]:\n        return self._add_vectors(self._client, self.table_name, vectors, documents)\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        vectors = self._embedding.embed_documents([query])\n        return self.similarity_search_by_vector(vectors[0], k)\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        result = self.similarity_search_by_vector_with_relevance_scores(embedding, k)\n        documents = [doc for doc, _ in result]\n        return documents\n[docs]    def similarity_search_with_relevance_scores(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        vectors = self._embedding.embed_documents([query])\n        return self.similarity_search_by_vector_with_relevance_scores(vectors[0], k)\n[docs]    def similarity_search_by_vector_with_relevance_scores(\n        self, query: List[float], k: int", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}748{"id": "8e082a667bb3-3", "text": "self, query: List[float], k: int\n    ) -> List[Tuple[Document, float]]:\n        match_documents_params = dict(query_embedding=query, match_count=k)\n        res = self._client.rpc(self.query_name, match_documents_params).execute()\n        match_result = [\n            (\n                Document(\n                    metadata=search.get(\"metadata\", {}),  # type: ignore\n                    page_content=search.get(\"content\", \"\"),\n                ),\n                search.get(\"similarity\", 0.0),\n            )\n            for search in res.data\n            if search.get(\"content\")\n        ]\n        return match_result\n[docs]    def similarity_search_by_vector_returning_embeddings(\n        self, query: List[float], k: int\n    ) -> List[Tuple[Document, float, np.ndarray[np.float32, Any]]]:\n        match_documents_params = dict(query_embedding=query, match_count=k)\n        res = self._client.rpc(self.query_name, match_documents_params).execute()\n        match_result = [\n            (\n                Document(\n                    metadata=search.get(\"metadata\", {}),  # type: ignore\n                    page_content=search.get(\"content\", \"\"),\n                ),\n                search.get(\"similarity\", 0.0),\n                # Supabase returns a vector type as its string represation (!).\n                # This is a hack to convert the string to numpy array.\n                np.fromstring(\n                    search.get(\"embedding\", \"\").strip(\"[]\"), np.float32, sep=\",\"\n                ),\n            )\n            for search in res.data\n            if search.get(\"content\")\n        ]\n        return match_result\n    @staticmethod\n    def _texts_to_documents(\n        texts: Iterable[str],\n        metadatas: Optional[Iterable[dict[Any, Any]]] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}749{"id": "8e082a667bb3-4", "text": "metadatas: Optional[Iterable[dict[Any, Any]]] = None,\n    ) -> List[Document]:\n        \"\"\"Return list of Documents from list of texts and metadatas.\"\"\"\n        if metadatas is None:\n            metadatas = repeat({})\n        docs = [\n            Document(page_content=text, metadata=metadata)\n            for text, metadata in zip(texts, metadatas)\n        ]\n        return docs\n    @staticmethod\n    def _add_vectors(\n        client: supabase.client.Client,\n        table_name: str,\n        vectors: List[List[float]],\n        documents: List[Document],\n    ) -> List[str]:\n        \"\"\"Add vectors to Supabase table.\"\"\"\n        rows: List[dict[str, Any]] = [\n            {\n                \"content\": documents[idx].page_content,\n                \"embedding\": embedding,\n                \"metadata\": documents[idx].metadata,  # type: ignore\n            }\n            for idx, embedding in enumerate(vectors)\n        ]\n        # According to the SupabaseVectorStore JS implementation, the best chunk size\n        # is 500\n        chunk_size = 500\n        id_list: List[str] = []\n        for i in range(0, len(rows), chunk_size):\n            chunk = rows[i : i + chunk_size]\n            result = client.from_(table_name).insert(chunk).execute()  # type: ignore\n            if len(result.data) == 0:\n                raise Exception(\"Error inserting: No rows added\")\n            # VectorStore.add_vectors returns ids as strings\n            ids = [str(i.get(\"id\")) for i in result.data if i.get(\"id\")]\n            id_list.extend(ids)\n        return id_list\n[docs]    def max_marginal_relevance_search_by_vector(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}750{"id": "8e082a667bb3-5", "text": "return id_list\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        result = self.similarity_search_by_vector_returning_embeddings(\n            embedding, fetch_k\n        )\n        matched_documents = [doc_tuple[0] for doc_tuple in result]\n        matched_embeddings = [doc_tuple[2] for doc_tuple in result]\n        mmr_selected = maximal_marginal_relevance(\n            np.array([embedding], dtype=np.float32),\n            matched_embeddings,\n            k=k,\n            lambda_mult=lambda_mult,\n        )\n        filtered_documents = [matched_documents[i] for i in mmr_selected]\n        return filtered_documents\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}751{"id": "8e082a667bb3-6", "text": "k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        `max_marginal_relevance_search` requires that `query_name` returns matched\n        embeddings alongside the match documents. The following function function\n        demonstrates how to do this:\n        ```sql\n        CREATE FUNCTION match_documents_embeddings(query_embedding vector(1536),\n                                                   match_count int)\n            RETURNS TABLE(\n                id bigint,\n                content text,\n                metadata jsonb,\n                embedding vector(1536),\n                similarity float)\n            LANGUAGE plpgsql\n            AS $$\n            # variable_conflict use_column\n        BEGIN\n            RETURN query\n            SELECT\n                id,\n                content,\n                metadata,\n                embedding,\n                1 -(docstore.embedding <=> query_embedding) AS similarity\n            FROM\n                docstore\n            ORDER BY\n                docstore.embedding <=> query_embedding\n            LIMIT match_count;\n        END;\n        $$;```\n        \"\"\"\n        embedding = self._embedding.embed_documents([query])", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}752{"id": "8e082a667bb3-7", "text": "$$;```\n        \"\"\"\n        embedding = self._embedding.embed_documents([query])\n        docs = self.max_marginal_relevance_search_by_vector(\n            embedding[0], k, fetch_k, lambda_mult=lambda_mult\n        )\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/supabase.html"}753{"id": "df14a039e6dc-0", "text": "Source code for langchain.vectorstores.qdrant\n\"\"\"Wrapper around Qdrant vector database.\"\"\"\nfrom __future__ import annotations\nimport uuid\nimport warnings\nfrom hashlib import md5\nfrom operator import itemgetter\nfrom typing import (\n    TYPE_CHECKING,\n    Any,\n    Callable,\n    Dict,\n    Iterable,\n    List,\n    Optional,\n    Tuple,\n    Type,\n    Union,\n)\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nif TYPE_CHECKING:\n    from qdrant_client.http import models as rest\nMetadataFilter = Dict[str, Union[str, int, bool, dict, list]]\n[docs]class Qdrant(VectorStore):\n    \"\"\"Wrapper around Qdrant vector database.\n    To use you should have the ``qdrant-client`` package installed.\n    Example:\n        .. code-block:: python\n            from qdrant_client import QdrantClient\n            from langchain import Qdrant\n            client = QdrantClient()\n            collection_name = \"MyCollection\"\n            qdrant = Qdrant(client, collection_name, embedding_function)\n    \"\"\"\n    CONTENT_KEY = \"page_content\"\n    METADATA_KEY = \"metadata\"\n    def __init__(\n        self,\n        client: Any,\n        collection_name: str,\n        embeddings: Optional[Embeddings] = None,\n        content_payload_key: str = CONTENT_KEY,\n        metadata_payload_key: str = METADATA_KEY,\n        embedding_function: Optional[Callable] = None,  # deprecated\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        try:\n            import qdrant_client", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}754{"id": "df14a039e6dc-1", "text": "\"\"\"Initialize with necessary components.\"\"\"\n        try:\n            import qdrant_client\n        except ImportError:\n            raise ValueError(\n                \"Could not import qdrant-client python package. \"\n                \"Please install it with `pip install qdrant-client`.\"\n            )\n        if not isinstance(client, qdrant_client.QdrantClient):\n            raise ValueError(\n                f\"client should be an instance of qdrant_client.QdrantClient, \"\n                f\"got {type(client)}\"\n            )\n        if embeddings is None and embedding_function is None:\n            raise ValueError(\n                \"`embeddings` value can't be None. Pass `Embeddings` instance.\"\n            )\n        if embeddings is not None and embedding_function is not None:\n            raise ValueError(\n                \"Both `embeddings` and `embedding_function` are passed. \"\n                \"Use `embeddings` only.\"\n            )\n        self.embeddings = embeddings\n        self._embeddings_function = embedding_function\n        self.client: qdrant_client.QdrantClient = client\n        self.collection_name = collection_name\n        self.content_payload_key = content_payload_key or self.CONTENT_KEY\n        self.metadata_payload_key = metadata_payload_key or self.METADATA_KEY\n        if embedding_function is not None:\n            warnings.warn(\n                \"Using `embedding_function` is deprecated. \"\n                \"Pass `Embeddings` instance to `embeddings` instead.\"\n            )\n        if not isinstance(embeddings, Embeddings):\n            warnings.warn(\n                \"`embeddings` should be an instance of `Embeddings`.\"\n                \"Using `embeddings` as `embedding_function` which is deprecated\"\n            )\n            self._embeddings_function = embeddings\n            self.embeddings = None", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}755{"id": "df14a039e6dc-2", "text": ")\n            self._embeddings_function = embeddings\n            self.embeddings = None\n    def _embed_query(self, query: str) -> List[float]:\n        \"\"\"Embed query text.\n        Used to provide backward compatibility with `embedding_function` argument.\n        Args:\n            query: Query text.\n        Returns:\n            List of floats representing the query embedding.\n        \"\"\"\n        if self.embeddings is not None:\n            embedding = self.embeddings.embed_query(query)\n        else:\n            if self._embeddings_function is not None:\n                embedding = self._embeddings_function(query)\n            else:\n                raise ValueError(\"Neither of embeddings or embedding_function is set\")\n        return embedding.tolist() if hasattr(embedding, \"tolist\") else embedding\n    def _embed_texts(self, texts: Iterable[str]) -> List[List[float]]:\n        \"\"\"Embed search texts.\n        Used to provide backward compatibility with `embedding_function` argument.\n        Args:\n            texts: Iterable of texts to embed.\n        Returns:\n            List of floats representing the texts embedding.\n        \"\"\"\n        if self.embeddings is not None:\n            embeddings = self.embeddings.embed_documents(list(texts))\n            if hasattr(embeddings, \"tolist\"):\n                embeddings = embeddings.tolist()\n        elif self._embeddings_function is not None:\n            embeddings = []\n            for text in texts:\n                embedding = self._embeddings_function(text)\n                if hasattr(embeddings, \"tolist\"):\n                    embedding = embedding.tolist()\n                embeddings.append(embedding)\n        else:\n            raise ValueError(\"Neither of embeddings or embedding_function is set\")\n        return embeddings\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}756{"id": "df14a039e6dc-3", "text": "metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        from qdrant_client.http import models as rest\n        texts = list(\n            texts\n        )  # otherwise iterable might be exhausted after id calculation\n        ids = [md5(text.encode(\"utf-8\")).hexdigest() for text in texts]\n        self.client.upsert(\n            collection_name=self.collection_name,\n            points=rest.Batch.construct(\n                ids=ids,\n                vectors=self._embed_texts(texts),\n                payloads=self._build_payloads(\n                    texts,\n                    metadatas,\n                    self.content_payload_key,\n                    self.metadata_payload_key,\n                ),\n            ),\n        )\n        return ids\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[MetadataFilter] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter: Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        results = self.similarity_search_with_score(query, k, filter)\n        return list(map(itemgetter(0), results))", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}757{"id": "df14a039e6dc-4", "text": "return list(map(itemgetter(0), results))\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4, filter: Optional[MetadataFilter] = None\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter: Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query and score for each.\n        \"\"\"\n        results = self.client.search(\n            collection_name=self.collection_name,\n            query_vector=self._embed_query(query),\n            query_filter=self._qdrant_filter_from_dict(filter),\n            with_payload=True,\n            limit=k,\n        )\n        return [\n            (\n                self._document_from_scored_point(\n                    result, self.content_payload_key, self.metadata_payload_key\n                ),\n                result.score,\n            )\n            for result in results\n        ]\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n                     Defaults to 20.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}758{"id": "df14a039e6dc-5", "text": "Defaults to 20.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        embedding = self._embed_query(query)\n        results = self.client.search(\n            collection_name=self.collection_name,\n            query_vector=embedding,\n            with_payload=True,\n            with_vectors=True,\n            limit=fetch_k,\n        )\n        embeddings = [result.vector for result in results]\n        mmr_selected = maximal_marginal_relevance(\n            np.array(embedding), embeddings, k=k, lambda_mult=lambda_mult\n        )\n        return [\n            self._document_from_scored_point(\n                results[i], self.content_payload_key, self.metadata_payload_key\n            )\n            for i in mmr_selected\n        ]\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Qdrant],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        location: Optional[str] = None,\n        url: Optional[str] = None,\n        port: Optional[int] = 6333,\n        grpc_port: int = 6334,\n        prefer_grpc: bool = False,\n        https: Optional[bool] = None,\n        api_key: Optional[str] = None,\n        prefix: Optional[str] = None,\n        timeout: Optional[float] = None,\n        host: Optional[str] = None,\n        path: Optional[str] = None,\n        collection_name: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}759{"id": "df14a039e6dc-6", "text": "path: Optional[str] = None,\n        collection_name: Optional[str] = None,\n        distance_func: str = \"Cosine\",\n        content_payload_key: str = CONTENT_KEY,\n        metadata_payload_key: str = METADATA_KEY,\n        **kwargs: Any,\n    ) -> Qdrant:\n        \"\"\"Construct Qdrant wrapper from a list of texts.\n        Args:\n            texts: A list of texts to be indexed in Qdrant.\n            embedding: A subclass of `Embeddings`, responsible for text vectorization.\n            metadatas:\n                An optional list of metadata. If provided it has to be of the same\n                length as a list of texts.\n            location:\n                If `:memory:` - use in-memory Qdrant instance.\n                If `str` - use it as a `url` parameter.\n                If `None` - fallback to relying on `host` and `port` parameters.\n            url: either host or str of \"Optional[scheme], host, Optional[port],\n                Optional[prefix]\". Default: `None`\n            port: Port of the REST API interface. Default: 6333\n            grpc_port: Port of the gRPC interface. Default: 6334\n            prefer_grpc:\n                If true - use gPRC interface whenever possible in custom methods.\n                Default: False\n            https: If true - use HTTPS(SSL) protocol. Default: None\n            api_key: API key for authentication in Qdrant Cloud. Default: None\n            prefix:\n                If not None - add prefix to the REST URL path.\n                Example: service/v1 will result in\n                    http://localhost:6333/service/v1/{qdrant-endpoint} for REST API.\n                Default: None\n            timeout:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}760{"id": "df14a039e6dc-7", "text": "Default: None\n            timeout:\n                Timeout for REST and gRPC API requests.\n                Default: 5.0 seconds for REST and unlimited for gRPC\n            host:\n                Host name of Qdrant service. If url and host are None, set to\n                'localhost'. Default: None\n            path:\n                Path in which the vectors will be stored while using local mode.\n                Default: None\n            collection_name:\n                Name of the Qdrant collection to be used. If not provided,\n                it will be created randomly. Default: None\n            distance_func:\n                Distance function. One of: \"Cosine\" / \"Euclid\" / \"Dot\".\n                Default: \"Cosine\"\n            content_payload_key:\n                A payload key used to store the content of the document.\n                Default: \"page_content\"\n            metadata_payload_key:\n                A payload key used to store the metadata of the document.\n                Default: \"metadata\"\n            **kwargs:\n                Additional arguments passed directly into REST client initialization\n        This is a user friendly interface that:\n            1. Creates embeddings, one for each text\n            2. Initializes the Qdrant database as an in-memory docstore by default\n               (and overridable to a remote docstore)\n            3. Adds the text embeddings to the Qdrant database\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import Qdrant\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                qdrant = Qdrant.from_texts(texts, embeddings, \"localhost\")\n        \"\"\"\n        try:\n            import qdrant_client\n        except ImportError:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}761{"id": "df14a039e6dc-8", "text": "try:\n            import qdrant_client\n        except ImportError:\n            raise ValueError(\n                \"Could not import qdrant-client python package. \"\n                \"Please install it with `pip install qdrant-client`.\"\n            )\n        from qdrant_client.http import models as rest\n        # Just do a single quick embedding to get vector size\n        partial_embeddings = embedding.embed_documents(texts[:1])\n        vector_size = len(partial_embeddings[0])\n        collection_name = collection_name or uuid.uuid4().hex\n        distance_func = distance_func.upper()\n        client = qdrant_client.QdrantClient(\n            location=location,\n            url=url,\n            port=port,\n            grpc_port=grpc_port,\n            prefer_grpc=prefer_grpc,\n            https=https,\n            api_key=api_key,\n            prefix=prefix,\n            timeout=timeout,\n            host=host,\n            path=path,\n            **kwargs,\n        )\n        client.recreate_collection(\n            collection_name=collection_name,\n            vectors_config=rest.VectorParams(\n                size=vector_size,\n                distance=rest.Distance[distance_func],\n            ),\n        )\n        # Now generate the embeddings for all the texts\n        embeddings = embedding.embed_documents(texts)\n        client.upsert(\n            collection_name=collection_name,\n            points=rest.Batch.construct(\n                ids=[md5(text.encode(\"utf-8\")).hexdigest() for text in texts],\n                vectors=embeddings,\n                payloads=cls._build_payloads(\n                    texts, metadatas, content_payload_key, metadata_payload_key\n                ),\n            ),\n        )\n        return cls(\n            client=client,\n            collection_name=collection_name,\n            embeddings=embedding,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}762{"id": "df14a039e6dc-9", "text": "client=client,\n            collection_name=collection_name,\n            embeddings=embedding,\n            content_payload_key=content_payload_key,\n            metadata_payload_key=metadata_payload_key,\n        )\n    @classmethod\n    def _build_payloads(\n        cls,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]],\n        content_payload_key: str,\n        metadata_payload_key: str,\n    ) -> List[dict]:\n        payloads = []\n        for i, text in enumerate(texts):\n            if text is None:\n                raise ValueError(\n                    \"At least one of the texts is None. Please remove it before \"\n                    \"calling .from_texts or .add_texts on Qdrant instance.\"\n                )\n            metadata = metadatas[i] if metadatas is not None else None\n            payloads.append(\n                {\n                    content_payload_key: text,\n                    metadata_payload_key: metadata,\n                }\n            )\n        return payloads\n    @classmethod\n    def _document_from_scored_point(\n        cls,\n        scored_point: Any,\n        content_payload_key: str,\n        metadata_payload_key: str,\n    ) -> Document:\n        return Document(\n            page_content=scored_point.payload.get(content_payload_key),\n            metadata=scored_point.payload.get(metadata_payload_key) or {},\n        )\n    def _build_condition(self, key: str, value: Any) -> List[rest.FieldCondition]:\n        from qdrant_client.http import models as rest\n        out = []\n        if isinstance(value, dict):\n            for _key, value in value.items():\n                out.extend(self._build_condition(f\"{key}.{_key}\", value))\n        elif isinstance(value, list):\n            for _value in value:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}763{"id": "df14a039e6dc-10", "text": "elif isinstance(value, list):\n            for _value in value:\n                if isinstance(_value, dict):\n                    out.extend(self._build_condition(f\"{key}[]\", _value))\n                else:\n                    out.extend(self._build_condition(f\"{key}\", _value))\n        else:\n            out.append(\n                rest.FieldCondition(\n                    key=f\"{self.metadata_payload_key}.{key}\",\n                    match=rest.MatchValue(value=value),\n                )\n            )\n        return out\n    def _qdrant_filter_from_dict(\n        self, filter: Optional[MetadataFilter]\n    ) -> Optional[rest.Filter]:\n        from qdrant_client.http import models as rest\n        if not filter:\n            return None\n        return rest.Filter(\n            must=[\n                condition\n                for key, value in filter.items()\n                for condition in self._build_condition(key, value)\n            ]\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/qdrant.html"}764{"id": "860bb0b94ce7-0", "text": "Source code for langchain.vectorstores.milvus\n\"\"\"Wrapper around the Milvus vector database.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom typing import Any, Iterable, List, Optional, Tuple, Union\nfrom uuid import uuid4\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nlogger = logging.getLogger(__name__)\nDEFAULT_MILVUS_CONNECTION = {\n    \"host\": \"localhost\",\n    \"port\": \"19530\",\n    \"user\": \"\",\n    \"password\": \"\",\n    \"secure\": False,\n}\n[docs]class Milvus(VectorStore):\n    \"\"\"Wrapper around the Milvus vector database.\"\"\"\n    def __init__(\n        self,\n        embedding_function: Embeddings,\n        collection_name: str = \"LangChainCollection\",\n        connection_args: Optional[dict[str, Any]] = None,\n        consistency_level: str = \"Session\",\n        index_params: Optional[dict] = None,\n        search_params: Optional[dict] = None,\n        drop_old: Optional[bool] = False,\n    ):\n        \"\"\"Initialize wrapper around the milvus vector database.\n        In order to use this you need to have `pymilvus` installed and a\n        running Milvus/Zilliz Cloud instance.\n        See the following documentation for how to run a Milvus instance:\n        https://milvus.io/docs/install_standalone-docker.md\n        If looking for a hosted Milvus, take a looka this documentation:\n        https://zilliz.com/cloud\n        IF USING L2/IP metric IT IS HIGHLY SUGGESTED TO NORMALIZE YOUR DATA.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}765{"id": "860bb0b94ce7-1", "text": "The connection args used for this class comes in the form of a dict,\n        here are a few of the options:\n            address (str): The actual address of Milvus\n                instance. Example address: \"localhost:19530\"\n            uri (str): The uri of Milvus instance. Example uri:\n                \"http://randomwebsite:19530\",\n                \"tcp:foobarsite:19530\",\n                \"https://ok.s3.south.com:19530\".\n            host (str): The host of Milvus instance. Default at \"localhost\",\n                PyMilvus will fill in the default host if only port is provided.\n            port (str/int): The port of Milvus instance. Default at 19530, PyMilvus\n                will fill in the default port if only host is provided.\n            user (str): Use which user to connect to Milvus instance. If user and\n                password are provided, we will add related header in every RPC call.\n            password (str): Required when user is provided. The password\n                corresponding to the user.\n            secure (bool): Default is false. If set to true, tls will be enabled.\n            client_key_path (str): If use tls two-way authentication, need to\n                write the client.key path.\n            client_pem_path (str): If use tls two-way authentication, need to\n                write the client.pem path.\n            ca_pem_path (str): If use tls two-way authentication, need to write\n                the ca.pem path.\n            server_pem_path (str): If use tls one-way authentication, need to\n                write the server.pem path.\n            server_name (str): If use tls, need to write the common name.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}766{"id": "860bb0b94ce7-2", "text": "Args:\n            embedding_function (Embeddings): Function used to embed the text.\n            collection_name (str): Which Milvus collection to use. Defaults to\n                \"LangChainCollection\".\n            connection_args (Optional[dict[str, any]]): The arguments for connection to\n                Milvus/Zilliz instance. Defaults to DEFAULT_MILVUS_CONNECTION.\n            consistency_level (str): The consistency level to use for a collection.\n                Defaults to \"Session\".\n            index_params (Optional[dict]): Which index params to use. Defaults to\n                HNSW/AUTOINDEX depending on service.\n            search_params (Optional[dict]): Which search params to use. Defaults to\n                default of index.\n            drop_old (Optional[bool]): Whether to drop the current collection. Defaults\n                to False.\n        \"\"\"\n        try:\n            from pymilvus import Collection, utility\n        except ImportError:\n            raise ValueError(\n                \"Could not import pymilvus python package. \"\n                \"Please install it with `pip install pymilvus`.\"\n            )\n        # Default search params when one is not provided.\n        self.default_search_params = {\n            \"IVF_FLAT\": {\"metric_type\": \"L2\", \"params\": {\"nprobe\": 10}},\n            \"IVF_SQ8\": {\"metric_type\": \"L2\", \"params\": {\"nprobe\": 10}},\n            \"IVF_PQ\": {\"metric_type\": \"L2\", \"params\": {\"nprobe\": 10}},\n            \"HNSW\": {\"metric_type\": \"L2\", \"params\": {\"ef\": 10}},\n            \"RHNSW_FLAT\": {\"metric_type\": \"L2\", \"params\": {\"ef\": 10}},", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}767{"id": "860bb0b94ce7-3", "text": "\"RHNSW_SQ\": {\"metric_type\": \"L2\", \"params\": {\"ef\": 10}},\n            \"RHNSW_PQ\": {\"metric_type\": \"L2\", \"params\": {\"ef\": 10}},\n            \"IVF_HNSW\": {\"metric_type\": \"L2\", \"params\": {\"nprobe\": 10, \"ef\": 10}},\n            \"ANNOY\": {\"metric_type\": \"L2\", \"params\": {\"search_k\": 10}},\n            \"AUTOINDEX\": {\"metric_type\": \"L2\", \"params\": {}},\n        }\n        self.embedding_func = embedding_function\n        self.collection_name = collection_name\n        self.index_params = index_params\n        self.search_params = search_params\n        self.consistency_level = consistency_level\n        # In order for a collection to be compatible, pk needs to be auto'id and int\n        self._primary_field = \"pk\"\n        # In order for compatiblility, the text field will need to be called \"text\"\n        self._text_field = \"text\"\n        # In order for compatbility, the vector field needs to be called \"vector\"\n        self._vector_field = \"vector\"\n        self.fields: list[str] = []\n        # Create the connection to the server\n        if connection_args is None:\n            connection_args = DEFAULT_MILVUS_CONNECTION\n        self.alias = self._create_connection_alias(connection_args)\n        self.col: Optional[Collection] = None\n        # Grab the existing colection if it exists\n        if utility.has_collection(self.collection_name, using=self.alias):\n            self.col = Collection(\n                self.collection_name,\n                using=self.alias,\n            )\n        # If need to drop old, drop it\n        if drop_old and isinstance(self.col, Collection):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}768{"id": "860bb0b94ce7-4", "text": "if drop_old and isinstance(self.col, Collection):\n            self.col.drop()\n            self.col = None\n        # Initialize the vector store\n        self._init()\n    def _create_connection_alias(self, connection_args: dict) -> str:\n        \"\"\"Create the connection to the Milvus server.\"\"\"\n        from pymilvus import MilvusException, connections\n        # Grab the connection arguments that are used for checking existing connection\n        host: str = connection_args.get(\"host\", None)\n        port: Union[str, int] = connection_args.get(\"port\", None)\n        address: str = connection_args.get(\"address\", None)\n        uri: str = connection_args.get(\"uri\", None)\n        user = connection_args.get(\"user\", None)\n        # Order of use is host/port, uri, address\n        if host is not None and port is not None:\n            given_address = str(host) + \":\" + str(port)\n        elif uri is not None:\n            given_address = uri.split(\"https://\")[1]\n        elif address is not None:\n            given_address = address\n        else:\n            given_address = None\n            logger.debug(\"Missing standard address type for reuse atttempt\")\n        # User defaults to empty string when getting connection info\n        if user is not None:\n            tmp_user = user\n        else:\n            tmp_user = \"\"\n        # If a valid address was given, then check if a connection exists\n        if given_address is not None:\n            for con in connections.list_connections():\n                addr = connections.get_connection_addr(con[0])\n                if (\n                    con[1]\n                    and (\"address\" in addr)\n                    and (addr[\"address\"] == given_address)\n                    and (\"user\" in addr)\n                    and (addr[\"user\"] == tmp_user)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}769{"id": "860bb0b94ce7-5", "text": "and (\"user\" in addr)\n                    and (addr[\"user\"] == tmp_user)\n                ):\n                    logger.debug(\"Using previous connection: %s\", con[0])\n                    return con[0]\n        # Generate a new connection if one doesnt exist\n        alias = uuid4().hex\n        try:\n            connections.connect(alias=alias, **connection_args)\n            logger.debug(\"Created new connection using: %s\", alias)\n            return alias\n        except MilvusException as e:\n            logger.error(\"Failed to create new connection using: %s\", alias)\n            raise e\n    def _init(\n        self, embeddings: Optional[list] = None, metadatas: Optional[list[dict]] = None\n    ) -> None:\n        if embeddings is not None:\n            self._create_collection(embeddings, metadatas)\n        self._extract_fields()\n        self._create_index()\n        self._create_search_params()\n        self._load()\n    def _create_collection(\n        self, embeddings: list, metadatas: Optional[list[dict]] = None\n    ) -> None:\n        from pymilvus import (\n            Collection,\n            CollectionSchema,\n            DataType,\n            FieldSchema,\n            MilvusException,\n        )\n        from pymilvus.orm.types import infer_dtype_bydata\n        # Determine embedding dim\n        dim = len(embeddings[0])\n        fields = []\n        # Determine metadata schema\n        if metadatas:\n            # Create FieldSchema for each entry in metadata.\n            for key, value in metadatas[0].items():\n                # Infer the corresponding datatype of the metadata\n                dtype = infer_dtype_bydata(value)\n                # Datatype isnt compatible\n                if dtype == DataType.UNKNOWN or dtype == DataType.NONE:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}770{"id": "860bb0b94ce7-6", "text": "# Datatype isnt compatible\n                if dtype == DataType.UNKNOWN or dtype == DataType.NONE:\n                    logger.error(\n                        \"Failure to create collection, unrecognized dtype for key: %s\",\n                        key,\n                    )\n                    raise ValueError(f\"Unrecognized datatype for {key}.\")\n                # Dataype is a string/varchar equivalent\n                elif dtype == DataType.VARCHAR:\n                    fields.append(FieldSchema(key, DataType.VARCHAR, max_length=65_535))\n                else:\n                    fields.append(FieldSchema(key, dtype))\n        # Create the text field\n        fields.append(\n            FieldSchema(self._text_field, DataType.VARCHAR, max_length=65_535)\n        )\n        # Create the primary key field\n        fields.append(\n            FieldSchema(\n                self._primary_field, DataType.INT64, is_primary=True, auto_id=True\n            )\n        )\n        # Create the vector field, supports binary or float vectors\n        fields.append(\n            FieldSchema(self._vector_field, infer_dtype_bydata(embeddings[0]), dim=dim)\n        )\n        # Create the schema for the collection\n        schema = CollectionSchema(fields)\n        # Create the collection\n        try:\n            self.col = Collection(\n                name=self.collection_name,\n                schema=schema,\n                consistency_level=self.consistency_level,\n                using=self.alias,\n            )\n        except MilvusException as e:\n            logger.error(\n                \"Failed to create collection: %s error: %s\", self.collection_name, e\n            )\n            raise e\n    def _extract_fields(self) -> None:\n        \"\"\"Grab the existing fields from the Collection\"\"\"\n        from pymilvus import Collection\n        if isinstance(self.col, Collection):\n            schema = self.col.schema\n            for x in schema.fields:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}771{"id": "860bb0b94ce7-7", "text": "schema = self.col.schema\n            for x in schema.fields:\n                self.fields.append(x.name)\n            # Since primary field is auto-id, no need to track it\n            self.fields.remove(self._primary_field)\n    def _get_index(self) -> Optional[dict[str, Any]]:\n        \"\"\"Return the vector index information if it exists\"\"\"\n        from pymilvus import Collection\n        if isinstance(self.col, Collection):\n            for x in self.col.indexes:\n                if x.field_name == self._vector_field:\n                    return x.to_dict()\n        return None\n    def _create_index(self) -> None:\n        \"\"\"Create a index on the collection\"\"\"\n        from pymilvus import Collection, MilvusException\n        if isinstance(self.col, Collection) and self._get_index() is None:\n            try:\n                # If no index params, use a default HNSW based one\n                if self.index_params is None:\n                    self.index_params = {\n                        \"metric_type\": \"L2\",\n                        \"index_type\": \"HNSW\",\n                        \"params\": {\"M\": 8, \"efConstruction\": 64},\n                    }\n                try:\n                    self.col.create_index(\n                        self._vector_field,\n                        index_params=self.index_params,\n                        using=self.alias,\n                    )\n                # If default did not work, most likely on Zilliz Cloud\n                except MilvusException:\n                    # Use AUTOINDEX based index\n                    self.index_params = {\n                        \"metric_type\": \"L2\",\n                        \"index_type\": \"AUTOINDEX\",\n                        \"params\": {},\n                    }\n                    self.col.create_index(\n                        self._vector_field,\n                        index_params=self.index_params,\n                        using=self.alias,\n                    )\n                logger.debug(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}772{"id": "860bb0b94ce7-8", "text": "using=self.alias,\n                    )\n                logger.debug(\n                    \"Successfully created an index on collection: %s\",\n                    self.collection_name,\n                )\n            except MilvusException as e:\n                logger.error(\n                    \"Failed to create an index on collection: %s\", self.collection_name\n                )\n                raise e\n    def _create_search_params(self) -> None:\n        \"\"\"Generate search params based on the current index type\"\"\"\n        from pymilvus import Collection\n        if isinstance(self.col, Collection) and self.search_params is None:\n            index = self._get_index()\n            if index is not None:\n                index_type: str = index[\"index_param\"][\"index_type\"]\n                metric_type: str = index[\"index_param\"][\"metric_type\"]\n                self.search_params = self.default_search_params[index_type]\n                self.search_params[\"metric_type\"] = metric_type\n    def _load(self) -> None:\n        \"\"\"Load the collection if available.\"\"\"\n        from pymilvus import Collection\n        if isinstance(self.col, Collection) and self._get_index() is not None:\n            self.col.load()\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        timeout: Optional[int] = None,\n        batch_size: int = 1000,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Insert text data into Milvus.\n        Inserting data when the collection has not be made yet will result\n        in creating a new Collection. The data of the first entity decides\n        the schema of the new collection, the dim is extracted from the first\n        embedding and the columns are decided by the first metadata dict.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}773{"id": "860bb0b94ce7-9", "text": "embedding and the columns are decided by the first metadata dict.\n        Metada keys will need to be present for all inserted values. At\n        the moment there is no None equivalent in Milvus.\n        Args:\n            texts (Iterable[str]): The texts to embed, it is assumed\n                that they all fit in memory.\n            metadatas (Optional[List[dict]]): Metadata dicts attached to each of\n                the texts. Defaults to None.\n            timeout (Optional[int]): Timeout for each batch insert. Defaults\n                to None.\n            batch_size (int, optional): Batch size to use for insertion.\n                Defaults to 1000.\n        Raises:\n            MilvusException: Failure to add texts\n        Returns:\n            List[str]: The resulting keys for each inserted element.\n        \"\"\"\n        from pymilvus import Collection, MilvusException\n        texts = list(texts)\n        try:\n            embeddings = self.embedding_func.embed_documents(texts)\n        except NotImplementedError:\n            embeddings = [self.embedding_func.embed_query(x) for x in texts]\n        if len(embeddings) == 0:\n            logger.debug(\"Nothing to insert, skipping.\")\n            return []\n        # If the collection hasnt been initialized yet, perform all steps to do so\n        if not isinstance(self.col, Collection):\n            self._init(embeddings, metadatas)\n        # Dict to hold all insert columns\n        insert_dict: dict[str, list] = {\n            self._text_field: texts,\n            self._vector_field: embeddings,\n        }\n        # Collect the metadata into the insert dict.\n        if metadatas is not None:\n            for d in metadatas:\n                for key, value in d.items():\n                    if key in self.fields:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}774{"id": "860bb0b94ce7-10", "text": "for key, value in d.items():\n                    if key in self.fields:\n                        insert_dict.setdefault(key, []).append(value)\n        # Total insert count\n        vectors: list = insert_dict[self._vector_field]\n        total_count = len(vectors)\n        pks: list[str] = []\n        assert isinstance(self.col, Collection)\n        for i in range(0, total_count, batch_size):\n            # Grab end index\n            end = min(i + batch_size, total_count)\n            # Convert dict to list of lists batch for insertion\n            insert_list = [insert_dict[x][i:end] for x in self.fields]\n            # Insert into the collection.\n            try:\n                res: Collection\n                res = self.col.insert(insert_list, timeout=timeout, **kwargs)\n                pks.extend(res.primary_keys)\n            except MilvusException as e:\n                logger.error(\n                    \"Failed to insert batch starting at entity: %s/%s\", i, total_count\n                )\n                raise e\n        return pks\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Perform a similarity search against the query string.\n        Args:\n            query (str): The text to search.\n            k (int, optional): How many results to return. Defaults to 4.\n            param (dict, optional): The search params for the index type.\n                Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}775{"id": "860bb0b94ce7-11", "text": "Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[Document]: Document results for search.\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        res = self.similarity_search_with_score(\n            query=query, k=k, param=param, expr=expr, timeout=timeout, **kwargs\n        )\n        return [doc for doc, _ in res]\n[docs]    def similarity_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Perform a similarity search against the query string.\n        Args:\n            embedding (List[float]): The embedding vector to search.\n            k (int, optional): How many results to return. Defaults to 4.\n            param (dict, optional): The search params for the index type.\n                Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[Document]: Document results for search.\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        res = self.similarity_search_with_score_by_vector(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}776{"id": "860bb0b94ce7-12", "text": "return []\n        res = self.similarity_search_with_score_by_vector(\n            embedding=embedding, k=k, param=param, expr=expr, timeout=timeout, **kwargs\n        )\n        return [doc for doc, _ in res]\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = 4,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Perform a search on a query string and return results with score.\n        For more information about the search parameters, take a look at the pymilvus\n        documentation found here:\n        https://milvus.io/api-reference/pymilvus/v2.2.6/Collection/search().md\n        Args:\n            query (str): The text being searched.\n            k (int, optional): The amount of results ot return. Defaults to 4.\n            param (dict): The search params for the specified index.\n                Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[float], List[Tuple[Document, any, any]]:\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        # Embed the query text.\n        embedding = self.embedding_func.embed_query(query)\n        res = self.similarity_search_with_score_by_vector(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}777{"id": "860bb0b94ce7-13", "text": "res = self.similarity_search_with_score_by_vector(\n            embedding=embedding, k=k, param=param, expr=expr, timeout=timeout, **kwargs\n        )\n        return res\n[docs]    def similarity_search_with_score_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Perform a search on a query string and return results with score.\n        For more information about the search parameters, take a look at the pymilvus\n        documentation found here:\n        https://milvus.io/api-reference/pymilvus/v2.2.6/Collection/search().md\n        Args:\n            embedding (List[float]): The embedding vector being searched.\n            k (int, optional): The amount of results ot return. Defaults to 4.\n            param (dict): The search params for the specified index.\n                Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[Tuple[Document, float]]: Result doc and score.\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        if param is None:\n            param = self.search_params\n        # Determine result metadata fields.\n        output_fields = self.fields[:]\n        output_fields.remove(self._vector_field)\n        # Perform the search.\n        res = self.col.search(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}778{"id": "860bb0b94ce7-14", "text": "# Perform the search.\n        res = self.col.search(\n            data=[embedding],\n            anns_field=self._vector_field,\n            param=param,\n            limit=k,\n            expr=expr,\n            output_fields=output_fields,\n            timeout=timeout,\n            **kwargs,\n        )\n        # Organize results.\n        ret = []\n        for result in res[0]:\n            meta = {x: result.entity.get(x) for x in output_fields}\n            doc = Document(page_content=meta.pop(self._text_field), metadata=meta)\n            pair = (doc, result.score)\n            ret.append(pair)\n        return ret\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Perform a search and return results that are reordered by MMR.\n        Args:\n            query (str): The text being searched.\n            k (int, optional): How many results to give. Defaults to 4.\n            fetch_k (int, optional): Total results to select k from.\n                Defaults to 20.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5\n            param (dict, optional): The search params for the specified index.\n                Defaults to None.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}779{"id": "860bb0b94ce7-15", "text": "Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[Document]: Document results for search.\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        embedding = self.embedding_func.embed_query(query)\n        return self.max_marginal_relevance_search_by_vector(\n            embedding=embedding,\n            k=k,\n            fetch_k=fetch_k,\n            lambda_mult=lambda_mult,\n            param=param,\n            expr=expr,\n            timeout=timeout,\n            **kwargs,\n        )\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: list[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        param: Optional[dict] = None,\n        expr: Optional[str] = None,\n        timeout: Optional[int] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Perform a search and return results that are reordered by MMR.\n        Args:\n            embedding (str): The embedding vector being searched.\n            k (int, optional): How many results to give. Defaults to 4.\n            fetch_k (int, optional): Total results to select k from.\n                Defaults to 20.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}780{"id": "860bb0b94ce7-16", "text": "to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5\n            param (dict, optional): The search params for the specified index.\n                Defaults to None.\n            expr (str, optional): Filtering expression. Defaults to None.\n            timeout (int, optional): How long to wait before timeout error.\n                Defaults to None.\n            kwargs: Collection.search() keyword arguments.\n        Returns:\n            List[Document]: Document results for search.\n        \"\"\"\n        if self.col is None:\n            logger.debug(\"No existing collection to search.\")\n            return []\n        if param is None:\n            param = self.search_params\n        # Determine result metadata fields.\n        output_fields = self.fields[:]\n        output_fields.remove(self._vector_field)\n        # Perform the search.\n        res = self.col.search(\n            data=[embedding],\n            anns_field=self._vector_field,\n            param=param,\n            limit=fetch_k,\n            expr=expr,\n            output_fields=output_fields,\n            timeout=timeout,\n            **kwargs,\n        )\n        # Organize results.\n        ids = []\n        documents = []\n        scores = []\n        for result in res[0]:\n            meta = {x: result.entity.get(x) for x in output_fields}\n            doc = Document(page_content=meta.pop(self._text_field), metadata=meta)\n            documents.append(doc)\n            scores.append(result.score)\n            ids.append(result.id)\n        vectors = self.col.query(\n            expr=f\"{self._primary_field} in {ids}\",\n            output_fields=[self._primary_field, self._vector_field],\n            timeout=timeout,\n        )\n        # Reorganize the results from query to match search order.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}781{"id": "860bb0b94ce7-17", "text": ")\n        # Reorganize the results from query to match search order.\n        vectors = {x[self._primary_field]: x[self._vector_field] for x in vectors}\n        ordered_result_embeddings = [vectors[x] for x in ids]\n        # Get the new order of results.\n        new_ordering = maximal_marginal_relevance(\n            np.array(embedding), ordered_result_embeddings, k=k, lambda_mult=lambda_mult\n        )\n        # Reorder the values and return.\n        ret = []\n        for x in new_ordering:\n            # Function can return -1 index\n            if x == -1:\n                break\n            else:\n                ret.append(documents[x])\n        return ret\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        collection_name: str = \"LangChainCollection\",\n        connection_args: dict[str, Any] = DEFAULT_MILVUS_CONNECTION,\n        consistency_level: str = \"Session\",\n        index_params: Optional[dict] = None,\n        search_params: Optional[dict] = None,\n        drop_old: bool = False,\n        **kwargs: Any,\n    ) -> Milvus:\n        \"\"\"Create a Milvus collection, indexes it with HNSW, and insert data.\n        Args:\n            texts (List[str]): Text data.\n            embedding (Embeddings): Embedding function.\n            metadatas (Optional[List[dict]]): Metadata for each text if it exists.\n                Defaults to None.\n            collection_name (str, optional): Collection name to use. Defaults to\n                \"LangChainCollection\".", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}782{"id": "860bb0b94ce7-18", "text": "\"LangChainCollection\".\n            connection_args (dict[str, Any], optional): Connection args to use. Defaults\n                to DEFAULT_MILVUS_CONNECTION.\n            consistency_level (str, optional): Which consistency level to use. Defaults\n                to \"Session\".\n            index_params (Optional[dict], optional): Which index_params to use. Defaults\n                to None.\n            search_params (Optional[dict], optional): Which search params to use.\n                Defaults to None.\n            drop_old (Optional[bool], optional): Whether to drop the collection with\n                that name if it exists. Defaults to False.\n        Returns:\n            Milvus: Milvus Vector Store\n        \"\"\"\n        vector_db = cls(\n            embedding_function=embedding,\n            collection_name=collection_name,\n            connection_args=connection_args,\n            consistency_level=consistency_level,\n            index_params=index_params,\n            search_params=search_params,\n            drop_old=drop_old,\n            **kwargs,\n        )\n        vector_db.add_texts(texts=texts, metadatas=metadatas)\n        return vector_db\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/milvus.html"}783{"id": "bc9a003352eb-0", "text": "Source code for langchain.vectorstores.chroma\n\"\"\"Wrapper around ChromaDB embeddings platform.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport uuid\nfrom typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Tuple, Type\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import xor_args\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nif TYPE_CHECKING:\n    import chromadb\n    import chromadb.config\nlogger = logging.getLogger()\nDEFAULT_K = 4  # Number of Documents to return.\ndef _results_to_docs(results: Any) -> List[Document]:\n    return [doc for doc, _ in _results_to_docs_and_scores(results)]\ndef _results_to_docs_and_scores(results: Any) -> List[Tuple[Document, float]]:\n    return [\n        # TODO: Chroma can do batch querying,\n        # we shouldn't hard code to the 1st result\n        (Document(page_content=result[0], metadata=result[1] or {}), result[2])\n        for result in zip(\n            results[\"documents\"][0],\n            results[\"metadatas\"][0],\n            results[\"distances\"][0],\n        )\n    ]\n[docs]class Chroma(VectorStore):\n    \"\"\"Wrapper around ChromaDB embeddings platform.\n    To use, you should have the ``chromadb`` python package installed.\n    Example:\n        .. code-block:: python\n                from langchain.vectorstores import Chroma\n                from langchain.embeddings.openai import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                vectorstore = Chroma(\"langchain_store\", embeddings.embed_query)\n    \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}784{"id": "bc9a003352eb-1", "text": "vectorstore = Chroma(\"langchain_store\", embeddings.embed_query)\n    \"\"\"\n    _LANGCHAIN_DEFAULT_COLLECTION_NAME = \"langchain\"\n    def __init__(\n        self,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        embedding_function: Optional[Embeddings] = None,\n        persist_directory: Optional[str] = None,\n        client_settings: Optional[chromadb.config.Settings] = None,\n        collection_metadata: Optional[Dict] = None,\n        client: Optional[chromadb.Client] = None,\n    ) -> None:\n        \"\"\"Initialize with Chroma client.\"\"\"\n        try:\n            import chromadb\n            import chromadb.config\n        except ImportError:\n            raise ValueError(\n                \"Could not import chromadb python package. \"\n                \"Please install it with `pip install chromadb`.\"\n            )\n        if client is not None:\n            self._client = client\n        else:\n            if client_settings:\n                self._client_settings = client_settings\n            else:\n                self._client_settings = chromadb.config.Settings()\n                if persist_directory is not None:\n                    self._client_settings = chromadb.config.Settings(\n                        chroma_db_impl=\"duckdb+parquet\",\n                        persist_directory=persist_directory,\n                    )\n            self._client = chromadb.Client(self._client_settings)\n        self._embedding_function = embedding_function\n        self._persist_directory = persist_directory\n        self._collection = self._client.get_or_create_collection(\n            name=collection_name,\n            embedding_function=self._embedding_function.embed_documents\n            if self._embedding_function is not None\n            else None,\n            metadata=collection_metadata,\n        )\n    @xor_args((\"query_texts\", \"query_embeddings\"))\n    def __query_collection(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}785{"id": "bc9a003352eb-2", "text": "def __query_collection(\n        self,\n        query_texts: Optional[List[str]] = None,\n        query_embeddings: Optional[List[List[float]]] = None,\n        n_results: int = 4,\n        where: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Query the chroma collection.\"\"\"\n        try:\n            import chromadb\n        except ImportError:\n            raise ValueError(\n                \"Could not import chromadb python package. \"\n                \"Please install it with `pip install chromadb`.\"\n            )\n        for i in range(n_results, 0, -1):\n            try:\n                return self._collection.query(\n                    query_texts=query_texts,\n                    query_embeddings=query_embeddings,\n                    n_results=i,\n                    where=where,\n                    **kwargs,\n                )\n            except chromadb.errors.NotEnoughElementsException:\n                logger.error(\n                    f\"Chroma collection {self._collection.name} \"\n                    f\"contains fewer than {i} elements.\"\n                )\n        raise chromadb.errors.NotEnoughElementsException(\n            f\"No documents found for Chroma collection {self._collection.name}\"\n        )\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts (Iterable[str]): Texts to add to the vectorstore.\n            metadatas (Optional[List[dict]], optional): Optional list of metadatas.\n            ids (Optional[List[str]], optional): Optional list of IDs.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}786{"id": "bc9a003352eb-3", "text": "ids (Optional[List[str]], optional): Optional list of IDs.\n        Returns:\n            List[str]: List of IDs of the added texts.\n        \"\"\"\n        # TODO: Handle the case where the user doesn't provide ids on the Collection\n        if ids is None:\n            ids = [str(uuid.uuid1()) for _ in texts]\n        embeddings = None\n        if self._embedding_function is not None:\n            embeddings = self._embedding_function.embed_documents(list(texts))\n        self._collection.add(\n            metadatas=metadatas, embeddings=embeddings, documents=texts, ids=ids\n        )\n        return ids\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = DEFAULT_K,\n        filter: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Run similarity search with Chroma.\n        Args:\n            query (str): Query text to search for.\n            k (int): Number of results to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List[Document]: List of documents most similar to the query text.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k, filter=filter)\n        return [doc for doc, _ in docs_and_scores]\n[docs]    def similarity_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = DEFAULT_K,\n        filter: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}787{"id": "bc9a003352eb-4", "text": "\"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding (str): Embedding to look up documents similar to.\n            k (int): Number of Documents to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query vector.\n        \"\"\"\n        results = self.__query_collection(\n            query_embeddings=embedding, n_results=k, where=filter\n        )\n        return _results_to_docs(results)\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = DEFAULT_K,\n        filter: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Run similarity search with Chroma with distance.\n        Args:\n            query (str): Query text to search for.\n            k (int): Number of results to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List[Tuple[Document, float]]: List of documents most similar to the query\n                text with distance in float.\n        \"\"\"\n        if self._embedding_function is None:\n            results = self.__query_collection(\n                query_texts=[query], n_results=k, where=filter\n            )\n        else:\n            query_embedding = self._embedding_function.embed_query(query)\n            results = self.__query_collection(\n                query_embeddings=[query_embedding], n_results=k, where=filter\n            )\n        return _results_to_docs_and_scores(results)\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}788{"id": "bc9a003352eb-5", "text": "[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = DEFAULT_K,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        filter: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        results = self.__query_collection(\n            query_embeddings=embedding,\n            n_results=fetch_k,\n            where=filter,\n            include=[\"metadatas\", \"documents\", \"distances\", \"embeddings\"],\n        )\n        mmr_selected = maximal_marginal_relevance(\n            np.array(embedding, dtype=np.float32),\n            results[\"embeddings\"][0],\n            k=k,\n            lambda_mult=lambda_mult,\n        )\n        candidates = _results_to_docs(results)\n        selected_results = [r for i, r in enumerate(candidates) if i in mmr_selected]\n        return selected_results", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}789{"id": "bc9a003352eb-6", "text": "return selected_results\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = DEFAULT_K,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        filter: Optional[Dict[str, str]] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        if self._embedding_function is None:\n            raise ValueError(\n                \"For MMR search, you must specify an embedding function on\" \"creation.\"\n            )\n        embedding = self._embedding_function.embed_query(query)\n        docs = self.max_marginal_relevance_search_by_vector(\n            embedding, k, fetch_k, lambda_mul=lambda_mult, filter=filter\n        )\n        return docs\n[docs]    def delete_collection(self) -> None:\n        \"\"\"Delete the collection.\"\"\"\n        self._client.delete_collection(self._collection.name)\n[docs]    def get(self, include: Optional[List[str]] = None) -> Dict[str, Any]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}790{"id": "bc9a003352eb-7", "text": "\"\"\"Gets the collection.\n        Args:\n            include (Optional[List[str]]): List of fields to include from db.\n                Defaults to None.\n        \"\"\"\n        if include is not None:\n            return self._collection.get(include=include)\n        else:\n            return self._collection.get()\n[docs]    def persist(self) -> None:\n        \"\"\"Persist the collection.\n        This can be used to explicitly persist the data to disk.\n        It will also be called automatically when the object is destroyed.\n        \"\"\"\n        if self._persist_directory is None:\n            raise ValueError(\n                \"You must specify a persist_directory on\"\n                \"creation to persist the collection.\"\n            )\n        self._client.persist()\n[docs]    def update_document(self, document_id: str, document: Document) -> None:\n        \"\"\"Update a document in the collection.\n        Args:\n            document_id (str): ID of the document to update.\n            document (Document): Document to update.\n        \"\"\"\n        text = document.page_content\n        metadata = document.metadata\n        self._collection.update_document(document_id, text, metadata)\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Chroma],\n        texts: List[str],\n        embedding: Optional[Embeddings] = None,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        persist_directory: Optional[str] = None,\n        client_settings: Optional[chromadb.config.Settings] = None,\n        client: Optional[chromadb.Client] = None,\n        **kwargs: Any,\n    ) -> Chroma:\n        \"\"\"Create a Chroma vectorstore from a raw documents.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}791{"id": "bc9a003352eb-8", "text": ") -> Chroma:\n        \"\"\"Create a Chroma vectorstore from a raw documents.\n        If a persist_directory is specified, the collection will be persisted there.\n        Otherwise, the data will be ephemeral in-memory.\n        Args:\n            texts (List[str]): List of texts to add to the collection.\n            collection_name (str): Name of the collection to create.\n            persist_directory (Optional[str]): Directory to persist the collection.\n            embedding (Optional[Embeddings]): Embedding function. Defaults to None.\n            metadatas (Optional[List[dict]]): List of metadatas. Defaults to None.\n            ids (Optional[List[str]]): List of document IDs. Defaults to None.\n            client_settings (Optional[chromadb.config.Settings]): Chroma client settings\n        Returns:\n            Chroma: Chroma vectorstore.\n        \"\"\"\n        chroma_collection = cls(\n            collection_name=collection_name,\n            embedding_function=embedding,\n            persist_directory=persist_directory,\n            client_settings=client_settings,\n            client=client,\n        )\n        chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids)\n        return chroma_collection\n[docs]    @classmethod\n    def from_documents(\n        cls: Type[Chroma],\n        documents: List[Document],\n        embedding: Optional[Embeddings] = None,\n        ids: Optional[List[str]] = None,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        persist_directory: Optional[str] = None,\n        client_settings: Optional[chromadb.config.Settings] = None,\n        client: Optional[chromadb.Client] = None,  # Add this line\n        **kwargs: Any,\n    ) -> Chroma:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}792{"id": "bc9a003352eb-9", "text": "**kwargs: Any,\n    ) -> Chroma:\n        \"\"\"Create a Chroma vectorstore from a list of documents.\n        If a persist_directory is specified, the collection will be persisted there.\n        Otherwise, the data will be ephemeral in-memory.\n        Args:\n            collection_name (str): Name of the collection to create.\n            persist_directory (Optional[str]): Directory to persist the collection.\n            ids (Optional[List[str]]): List of document IDs. Defaults to None.\n            documents (List[Document]): List of documents to add to the vectorstore.\n            embedding (Optional[Embeddings]): Embedding function. Defaults to None.\n            client_settings (Optional[chromadb.config.Settings]): Chroma client settings\n        Returns:\n            Chroma: Chroma vectorstore.\n        \"\"\"\n        texts = [doc.page_content for doc in documents]\n        metadatas = [doc.metadata for doc in documents]\n        return cls.from_texts(\n            texts=texts,\n            embedding=embedding,\n            metadatas=metadatas,\n            ids=ids,\n            collection_name=collection_name,\n            persist_directory=persist_directory,\n            client_settings=client_settings,\n            client=client,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/chroma.html"}793{"id": "3100f557aa41-0", "text": "Source code for langchain.vectorstores.deeplake\n\"\"\"Wrapper around Activeloop Deep Lake.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport uuid\nfrom functools import partial\nfrom typing import Any, Callable, Dict, Iterable, List, Optional, Sequence, Tuple\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\nlogger = logging.getLogger(__name__)\ndistance_metric_map = {\n    \"l2\": lambda a, b: np.linalg.norm(a - b, axis=1, ord=2),\n    \"l1\": lambda a, b: np.linalg.norm(a - b, axis=1, ord=1),\n    \"max\": lambda a, b: np.linalg.norm(a - b, axis=1, ord=np.inf),\n    \"cos\": lambda a, b: np.dot(a, b.T)\n    / (np.linalg.norm(a) * np.linalg.norm(b, axis=1)),\n    \"dot\": lambda a, b: np.dot(a, b.T),\n}\ndef vector_search(\n    query_embedding: np.ndarray,\n    data_vectors: np.ndarray,\n    distance_metric: str = \"L2\",\n    k: Optional[int] = 4,\n) -> Tuple[List, List]:\n    \"\"\"Naive search for nearest neighbors\n    args:\n        query_embedding: np.ndarray\n        data_vectors: np.ndarray\n        k (int): number of nearest neighbors\n        distance_metric: distance function 'L2' for Euclidean, 'L1' for Nuclear, 'Max'\n            l-infinity distnace, 'cos' for cosine similarity, 'dot' for dot product\n    returns:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}794{"id": "3100f557aa41-1", "text": "returns:\n        nearest_indices: List, indices of nearest neighbors\n    \"\"\"\n    if data_vectors.shape[0] == 0:\n        return [], []\n    # Calculate the distance between the query_vector and all data_vectors\n    distances = distance_metric_map[distance_metric](query_embedding, data_vectors)\n    nearest_indices = np.argsort(distances)\n    nearest_indices = (\n        nearest_indices[::-1][:k] if distance_metric in [\"cos\"] else nearest_indices[:k]\n    )\n    return nearest_indices.tolist(), distances[nearest_indices].tolist()\ndef dp_filter(x: dict, filter: Dict[str, str]) -> bool:\n    \"\"\"Filter helper function for Deep Lake\"\"\"\n    metadata = x[\"metadata\"].data()[\"value\"]\n    return all(k in metadata and v == metadata[k] for k, v in filter.items())\n[docs]class DeepLake(VectorStore):\n    \"\"\"Wrapper around Deep Lake, a data lake for deep learning applications.\n    We implement naive similarity search and filtering for fast prototyping,\n    but it can be extended with Tensor Query Language (TQL) for production use cases\n    over billion rows.\n    Why Deep Lake?\n    - Not only stores embeddings, but also the original data with version control.\n    - Serverless, doesn't require another service and can be used with major\n        cloud providers (S3, GCS, etc.)\n    - More than just a multi-modal vector store. You can use the dataset\n        to fine-tune your own LLM models.\n    To use, you should have the ``deeplake`` python package installed.\n    Example:\n        .. code-block:: python\n                from langchain.vectorstores import DeepLake\n                from langchain.embeddings.openai import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}795{"id": "3100f557aa41-2", "text": "embeddings = OpenAIEmbeddings()\n                vectorstore = DeepLake(\"langchain_store\", embeddings.embed_query)\n    \"\"\"\n    _LANGCHAIN_DEFAULT_DEEPLAKE_PATH = \"./deeplake/\"\n    def __init__(\n        self,\n        dataset_path: str = _LANGCHAIN_DEFAULT_DEEPLAKE_PATH,\n        token: Optional[str] = None,\n        embedding_function: Optional[Embeddings] = None,\n        read_only: Optional[bool] = False,\n        ingestion_batch_size: int = 1024,\n        num_workers: int = 0,\n        verbose: bool = True,\n        **kwargs: Any,\n    ) -> None:\n        \"\"\"Initialize with Deep Lake client.\"\"\"\n        self.ingestion_batch_size = ingestion_batch_size\n        self.num_workers = num_workers\n        self.verbose = verbose\n        try:\n            import deeplake\n            from deeplake.constants import MB\n        except ImportError:\n            raise ValueError(\n                \"Could not import deeplake python package. \"\n                \"Please install it with `pip install deeplake`.\"\n            )\n        self._deeplake = deeplake\n        self.dataset_path = dataset_path\n        creds_args = {\"creds\": kwargs[\"creds\"]} if \"creds\" in kwargs else {}\n        if deeplake.exists(dataset_path, token=token, **creds_args) and not kwargs.get(\n            \"overwrite\", False\n        ):\n            if \"overwrite\" in kwargs:\n                del kwargs[\"overwrite\"]\n            self.ds = deeplake.load(\n                dataset_path,\n                token=token,\n                read_only=read_only,\n                verbose=self.verbose,\n                **kwargs,\n            )\n            logger.info(f\"Loading deeplake {dataset_path} from storage.\")\n            if self.verbose:\n                print(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}796{"id": "3100f557aa41-3", "text": "if self.verbose:\n                print(\n                    f\"Deep Lake Dataset in {dataset_path} already exists, \"\n                    f\"loading from the storage\"\n                )\n                self.ds.summary()\n        else:\n            if \"overwrite\" in kwargs:\n                del kwargs[\"overwrite\"]\n            self.ds = deeplake.empty(\n                dataset_path,\n                token=token,\n                overwrite=True,\n                verbose=self.verbose,\n                **kwargs,\n            )\n            with self.ds:\n                self.ds.create_tensor(\n                    \"text\",\n                    htype=\"text\",\n                    create_id_tensor=False,\n                    create_sample_info_tensor=False,\n                    create_shape_tensor=False,\n                    chunk_compression=\"lz4\",\n                )\n                self.ds.create_tensor(\n                    \"metadata\",\n                    htype=\"json\",\n                    create_id_tensor=False,\n                    create_sample_info_tensor=False,\n                    create_shape_tensor=False,\n                    chunk_compression=\"lz4\",\n                )\n                self.ds.create_tensor(\n                    \"embedding\",\n                    htype=\"generic\",\n                    dtype=np.float32,\n                    create_id_tensor=False,\n                    create_sample_info_tensor=False,\n                    max_chunk_size=64 * MB,\n                    create_shape_tensor=True,\n                )\n                self.ds.create_tensor(\n                    \"ids\",\n                    htype=\"text\",\n                    create_id_tensor=False,\n                    create_sample_info_tensor=False,\n                    create_shape_tensor=False,\n                    chunk_compression=\"lz4\",\n                )\n        self._embedding_function = embedding_function\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}797{"id": "3100f557aa41-4", "text": "**kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts (Iterable[str]): Texts to add to the vectorstore.\n            metadatas (Optional[List[dict]], optional): Optional list of metadatas.\n            ids (Optional[List[str]], optional): Optional list of IDs.\n        Returns:\n            List[str]: List of IDs of the added texts.\n        \"\"\"\n        if ids is None:\n            ids = [str(uuid.uuid1()) for _ in texts]\n        text_list = list(texts)\n        if metadatas is None:\n            metadatas = [{}] * len(text_list)\n        elements = list(zip(text_list, metadatas, ids))\n        @self._deeplake.compute\n        def ingest(sample_in: list, sample_out: list) -> None:\n            text_list = [s[0] for s in sample_in]\n            embeds: Sequence[Optional[np.ndarray]] = []\n            if self._embedding_function is not None:\n                embeddings = self._embedding_function.embed_documents(text_list)\n                embeds = [np.array(e, dtype=np.float32) for e in embeddings]\n            else:\n                embeds = [None] * len(text_list)\n            for s, e in zip(sample_in, embeds):\n                sample_out.append(\n                    {\n                        \"text\": s[0],\n                        \"metadata\": s[1],\n                        \"ids\": s[2],\n                        \"embedding\": e,\n                    }\n                )\n        batch_size = min(self.ingestion_batch_size, len(elements))\n        if batch_size == 0:\n            return []\n        batched = [", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}798{"id": "3100f557aa41-5", "text": "if batch_size == 0:\n            return []\n        batched = [\n            elements[i : i + batch_size] for i in range(0, len(elements), batch_size)\n        ]\n        ingest().eval(\n            batched,\n            self.ds,\n            num_workers=min(self.num_workers, len(batched) // max(self.num_workers, 1)),\n            **kwargs,\n        )\n        self.ds.commit(allow_empty=True)\n        if self.verbose:\n            self.ds.summary()\n        return ids\n    def _search_helper(\n        self,\n        query: Any[str, None] = None,\n        embedding: Any[float, None] = None,\n        k: int = 4,\n        distance_metric: str = \"L2\",\n        use_maximal_marginal_relevance: Optional[bool] = False,\n        fetch_k: Optional[int] = 20,\n        filter: Optional[Any[Dict[str, str], Callable, str]] = None,\n        return_score: Optional[bool] = False,\n        **kwargs: Any,\n    ) -> Any[List[Document], List[Tuple[Document, float]]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            embedding: Embedding function to use. Defaults to None.\n            k: Number of Documents to return. Defaults to 4.\n            distance_metric: `L2` for Euclidean, `L1` for Nuclear,\n                `max` L-infinity distance, `cos` for cosine similarity,\n                'dot' for dot product. Defaults to `L2`.\n            filter: Attribute filter by metadata example {'key': 'value'}. It can also\n            take [Deep Lake filter]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}799{"id": "3100f557aa41-6", "text": "take [Deep Lake filter]\n            (https://docs.deeplake.ai/en/latest/deeplake.core.dataset.html#deeplake.core.dataset.Dataset.filter)\n                Defaults to None.\n            maximal_marginal_relevance: Whether to use maximal marginal relevance.\n                Defaults to False.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n                Defaults to 20.\n            return_score: Whether to return the score. Defaults to False.\n        Returns:\n            List of Documents selected by the specified distance metric,\n            if return_score True, return a tuple of (Document, score)\n        \"\"\"\n        view = self.ds\n        # attribute based filtering\n        if filter is not None:\n            if isinstance(filter, dict):\n                filter = partial(dp_filter, filter=filter)\n            view = view.filter(filter)\n            if len(view) == 0:\n                return []\n        if self._embedding_function is None:\n            view = view.filter(lambda x: query in x[\"text\"].data()[\"value\"])\n            scores = [1.0] * len(view)\n            if use_maximal_marginal_relevance:\n                raise ValueError(\n                    \"For MMR search, you must specify an embedding function on\"\n                    \"creation.\"\n                )\n        else:\n            emb = embedding or self._embedding_function.embed_query(\n                query\n            )  # type: ignore\n            query_emb = np.array(emb, dtype=np.float32)\n            embeddings = view.embedding.numpy(fetch_chunks=True)\n            k_search = fetch_k if use_maximal_marginal_relevance else k\n            indices, scores = vector_search(\n                query_emb,\n                embeddings,\n                k=k_search,\n                distance_metric=distance_metric.lower(),\n            )\n            view = view[indices]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}800{"id": "3100f557aa41-7", "text": "distance_metric=distance_metric.lower(),\n            )\n            view = view[indices]\n            if use_maximal_marginal_relevance:\n                lambda_mult = kwargs.get(\"lambda_mult\", 0.5)\n                indices = maximal_marginal_relevance(\n                    query_emb,\n                    embeddings[indices],\n                    k=min(k, len(indices)),\n                    lambda_mult=lambda_mult,\n                )\n                view = view[indices]\n                scores = [scores[i] for i in indices]\n        docs = [\n            Document(\n                page_content=el[\"text\"].data()[\"value\"],\n                metadata=el[\"metadata\"].data()[\"value\"],\n            )\n            for el in view\n        ]\n        if return_score:\n            return [(doc, score) for doc, score in zip(docs, scores)]\n        return docs\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: text to embed and run the query on.\n            k: Number of Documents to return.\n                Defaults to 4.\n            query: Text to look up documents similar to.\n            embedding: Embedding function to use.\n                Defaults to None.\n            k: Number of Documents to return.\n                Defaults to 4.\n            distance_metric: `L2` for Euclidean, `L1` for Nuclear, `max`\n                L-infinity distance, `cos` for cosine similarity, 'dot' for dot product\n                Defaults to `L2`.\n            filter: Attribute filter by metadata example {'key': 'value'}.\n                Defaults to None.\n            maximal_marginal_relevance: Whether to use maximal marginal relevance.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}801{"id": "3100f557aa41-8", "text": "maximal_marginal_relevance: Whether to use maximal marginal relevance.\n                Defaults to False.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n                Defaults to 20.\n            return_score: Whether to return the score. Defaults to False.\n        Returns:\n            List of Documents most similar to the query vector.\n        \"\"\"\n        return self._search_helper(query=query, k=k, **kwargs)\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query vector.\n        \"\"\"\n        return self._search_helper(embedding=embedding, k=k, **kwargs)\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        distance_metric: str = \"L2\",\n        k: int = 4,\n        filter: Optional[Dict[str, str]] = None,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Run similarity search with Deep Lake with distance returned.\n        Args:\n            query (str): Query text to search for.\n            distance_metric: `L2` for Euclidean, `L1` for Nuclear, `max` L-infinity\n                distance, `cos` for cosine similarity, 'dot' for dot product.\n                Defaults to `L2`.\n            k (int): Number of results to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}802{"id": "3100f557aa41-9", "text": "filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List[Tuple[Document, float]]: List of documents most similar to the query\n                text with distance in float.\n        \"\"\"\n        return self._search_helper(\n            query=query,\n            k=k,\n            filter=filter,\n            return_score=True,\n            distance_metric=distance_metric,\n        )\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        return self._search_helper(\n            embedding=embedding,\n            k=k,\n            fetch_k=fetch_k,\n            use_maximal_marginal_relevance=True,\n            lambda_mult=lambda_mult,\n            **kwargs,\n        )\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}803{"id": "3100f557aa41-10", "text": "self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        if self._embedding_function is None:\n            raise ValueError(\n                \"For MMR search, you must specify an embedding function on\" \"creation.\"\n            )\n        return self._search_helper(\n            query=query,\n            k=k,\n            fetch_k=fetch_k,\n            use_maximal_marginal_relevance=True,\n            lambda_mult=lambda_mult,\n            **kwargs,\n        )\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Optional[Embeddings] = None,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        dataset_path: str = _LANGCHAIN_DEFAULT_DEEPLAKE_PATH,\n        **kwargs: Any,\n    ) -> DeepLake:\n        \"\"\"Create a Deep Lake dataset from a raw documents.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}804{"id": "3100f557aa41-11", "text": ") -> DeepLake:\n        \"\"\"Create a Deep Lake dataset from a raw documents.\n        If a dataset_path is specified, the dataset will be persisted in that location,\n        otherwise by default at `./deeplake`\n        Args:\n            path (str, pathlib.Path): - The full path to the dataset. Can be:\n                - Deep Lake cloud path of the form ``hub://username/dataset_name``.\n                    To write to Deep Lake cloud datasets,\n                    ensure that you are logged in to Deep Lake\n                    (use 'activeloop login' from command line)\n                - AWS S3 path of the form ``s3://bucketname/path/to/dataset``.\n                    Credentials are required in either the environment\n                - Google Cloud Storage path of the form\n                    ``gcs://bucketname/path/to/dataset``Credentials are required\n                    in either the environment\n                - Local file system path of the form ``./path/to/dataset`` or\n                    ``~/path/to/dataset`` or ``path/to/dataset``.\n                - In-memory path of the form ``mem://path/to/dataset`` which doesn't\n                    save the dataset, but keeps it in memory instead.\n                    Should be used only for testing as it does not persist.\n            documents (List[Document]): List of documents to add.\n            embedding (Optional[Embeddings]): Embedding function. Defaults to None.\n            metadatas (Optional[List[dict]]): List of metadatas. Defaults to None.\n            ids (Optional[List[str]]): List of document IDs. Defaults to None.\n        Returns:\n            DeepLake: Deep Lake dataset.\n        \"\"\"\n        deeplake_dataset = cls(\n            dataset_path=dataset_path, embedding_function=embedding, **kwargs\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}805{"id": "3100f557aa41-12", "text": "dataset_path=dataset_path, embedding_function=embedding, **kwargs\n        )\n        deeplake_dataset.add_texts(texts=texts, metadatas=metadatas, ids=ids)\n        return deeplake_dataset\n[docs]    def delete(\n        self,\n        ids: Any[List[str], None] = None,\n        filter: Any[Dict[str, str], None] = None,\n        delete_all: Any[bool, None] = None,\n    ) -> bool:\n        \"\"\"Delete the entities in the dataset\n        Args:\n            ids (Optional[List[str]], optional): The document_ids to delete.\n                Defaults to None.\n            filter (Optional[Dict[str, str]], optional): The filter to delete by.\n                Defaults to None.\n            delete_all (Optional[bool], optional): Whether to drop the dataset.\n                Defaults to None.\n        \"\"\"\n        if delete_all:\n            self.ds.delete(large_ok=True)\n            return True\n        view = None\n        if ids:\n            view = self.ds.filter(lambda x: x[\"ids\"].data()[\"value\"] in ids)\n            ids = list(view.sample_indices)\n        if filter:\n            if view is None:\n                view = self.ds\n            view = view.filter(partial(dp_filter, filter=filter))\n            ids = list(view.sample_indices)\n        with self.ds:\n            for id in sorted(ids)[::-1]:\n                self.ds.pop(id)\n            self.ds.commit(f\"deleted {len(ids)} samples\", allow_empty=True)\n        return True\n[docs]    @classmethod\n    def force_delete_by_path(cls, path: str) -> None:\n        \"\"\"Force delete dataset by path\"\"\"\n        try:\n            import deeplake\n        except ImportError:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}806{"id": "3100f557aa41-13", "text": "try:\n            import deeplake\n        except ImportError:\n            raise ValueError(\n                \"Could not import deeplake python package. \"\n                \"Please install it with `pip install deeplake`.\"\n            )\n        deeplake.delete(path, large_ok=True, force=True)\n[docs]    def delete_dataset(self) -> None:\n        \"\"\"Delete the collection.\"\"\"\n        self.delete(delete_all=True)\n[docs]    def persist(self) -> None:\n        \"\"\"Persist the collection.\"\"\"\n        self.ds.flush()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/deeplake.html"}807{"id": "ef92a93029d2-0", "text": "Source code for langchain.vectorstores.weaviate\n\"\"\"Wrapper around weaviate vector database.\"\"\"\nfrom __future__ import annotations\nimport datetime\nfrom typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Type\nfrom uuid import uuid4\nimport numpy as np\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nfrom langchain.vectorstores.base import VectorStore\nfrom langchain.vectorstores.utils import maximal_marginal_relevance\ndef _default_schema(index_name: str) -> Dict:\n    return {\n        \"class\": index_name,\n        \"properties\": [\n            {\n                \"name\": \"text\",\n                \"dataType\": [\"text\"],\n            }\n        ],\n    }\ndef _create_weaviate_client(**kwargs: Any) -> Any:\n    client = kwargs.get(\"client\")\n    if client is not None:\n        return client\n    weaviate_url = get_from_dict_or_env(kwargs, \"weaviate_url\", \"WEAVIATE_URL\")\n    try:\n        # the weaviate api key param should not be mandatory\n        weaviate_api_key = get_from_dict_or_env(\n            kwargs, \"weaviate_api_key\", \"WEAVIATE_API_KEY\", None\n        )\n    except ValueError:\n        weaviate_api_key = None\n    try:\n        import weaviate\n    except ImportError:\n        raise ValueError(\n            \"Could not import weaviate python  package. \"\n            \"Please install it with `pip instal weaviate-client`\"\n        )\n    auth = (\n        weaviate.auth.AuthApiKey(api_key=weaviate_api_key)\n        if weaviate_api_key is not None\n        else None\n    )", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}808{"id": "ef92a93029d2-1", "text": "if weaviate_api_key is not None\n        else None\n    )\n    client = weaviate.Client(weaviate_url, auth_client_secret=auth)\n    return client\ndef _default_score_normalizer(val: float) -> float:\n    return 1 - 1 / (1 + np.exp(val))\ndef _json_serializable(value: Any) -> Any:\n    if isinstance(value, datetime.datetime):\n        return value.isoformat()\n    return value\n[docs]class Weaviate(VectorStore):\n    \"\"\"Wrapper around Weaviate vector database.\n    To use, you should have the ``weaviate-client`` python package installed.\n    Example:\n        .. code-block:: python\n            import weaviate\n            from langchain.vectorstores import Weaviate\n            client = weaviate.Client(url=os.environ[\"WEAVIATE_URL\"], ...)\n            weaviate = Weaviate(client, index_name, text_key)\n    \"\"\"\n    def __init__(\n        self,\n        client: Any,\n        index_name: str,\n        text_key: str,\n        embedding: Optional[Embeddings] = None,\n        attributes: Optional[List[str]] = None,\n        relevance_score_fn: Optional[\n            Callable[[float], float]\n        ] = _default_score_normalizer,\n        by_text: bool = True,\n    ):\n        \"\"\"Initialize with Weaviate client.\"\"\"\n        try:\n            import weaviate\n        except ImportError:\n            raise ValueError(\n                \"Could not import weaviate python package. \"\n                \"Please install it with `pip install weaviate-client`.\"\n            )\n        if not isinstance(client, weaviate.Client):\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}809{"id": "ef92a93029d2-2", "text": ")\n        if not isinstance(client, weaviate.Client):\n            raise ValueError(\n                f\"client should be an instance of weaviate.Client, got {type(client)}\"\n            )\n        self._client = client\n        self._index_name = index_name\n        self._embedding = embedding\n        self._text_key = text_key\n        self._query_attrs = [self._text_key]\n        self._relevance_score_fn = relevance_score_fn\n        self._by_text = by_text\n        if attributes is not None:\n            self._query_attrs.extend(attributes)\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Upload texts with metadata (properties) to Weaviate.\"\"\"\n        from weaviate.util import get_valid_uuid\n        ids = []\n        with self._client.batch as batch:\n            for i, text in enumerate(texts):\n                data_properties = {self._text_key: text}\n                if metadatas is not None:\n                    for key, val in metadatas[i].items():\n                        data_properties[key] = _json_serializable(val)\n                # If the UUID of one of the objects already exists\n                # then the existing object will be replaced by the new object.\n                _id = (\n                    kwargs[\"uuids\"][i] if \"uuids\" in kwargs else get_valid_uuid(uuid4())\n                )\n                if self._embedding is not None:\n                    vector = self._embedding.embed_documents([text])[0]\n                else:\n                    vector = None\n                batch.add_data_object(\n                    data_object=data_properties,\n                    class_name=self._index_name,\n                    uuid=_id,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}810{"id": "ef92a93029d2-3", "text": "class_name=self._index_name,\n                    uuid=_id,\n                    vector=vector,\n                )\n                ids.append(_id)\n        return ids\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        if self._by_text:\n            return self.similarity_search_by_text(query, k, **kwargs)\n        else:\n            if self._embedding is None:\n                raise ValueError(\n                    \"_embedding cannot be None for similarity_search when \"\n                    \"_by_text=False\"\n                )\n            embedding = self._embedding.embed_query(query)\n            return self.similarity_search_by_vector(embedding, k, **kwargs)\n[docs]    def similarity_search_by_text(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        content: Dict[str, Any] = {\"concepts\": [query]}\n        if kwargs.get(\"search_distance\"):\n            content[\"certainty\"] = kwargs.get(\"search_distance\")\n        query_obj = self._client.query.get(self._index_name, self._query_attrs)\n        if kwargs.get(\"where_filter\"):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}811{"id": "ef92a93029d2-4", "text": "if kwargs.get(\"where_filter\"):\n            query_obj = query_obj.with_where(kwargs.get(\"where_filter\"))\n        if kwargs.get(\"additional\"):\n            query_obj = query_obj.with_additional(kwargs.get(\"additional\"))\n        result = query_obj.with_near_text(content).with_limit(k).do()\n        if \"errors\" in result:\n            raise ValueError(f\"Error during query: {result['errors']}\")\n        docs = []\n        for res in result[\"data\"][\"Get\"][self._index_name]:\n            text = res.pop(self._text_key)\n            docs.append(Document(page_content=text, metadata=res))\n        return docs\n[docs]    def similarity_search_by_vector(\n        self, embedding: List[float], k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Look up similar documents by embedding vector in Weaviate.\"\"\"\n        vector = {\"vector\": embedding}\n        query_obj = self._client.query.get(self._index_name, self._query_attrs)\n        if kwargs.get(\"where_filter\"):\n            query_obj = query_obj.with_where(kwargs.get(\"where_filter\"))\n        if kwargs.get(\"additional\"):\n            query_obj = query_obj.with_additional(kwargs.get(\"additional\"))\n        result = query_obj.with_near_vector(vector).with_limit(k).do()\n        if \"errors\" in result:\n            raise ValueError(f\"Error during query: {result['errors']}\")\n        docs = []\n        for res in result[\"data\"][\"Get\"][self._index_name]:\n            text = res.pop(self._text_key)\n            docs.append(Document(page_content=text, metadata=res))\n        return docs\n[docs]    def max_marginal_relevance_search(\n        self,\n        query: str,\n        k: int = 4,\n        fetch_k: int = 20,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}812{"id": "ef92a93029d2-5", "text": "k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        if self._embedding is not None:\n            embedding = self._embedding.embed_query(query)\n        else:\n            raise ValueError(\n                \"max_marginal_relevance_search requires a suitable Embeddings object\"\n            )\n        return self.max_marginal_relevance_search_by_vector(\n            embedding, k=k, fetch_k=fetch_k, lambda_mult=lambda_mult, **kwargs\n        )\n[docs]    def max_marginal_relevance_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        fetch_k: int = 20,\n        lambda_mult: float = 0.5,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs selected using the maximal marginal relevance.\n        Maximal marginal relevance optimizes for similarity to query AND diversity\n        among selected documents.\n        Args:\n            embedding: Embedding to look up documents similar to.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}813{"id": "ef92a93029d2-6", "text": "Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            fetch_k: Number of Documents to fetch to pass to MMR algorithm.\n            lambda_mult: Number between 0 and 1 that determines the degree\n                        of diversity among the results with 0 corresponding\n                        to maximum diversity and 1 to minimum diversity.\n                        Defaults to 0.5.\n        Returns:\n            List of Documents selected by maximal marginal relevance.\n        \"\"\"\n        vector = {\"vector\": embedding}\n        query_obj = self._client.query.get(self._index_name, self._query_attrs)\n        if kwargs.get(\"where_filter\"):\n            query_obj = query_obj.with_where(kwargs.get(\"where_filter\"))\n        results = (\n            query_obj.with_additional(\"vector\")\n            .with_near_vector(vector)\n            .with_limit(fetch_k)\n            .do()\n        )\n        payload = results[\"data\"][\"Get\"][self._index_name]\n        embeddings = [result[\"_additional\"][\"vector\"] for result in payload]\n        mmr_selected = maximal_marginal_relevance(\n            np.array(embedding), embeddings, k=k, lambda_mult=lambda_mult\n        )\n        docs = []\n        for idx in mmr_selected:\n            text = payload[idx].pop(self._text_key)\n            payload[idx].pop(\"_additional\")\n            meta = payload[idx]\n            docs.append(Document(page_content=text, metadata=meta))\n        return docs\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        if self._embedding is None:\n            raise ValueError(\n                \"_embedding cannot be None for similarity_search_with_score\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}814{"id": "ef92a93029d2-7", "text": "raise ValueError(\n                \"_embedding cannot be None for similarity_search_with_score\"\n            )\n        content: Dict[str, Any] = {\"concepts\": [query]}\n        if kwargs.get(\"search_distance\"):\n            content[\"certainty\"] = kwargs.get(\"search_distance\")\n        query_obj = self._client.query.get(self._index_name, self._query_attrs)\n        if not self._by_text:\n            embedding = self._embedding.embed_query(query)\n            vector = {\"vector\": embedding}\n            result = (\n                query_obj.with_near_vector(vector)\n                .with_limit(k)\n                .with_additional(\"vector\")\n                .do()\n            )\n        else:\n            result = (\n                query_obj.with_near_text(content)\n                .with_limit(k)\n                .with_additional(\"vector\")\n                .do()\n            )\n        if \"errors\" in result:\n            raise ValueError(f\"Error during query: {result['errors']}\")\n        docs_and_scores = []\n        for res in result[\"data\"][\"Get\"][self._index_name]:\n            text = res.pop(self._text_key)\n            score = np.dot(\n                res[\"_additional\"][\"vector\"], self._embedding.embed_query(query)\n            )\n            docs_and_scores.append((Document(page_content=text, metadata=res), score))\n        return docs_and_scores\n    def _similarity_search_with_relevance_scores(\n        self,\n        query: str,\n        k: int = 4,\n        **kwargs: Any,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs and relevance scores, normalized on a scale from 0 to 1.\n        0 is dissimilar, 1 is most similar.\n        \"\"\"\n        if self._relevance_score_fn is None:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}815{"id": "ef92a93029d2-8", "text": "\"\"\"\n        if self._relevance_score_fn is None:\n            raise ValueError(\n                \"relevance_score_fn must be provided to\"\n                \" Weaviate constructor to normalize scores\"\n            )\n        docs_and_scores = self.similarity_search_with_score(query, k=k, **kwargs)\n        return [\n            (doc, self._relevance_score_fn(score)) for doc, score in docs_and_scores\n        ]\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Weaviate],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> Weaviate:\n        \"\"\"Construct Weaviate wrapper from raw documents.\n        This is a user-friendly interface that:\n            1. Embeds documents.\n            2. Creates a new index for the embeddings in the Weaviate instance.\n            3. Adds the documents to the newly created Weaviate index.\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain.vectorstores.weaviate import Weaviate\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                weaviate = Weaviate.from_texts(\n                    texts,\n                    embeddings,\n                    weaviate_url=\"http://localhost:8080\"\n                )\n        \"\"\"\n        client = _create_weaviate_client(**kwargs)\n        from weaviate.util import get_valid_uuid\n        index_name = kwargs.get(\"index_name\", f\"LangChain_{uuid4().hex}\")\n        embeddings = embedding.embed_documents(texts) if embedding else None\n        text_key = \"text\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}816{"id": "ef92a93029d2-9", "text": "text_key = \"text\"\n        schema = _default_schema(index_name)\n        attributes = list(metadatas[0].keys()) if metadatas else None\n        # check whether the index already exists\n        if not client.schema.contains(schema):\n            client.schema.create_class(schema)\n        with client.batch as batch:\n            for i, text in enumerate(texts):\n                data_properties = {\n                    text_key: text,\n                }\n                if metadatas is not None:\n                    for key in metadatas[i].keys():\n                        data_properties[key] = metadatas[i][key]\n                # If the UUID of one of the objects already exists\n                # then the existing objectwill be replaced by the new object.\n                if \"uuids\" in kwargs:\n                    _id = kwargs[\"uuids\"][i]\n                else:\n                    _id = get_valid_uuid(uuid4())\n                # if an embedding strategy is not provided, we let\n                # weaviate create the embedding. Note that this will only\n                # work if weaviate has been installed with a vectorizer module\n                # like text2vec-contextionary for example\n                params = {\n                    \"uuid\": _id,\n                    \"data_object\": data_properties,\n                    \"class_name\": index_name,\n                }\n                if embeddings is not None:\n                    params[\"vector\"] = embeddings[i]\n                batch.add_data_object(**params)\n            batch.flush()\n        relevance_score_fn = kwargs.get(\"relevance_score_fn\")\n        by_text: bool = kwargs.get(\"by_text\", False)\n        return cls(\n            client,\n            index_name,\n            text_key,\n            embedding=embedding,\n            attributes=attributes,\n            relevance_score_fn=relevance_score_fn,\n            by_text=by_text,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}817{"id": "ef92a93029d2-10", "text": "relevance_score_fn=relevance_score_fn,\n            by_text=by_text,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/weaviate.html"}818{"id": "b550fe3ebf45-0", "text": "Source code for langchain.vectorstores.typesense\n\"\"\"Wrapper around Typesense vector search\"\"\"\nfrom __future__ import annotations\nimport uuid\nfrom typing import TYPE_CHECKING, Any, Iterable, List, Optional, Tuple, Union\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_env\nfrom langchain.vectorstores.base import VectorStore\nif TYPE_CHECKING:\n    from typesense.client import Client\n    from typesense.collection import Collection\n[docs]class Typesense(VectorStore):\n    \"\"\"Wrapper around Typesense vector search.\n    To use, you should have the ``typesense`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.embedding.openai import OpenAIEmbeddings\n            from langchain.vectorstores import Typesense\n            import typesense\n            node = {\n                \"host\": \"localhost\",  # For Typesense Cloud use xxx.a1.typesense.net\n                \"port\": \"8108\",       # For Typesense Cloud use 443\n                \"protocol\": \"http\"    # For Typesense Cloud use https\n            }\n            typesense_client = typesense.Client(\n                {\n                  \"nodes\": [node],\n                  \"api_key\": \"<API_KEY>\",\n                  \"connection_timeout_seconds\": 2\n                }\n            )\n            typesense_collection_name = \"langchain-memory\"\n            embedding = OpenAIEmbeddings()\n            vectorstore = Typesense(\n                typesense_client,\n                typesense_collection_name,\n                embedding.embed_query,\n                \"text\",\n            )\n    \"\"\"\n    def __init__(\n        self,\n        typesense_client: Client,\n        embedding: Embeddings,\n        *,\n        typesense_collection_name: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}819{"id": "b550fe3ebf45-1", "text": "*,\n        typesense_collection_name: Optional[str] = None,\n        text_key: str = \"text\",\n    ):\n        \"\"\"Initialize with Typesense client.\"\"\"\n        try:\n            from typesense import Client\n        except ImportError:\n            raise ValueError(\n                \"Could not import typesense python package. \"\n                \"Please install it with `pip install typesense`.\"\n            )\n        if not isinstance(typesense_client, Client):\n            raise ValueError(\n                f\"typesense_client should be an instance of typesense.Client, \"\n                f\"got {type(typesense_client)}\"\n            )\n        self._typesense_client = typesense_client\n        self._embedding = embedding\n        self._typesense_collection_name = (\n            typesense_collection_name or f\"langchain-{str(uuid.uuid4())}\"\n        )\n        self._text_key = text_key\n    @property\n    def _collection(self) -> Collection:\n        return self._typesense_client.collections[self._typesense_collection_name]\n    def _prep_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]],\n        ids: Optional[List[str]],\n    ) -> List[dict]:\n        \"\"\"Embed and create the documents\"\"\"\n        _ids = ids or (str(uuid.uuid4()) for _ in texts)\n        _metadatas: Iterable[dict] = metadatas or ({} for _ in texts)\n        embedded_texts = self._embedding.embed_documents(list(texts))\n        return [\n            {\"id\": _id, \"vec\": vec, f\"{self._text_key}\": text, \"metadata\": metadata}\n            for _id, vec, text, metadata in zip(_ids, embedded_texts, texts, _metadatas)\n        ]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}820{"id": "b550fe3ebf45-2", "text": "]\n    def _create_collection(self, num_dim: int) -> None:\n        fields = [\n            {\"name\": \"vec\", \"type\": \"float[]\", \"num_dim\": num_dim},\n            {\"name\": f\"{self._text_key}\", \"type\": \"string\"},\n            {\"name\": \".*\", \"type\": \"auto\"},\n        ]\n        self._typesense_client.collections.create(\n            {\"name\": self._typesense_collection_name, \"fields\": fields}\n        )\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embedding and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            ids: Optional list of ids to associate with the texts.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        from typesense.exceptions import ObjectNotFound\n        docs = self._prep_texts(texts, metadatas, ids)\n        try:\n            self._collection.documents.import_(docs, {\"action\": \"upsert\"})\n        except ObjectNotFound:\n            # Create the collection if it doesn't already exist\n            self._create_collection(len(docs[0][\"vec\"]))\n            self._collection.documents.import_(docs, {\"action\": \"upsert\"})\n        return [doc[\"id\"] for doc in docs]\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = 4,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}821{"id": "b550fe3ebf45-3", "text": "self,\n        query: str,\n        k: int = 4,\n        filter: Optional[str] = \"\",\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return typesense documents most similar to query, along with scores.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter: typesense filter_by expression to filter documents on\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        embedded_query = [str(x) for x in self._embedding.embed_query(query)]\n        query_obj = {\n            \"q\": \"*\",\n            \"vector_query\": f'vec:([{\",\".join(embedded_query)}], k:{k})',\n            \"filter_by\": filter,\n            \"collection\": self._typesense_collection_name,\n        }\n        docs = []\n        response = self._typesense_client.multi_search.perform(\n            {\"searches\": [query_obj]}, {}\n        )\n        for hit in response[\"results\"][0][\"hits\"]:\n            document = hit[\"document\"]\n            metadata = document[\"metadata\"]\n            text = document[self._text_key]\n            score = hit[\"vector_distance\"]\n            docs.append((Document(page_content=text, metadata=metadata), score))\n        return docs\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[str] = \"\",\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return typesense documents most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}822{"id": "b550fe3ebf45-4", "text": "k: Number of Documents to return. Defaults to 4.\n            filter: typesense filter_by expression to filter documents on\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        docs_and_score = self.similarity_search_with_score(query, k=k, filter=filter)\n        return [doc for doc, _ in docs_and_score]\n[docs]    @classmethod\n    def from_client_params(\n        cls,\n        embedding: Embeddings,\n        *,\n        host: str = \"localhost\",\n        port: Union[str, int] = \"8108\",\n        protocol: str = \"http\",\n        typesense_api_key: Optional[str] = None,\n        connection_timeout_seconds: int = 2,\n        **kwargs: Any,\n    ) -> Typesense:\n        \"\"\"Initialize Typesense directly from client parameters.\n        Example:\n            .. code-block:: python\n                from langchain.embedding.openai import OpenAIEmbeddings\n                from langchain.vectorstores import Typesense\n                # Pass in typesense_api_key as kwarg or set env var \"TYPESENSE_API_KEY\".\n                vectorstore = Typesense(\n                    OpenAIEmbeddings(),\n                    host=\"localhost\",\n                    port=\"8108\",\n                    protocol=\"http\",\n                    typesense_collection_name=\"langchain-memory\",\n                )\n        \"\"\"\n        try:\n            from typesense import Client\n        except ImportError:\n            raise ValueError(\n                \"Could not import typesense python package. \"\n                \"Please install it with `pip install typesense`.\"\n            )\n        node = {\n            \"host\": host,\n            \"port\": str(port),\n            \"protocol\": protocol,\n        }\n        typesense_api_key = typesense_api_key or get_from_env(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}823{"id": "b550fe3ebf45-5", "text": "}\n        typesense_api_key = typesense_api_key or get_from_env(\n            \"typesense_api_key\", \"TYPESENSE_API_KEY\"\n        )\n        client_config = {\n            \"nodes\": [node],\n            \"api_key\": typesense_api_key,\n            \"connection_timeout_seconds\": connection_timeout_seconds,\n        }\n        return cls(Client(client_config), embedding, **kwargs)\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        typesense_client: Optional[Client] = None,\n        typesense_client_params: Optional[dict] = None,\n        typesense_collection_name: Optional[str] = None,\n        text_key: str = \"text\",\n        **kwargs: Any,\n    ) -> Typesense:\n        \"\"\"Construct Typesense wrapper from raw text.\"\"\"\n        if typesense_client:\n            vectorstore = cls(typesense_client, embedding, **kwargs)\n        elif typesense_client_params:\n            vectorstore = cls.from_client_params(\n                embedding, **typesense_client_params, **kwargs\n            )\n        else:\n            raise ValueError(\n                \"Must specify one of typesense_client or typesense_client_params.\"\n            )\n        vectorstore.add_texts(texts, metadatas=metadatas, ids=ids)\n        return vectorstore\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/typesense.html"}824{"id": "def0b5c21fa5-0", "text": "Source code for langchain.vectorstores.analyticdb\n\"\"\"VectorStore wrapper around a Postgres/PGVector database.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport uuid\nfrom typing import Any, Dict, Iterable, List, Optional, Tuple\nimport sqlalchemy\nfrom sqlalchemy import REAL, Index\nfrom sqlalchemy.dialects.postgresql import ARRAY, JSON, UUID\nfrom sqlalchemy.ext.declarative import declarative_base\nfrom sqlalchemy.orm import Session, relationship\nfrom sqlalchemy.sql.expression import func\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nfrom langchain.vectorstores.base import VectorStore\nBase = declarative_base()  # type: Any\nADA_TOKEN_COUNT = 1536\n_LANGCHAIN_DEFAULT_COLLECTION_NAME = \"langchain\"\nclass BaseModel(Base):\n    __abstract__ = True\n    uuid = sqlalchemy.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)\nclass CollectionStore(BaseModel):\n    __tablename__ = \"langchain_pg_collection\"\n    name = sqlalchemy.Column(sqlalchemy.String)\n    cmetadata = sqlalchemy.Column(JSON)\n    embeddings = relationship(\n        \"EmbeddingStore\",\n        back_populates=\"collection\",\n        passive_deletes=True,\n    )\n    @classmethod\n    def get_by_name(cls, session: Session, name: str) -> Optional[\"CollectionStore\"]:\n        return session.query(cls).filter(cls.name == name).first()  # type: ignore\n    @classmethod\n    def get_or_create(\n        cls,\n        session: Session,\n        name: str,\n        cmetadata: Optional[dict] = None,\n    ) -> Tuple[\"CollectionStore\", bool]:\n        \"\"\"\n        Get or create a collection.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}825{"id": "def0b5c21fa5-1", "text": "\"\"\"\n        Get or create a collection.\n        Returns [Collection, bool] where the bool is True if the collection was created.\n        \"\"\"\n        created = False\n        collection = cls.get_by_name(session, name)\n        if collection:\n            return collection, created\n        collection = cls(name=name, cmetadata=cmetadata)\n        session.add(collection)\n        session.commit()\n        created = True\n        return collection, created\nclass EmbeddingStore(BaseModel):\n    __tablename__ = \"langchain_pg_embedding\"\n    collection_id = sqlalchemy.Column(\n        UUID(as_uuid=True),\n        sqlalchemy.ForeignKey(\n            f\"{CollectionStore.__tablename__}.uuid\",\n            ondelete=\"CASCADE\",\n        ),\n    )\n    collection = relationship(CollectionStore, back_populates=\"embeddings\")\n    embedding: sqlalchemy.Column = sqlalchemy.Column(ARRAY(REAL))\n    document = sqlalchemy.Column(sqlalchemy.String, nullable=True)\n    cmetadata = sqlalchemy.Column(JSON, nullable=True)\n    # custom_id : any user defined id\n    custom_id = sqlalchemy.Column(sqlalchemy.String, nullable=True)\n    # The following line creates an index named 'langchain_pg_embedding_vector_idx'\n    langchain_pg_embedding_vector_idx = Index(\n        \"langchain_pg_embedding_vector_idx\",\n        embedding,\n        postgresql_using=\"ann\",\n        postgresql_with={\n            \"distancemeasure\": \"L2\",\n            \"dim\": 1536,\n            \"pq_segments\": 64,\n            \"hnsw_m\": 100,\n            \"pq_centers\": 2048,\n        },\n    )\nclass QueryResult:\n    EmbeddingStore: EmbeddingStore\n    distance: float\n[docs]class AnalyticDB(VectorStore):\n    \"\"\"\n    VectorStore implementation using AnalyticDB.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}826{"id": "def0b5c21fa5-2", "text": "\"\"\"\n    VectorStore implementation using AnalyticDB.\n    AnalyticDB is a distributed full PostgresSQL syntax cloud-native database.\n    - `connection_string` is a postgres connection string.\n    - `embedding_function` any embedding function implementing\n        `langchain.embeddings.base.Embeddings` interface.\n    - `collection_name` is the name of the collection to use. (default: langchain)\n        - NOTE: This is not the name of the table, but the name of the collection.\n            The tables will be created when initializing the store (if not exists)\n            So, make sure the user has the right permissions to create tables.\n    - `pre_delete_collection` if True, will delete the collection if it exists.\n        (default: False)\n        - Useful for testing.\n    \"\"\"\n    def __init__(\n        self,\n        connection_string: str,\n        embedding_function: Embeddings,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        collection_metadata: Optional[dict] = None,\n        pre_delete_collection: bool = False,\n        logger: Optional[logging.Logger] = None,\n    ) -> None:\n        self.connection_string = connection_string\n        self.embedding_function = embedding_function\n        self.collection_name = collection_name\n        self.collection_metadata = collection_metadata\n        self.pre_delete_collection = pre_delete_collection\n        self.logger = logger or logging.getLogger(__name__)\n        self.__post_init__()\n    def __post_init__(\n        self,\n    ) -> None:\n        \"\"\"\n        Initialize the store.\n        \"\"\"\n        self._conn = self.connect()\n        self.create_tables_if_not_exists()\n        self.create_collection()\n[docs]    def connect(self) -> sqlalchemy.engine.Connection:\n        engine = sqlalchemy.create_engine(self.connection_string)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}827{"id": "def0b5c21fa5-3", "text": "engine = sqlalchemy.create_engine(self.connection_string)\n        conn = engine.connect()\n        return conn\n[docs]    def create_tables_if_not_exists(self) -> None:\n        Base.metadata.create_all(self._conn)\n[docs]    def drop_tables(self) -> None:\n        Base.metadata.drop_all(self._conn)\n[docs]    def create_collection(self) -> None:\n        if self.pre_delete_collection:\n            self.delete_collection()\n        with Session(self._conn) as session:\n            CollectionStore.get_or_create(\n                session, self.collection_name, cmetadata=self.collection_metadata\n            )\n[docs]    def delete_collection(self) -> None:\n        self.logger.debug(\"Trying to delete collection\")\n        with Session(self._conn) as session:\n            collection = self.get_collection(session)\n            if not collection:\n                self.logger.error(\"Collection not found\")\n                return\n            session.delete(collection)\n            session.commit()\n[docs]    def get_collection(self, session: Session) -> Optional[\"CollectionStore\"]:\n        return CollectionStore.get_by_name(session, self.collection_name)\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            kwargs: vectorstore specific parameters\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        if ids is None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}828{"id": "def0b5c21fa5-4", "text": "\"\"\"\n        if ids is None:\n            ids = [str(uuid.uuid1()) for _ in texts]\n        embeddings = self.embedding_function.embed_documents(list(texts))\n        if not metadatas:\n            metadatas = [{} for _ in texts]\n        with Session(self._conn) as session:\n            collection = self.get_collection(session)\n            if not collection:\n                raise ValueError(\"Collection not found\")\n            for text, metadata, embedding, id in zip(texts, metadatas, embeddings, ids):\n                embedding_store = EmbeddingStore(\n                    embedding=embedding,\n                    document=text,\n                    cmetadata=metadata,\n                    custom_id=id,\n                )\n                collection.embeddings.append(embedding_store)\n                session.add(embedding_store)\n            session.commit()\n        return ids\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[dict] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Run similarity search with AnalyticDB with distance.\n        Args:\n            query (str): Query text to search for.\n            k (int): Number of results to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        embedding = self.embedding_function.embed_query(text=query)\n        return self.similarity_search_by_vector(\n            embedding=embedding,\n            k=k,\n            filter=filter,\n        )\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[dict] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}829{"id": "def0b5c21fa5-5", "text": "k: int = 4,\n        filter: Optional[dict] = None,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        embedding = self.embedding_function.embed_query(query)\n        docs = self.similarity_search_with_score_by_vector(\n            embedding=embedding, k=k, filter=filter\n        )\n        return docs\n[docs]    def similarity_search_with_score_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        filter: Optional[dict] = None,\n    ) -> List[Tuple[Document, float]]:\n        with Session(self._conn) as session:\n            collection = self.get_collection(session)\n            if not collection:\n                raise ValueError(\"Collection not found\")\n        filter_by = EmbeddingStore.collection_id == collection.uuid\n        if filter is not None:\n            filter_clauses = []\n            for key, value in filter.items():\n                filter_by_metadata = EmbeddingStore.cmetadata[key].astext == str(value)\n                filter_clauses.append(filter_by_metadata)\n            filter_by = sqlalchemy.and_(filter_by, *filter_clauses)\n        results: List[QueryResult] = (\n            session.query(\n                EmbeddingStore,\n                func.l2_distance(EmbeddingStore.embedding, embedding).label(\"distance\"),\n            )\n            .filter(filter_by)\n            .order_by(EmbeddingStore.embedding.op(\"<->\")(embedding))", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}830{"id": "def0b5c21fa5-6", "text": ".order_by(EmbeddingStore.embedding.op(\"<->\")(embedding))\n            .join(\n                CollectionStore,\n                EmbeddingStore.collection_id == CollectionStore.uuid,\n            )\n            .limit(k)\n            .all()\n        )\n        docs = [\n            (\n                Document(\n                    page_content=result.EmbeddingStore.document,\n                    metadata=result.EmbeddingStore.cmetadata,\n                ),\n                result.distance if self.embedding_function is not None else None,\n            )\n            for result in results\n        ]\n        return docs\n[docs]    def similarity_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        filter: Optional[dict] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to embedding vector.\n        Args:\n            embedding: Embedding to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None.\n        Returns:\n            List of Documents most similar to the query vector.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score_by_vector(\n            embedding=embedding, k=k, filter=filter\n        )\n        return [doc for doc, _ in docs_and_scores]\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        ids: Optional[List[str]] = None,\n        pre_delete_collection: bool = False,\n        **kwargs: Any,\n    ) -> AnalyticDB:\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}831{"id": "def0b5c21fa5-7", "text": "**kwargs: Any,\n    ) -> AnalyticDB:\n        \"\"\"\n        Return VectorStore initialized from texts and embeddings.\n        Postgres connection string is required\n        Either pass it as a parameter\n        or set the PGVECTOR_CONNECTION_STRING environment variable.\n        \"\"\"\n        connection_string = cls.get_connection_string(kwargs)\n        store = cls(\n            connection_string=connection_string,\n            collection_name=collection_name,\n            embedding_function=embedding,\n            pre_delete_collection=pre_delete_collection,\n        )\n        store.add_texts(texts=texts, metadatas=metadatas, ids=ids, **kwargs)\n        return store\n[docs]    @classmethod\n    def get_connection_string(cls, kwargs: Dict[str, Any]) -> str:\n        connection_string: str = get_from_dict_or_env(\n            data=kwargs,\n            key=\"connection_string\",\n            env_key=\"PGVECTOR_CONNECTION_STRING\",\n        )\n        if not connection_string:\n            raise ValueError(\n                \"Postgres connection string is required\"\n                \"Either pass it as a parameter\"\n                \"or set the PGVECTOR_CONNECTION_STRING environment variable.\"\n            )\n        return connection_string\n[docs]    @classmethod\n    def from_documents(\n        cls,\n        documents: List[Document],\n        embedding: Embeddings,\n        collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,\n        ids: Optional[List[str]] = None,\n        pre_delete_collection: bool = False,\n        **kwargs: Any,\n    ) -> AnalyticDB:\n        \"\"\"\n        Return VectorStore initialized from documents and embeddings.\n        Postgres connection string is required\n        Either pass it as a parameter\n        or set the PGVECTOR_CONNECTION_STRING environment variable.\n        \"\"\"\n        texts = [d.page_content for d in documents]", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}832{"id": "def0b5c21fa5-8", "text": "\"\"\"\n        texts = [d.page_content for d in documents]\n        metadatas = [d.metadata for d in documents]\n        connection_string = cls.get_connection_string(kwargs)\n        kwargs[\"connection_string\"] = connection_string\n        return cls.from_texts(\n            texts=texts,\n            pre_delete_collection=pre_delete_collection,\n            embedding=embedding,\n            metadatas=metadatas,\n            ids=ids,\n            collection_name=collection_name,\n            **kwargs,\n        )\n[docs]    @classmethod\n    def connection_string_from_db_params(\n        cls,\n        driver: str,\n        host: str,\n        port: int,\n        database: str,\n        user: str,\n        password: str,\n    ) -> str:\n        \"\"\"Return connection string from database parameters.\"\"\"\n        return f\"postgresql+{driver}://{user}:{password}@{host}:{port}/{database}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/analyticdb.html"}833{"id": "c18ce645c299-0", "text": "Source code for langchain.vectorstores.elastic_vector_search\n\"\"\"Wrapper around Elasticsearch vector database.\"\"\"\nfrom __future__ import annotations\nimport uuid\nfrom abc import ABC\nfrom typing import Any, Dict, Iterable, List, Optional, Tuple\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_env\nfrom langchain.vectorstores.base import VectorStore\ndef _default_text_mapping(dim: int) -> Dict:\n    return {\n        \"properties\": {\n            \"text\": {\"type\": \"text\"},\n            \"vector\": {\"type\": \"dense_vector\", \"dims\": dim},\n        }\n    }\ndef _default_script_query(query_vector: List[float], filter: Optional[dict]) -> Dict:\n    if filter:\n        ((key, value),) = filter.items()\n        filter = {\"match\": {f\"metadata.{key}.keyword\": f\"{value}\"}}\n    else:\n        filter = {\"match_all\": {}}\n    return {\n        \"script_score\": {\n            \"query\": filter,\n            \"script\": {\n                \"source\": \"cosineSimilarity(params.query_vector, 'vector') + 1.0\",\n                \"params\": {\"query_vector\": query_vector},\n            },\n        }\n    }\n# ElasticVectorSearch is a concrete implementation of the abstract base class\n# VectorStore, which defines a common interface for all vector database\n# implementations. By inheriting from the ABC class, ElasticVectorSearch can be\n# defined as an abstract base class itself, allowing the creation of subclasses with\n# their own specific implementations. If you plan to subclass ElasticVectorSearch,\n# you can inherit from it and define your own implementation of the necessary methods\n# and attributes.\n[docs]class ElasticVectorSearch(VectorStore, ABC):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}834{"id": "c18ce645c299-1", "text": "# and attributes.\n[docs]class ElasticVectorSearch(VectorStore, ABC):\n    \"\"\"Wrapper around Elasticsearch as a vector database.\n    To connect to an Elasticsearch instance that does not require\n    login credentials, pass the Elasticsearch URL and index name along with the\n    embedding object to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain import ElasticVectorSearch\n            from langchain.embeddings import OpenAIEmbeddings\n            embedding = OpenAIEmbeddings()\n            elastic_vector_search = ElasticVectorSearch(\n                elasticsearch_url=\"http://localhost:9200\",\n                index_name=\"test_index\",\n                embedding=embedding\n            )\n    To connect to an Elasticsearch instance that requires login credentials,\n    including Elastic Cloud, use the Elasticsearch URL format\n    https://username:password@es_host:9243. For example, to connect to Elastic\n    Cloud, create the Elasticsearch URL with the required authentication details and\n    pass it to the ElasticVectorSearch constructor as the named parameter\n    elasticsearch_url.\n    You can obtain your Elastic Cloud URL and login credentials by logging in to the\n    Elastic Cloud console at https://cloud.elastic.co, selecting your deployment, and\n    navigating to the \"Deployments\" page.\n    To obtain your Elastic Cloud password for the default \"elastic\" user:\n    1. Log in to the Elastic Cloud console at https://cloud.elastic.co\n    2. Go to \"Security\" > \"Users\"\n    3. Locate the \"elastic\" user and click \"Edit\"\n    4. Click \"Reset password\"\n    5. Follow the prompts to reset the password\n    The format for Elastic Cloud URLs is\n    https://username:password@cluster_id.region_id.gcp.cloud.es.io:9243.\n    Example:\n        .. code-block:: python", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}835{"id": "c18ce645c299-2", "text": "Example:\n        .. code-block:: python\n            from langchain import ElasticVectorSearch\n            from langchain.embeddings import OpenAIEmbeddings\n            embedding = OpenAIEmbeddings()\n            elastic_host = \"cluster_id.region_id.gcp.cloud.es.io\"\n            elasticsearch_url = f\"https://username:password@{elastic_host}:9243\"\n            elastic_vector_search = ElasticVectorSearch(\n                elasticsearch_url=elasticsearch_url,\n                index_name=\"test_index\",\n                embedding=embedding\n            )\n    Args:\n        elasticsearch_url (str): The URL for the Elasticsearch instance.\n        index_name (str): The name of the Elasticsearch index for the embeddings.\n        embedding (Embeddings): An object that provides the ability to embed text.\n                It should be an instance of a class that subclasses the Embeddings\n                abstract base class, such as OpenAIEmbeddings()\n    Raises:\n        ValueError: If the elasticsearch python package is not installed.\n    \"\"\"\n    def __init__(\n        self,\n        elasticsearch_url: str,\n        index_name: str,\n        embedding: Embeddings,\n        *,\n        ssl_verify: Optional[Dict[str, Any]] = None,\n    ):\n        \"\"\"Initialize with necessary components.\"\"\"\n        try:\n            import elasticsearch\n        except ImportError:\n            raise ImportError(\n                \"Could not import elasticsearch python package. \"\n                \"Please install it with `pip install elasticsearch`.\"\n            )\n        self.embedding = embedding\n        self.index_name = index_name\n        _ssl_verify = ssl_verify or {}\n        try:\n            self.client = elasticsearch.Elasticsearch(elasticsearch_url, **_ssl_verify)\n        except ValueError as e:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}836{"id": "c18ce645c299-3", "text": "except ValueError as e:\n            raise ValueError(\n                f\"Your elasticsearch client string is mis-formatted. Got error: {e} \"\n            )\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        refresh_indices: bool = True,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            refresh_indices: bool to refresh ElasticSearch indices\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        try:\n            from elasticsearch.exceptions import NotFoundError\n            from elasticsearch.helpers import bulk\n        except ImportError:\n            raise ImportError(\n                \"Could not import elasticsearch python package. \"\n                \"Please install it with `pip install elasticsearch`.\"\n            )\n        requests = []\n        ids = []\n        embeddings = self.embedding.embed_documents(list(texts))\n        dim = len(embeddings[0])\n        mapping = _default_text_mapping(dim)\n        # check to see if the index already exists\n        try:\n            self.client.indices.get(index=self.index_name)\n        except NotFoundError:\n            # TODO would be nice to create index before embedding,\n            # just to save expensive steps for last\n            self.client.indices.create(index=self.index_name, mappings=mapping)\n        for i, text in enumerate(texts):\n            metadata = metadatas[i] if metadatas else {}\n            _id = str(uuid.uuid4())\n            request = {\n                \"_op_type\": \"index\",", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}837{"id": "c18ce645c299-4", "text": "request = {\n                \"_op_type\": \"index\",\n                \"_index\": self.index_name,\n                \"vector\": embeddings[i],\n                \"text\": text,\n                \"metadata\": metadata,\n                \"_id\": _id,\n            }\n            ids.append(_id)\n            requests.append(request)\n        bulk(self.client, requests)\n        if refresh_indices:\n            self.client.indices.refresh(index=self.index_name)\n        return ids\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, filter: Optional[dict] = None, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(query, k, filter=filter)\n        documents = [d[0] for d in docs_and_scores]\n        return documents\n[docs]    def similarity_search_with_score(\n        self, query: str, k: int = 4, filter: Optional[dict] = None, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return docs most similar to query.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n        Returns:\n            List of Documents most similar to the query.\n        \"\"\"\n        embedding = self.embedding.embed_query(query)\n        script_query = _default_script_query(embedding, filter)\n        response = self.client.search(index=self.index_name, query=script_query, size=k)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}838{"id": "c18ce645c299-5", "text": "response = self.client.search(index=self.index_name, query=script_query, size=k)\n        hits = [hit for hit in response[\"hits\"][\"hits\"]]\n        docs_and_scores = [\n            (\n                Document(\n                    page_content=hit[\"_source\"][\"text\"],\n                    metadata=hit[\"_source\"][\"metadata\"],\n                ),\n                hit[\"_score\"],\n            )\n            for hit in hits\n        ]\n        return docs_and_scores\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        elasticsearch_url: Optional[str] = None,\n        index_name: Optional[str] = None,\n        refresh_indices: bool = True,\n        **kwargs: Any,\n    ) -> ElasticVectorSearch:\n        \"\"\"Construct ElasticVectorSearch wrapper from raw documents.\n        This is a user-friendly interface that:\n            1. Embeds documents.\n            2. Creates a new index for the embeddings in the Elasticsearch instance.\n            3. Adds the documents to the newly created Elasticsearch index.\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import ElasticVectorSearch\n                from langchain.embeddings import OpenAIEmbeddings\n                embeddings = OpenAIEmbeddings()\n                elastic_vector_search = ElasticVectorSearch.from_texts(\n                    texts,\n                    embeddings,\n                    elasticsearch_url=\"http://localhost:9200\"\n                )\n        \"\"\"\n        elasticsearch_url = elasticsearch_url or get_from_env(\n            \"elasticsearch_url\", \"ELASTICSEARCH_URL\"\n        )\n        index_name = index_name or uuid.uuid4().hex", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}839{"id": "c18ce645c299-6", "text": ")\n        index_name = index_name or uuid.uuid4().hex\n        vectorsearch = cls(elasticsearch_url, index_name, embedding, **kwargs)\n        vectorsearch.add_texts(\n            texts, metadatas=metadatas, refresh_indices=refresh_indices\n        )\n        return vectorsearch\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/elastic_vector_search.html"}840{"id": "b0bc89b0289b-0", "text": "Source code for langchain.vectorstores.pinecone\n\"\"\"Wrapper around Pinecone vector database.\"\"\"\nfrom __future__ import annotations\nimport logging\nimport uuid\nfrom typing import Any, Callable, Iterable, List, Optional, Tuple\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nlogger = logging.getLogger(__name__)\n[docs]class Pinecone(VectorStore):\n    \"\"\"Wrapper around Pinecone vector database.\n    To use, you should have the ``pinecone-client`` python package installed.\n    Example:\n        .. code-block:: python\n            from langchain.vectorstores import Pinecone\n            from langchain.embeddings.openai import OpenAIEmbeddings\n            import pinecone\n            # The environment should be the one specified next to the API key\n            # in your Pinecone console\n            pinecone.init(api_key=\"***\", environment=\"...\")\n            index = pinecone.Index(\"langchain-demo\")\n            embeddings = OpenAIEmbeddings()\n            vectorstore = Pinecone(index, embeddings.embed_query, \"text\")\n    \"\"\"\n    def __init__(\n        self,\n        index: Any,\n        embedding_function: Callable,\n        text_key: str,\n        namespace: Optional[str] = None,\n    ):\n        \"\"\"Initialize with Pinecone client.\"\"\"\n        try:\n            import pinecone\n        except ImportError:\n            raise ValueError(\n                \"Could not import pinecone python package. \"\n                \"Please install it with `pip install pinecone-client`.\"\n            )\n        if not isinstance(index, pinecone.index.Index):\n            raise ValueError(\n                f\"client should be an instance of pinecone.index.Index, \"\n                f\"got {type(index)}\"\n            )\n        self._index = index", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}841{"id": "b0bc89b0289b-1", "text": "f\"got {type(index)}\"\n            )\n        self._index = index\n        self._embedding_function = embedding_function\n        self._text_key = text_key\n        self._namespace = namespace\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        namespace: Optional[str] = None,\n        batch_size: int = 32,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            metadatas: Optional list of metadatas associated with the texts.\n            ids: Optional list of ids to associate with the texts.\n            namespace: Optional pinecone namespace to add the texts to.\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        if namespace is None:\n            namespace = self._namespace\n        # Embed and create the documents\n        docs = []\n        ids = ids or [str(uuid.uuid4()) for _ in texts]\n        for i, text in enumerate(texts):\n            embedding = self._embedding_function(text)\n            metadata = metadatas[i] if metadatas else {}\n            metadata[self._text_key] = text\n            docs.append((ids[i], embedding, metadata))\n        # upsert to Pinecone\n        self._index.upsert(vectors=docs, namespace=namespace, batch_size=batch_size)\n        return ids\n[docs]    def similarity_search_with_score(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[dict] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}842{"id": "b0bc89b0289b-2", "text": "k: int = 4,\n        filter: Optional[dict] = None,\n        namespace: Optional[str] = None,\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Return pinecone documents most similar to query, along with scores.\n        Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter: Dictionary of argument(s) to filter on metadata\n            namespace: Namespace to search in. Default will search in '' namespace.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        if namespace is None:\n            namespace = self._namespace\n        query_obj = self._embedding_function(query)\n        docs = []\n        results = self._index.query(\n            [query_obj],\n            top_k=k,\n            include_metadata=True,\n            namespace=namespace,\n            filter=filter,\n        )\n        for res in results[\"matches\"]:\n            metadata = res[\"metadata\"]\n            if self._text_key in metadata:\n                text = metadata.pop(self._text_key)\n                score = res[\"score\"]\n                docs.append((Document(page_content=text, metadata=metadata), score))\n            else:\n                logger.warning(\n                    f\"Found document with no `{self._text_key}` key. Skipping.\"\n                )\n        return docs\n[docs]    def similarity_search(\n        self,\n        query: str,\n        k: int = 4,\n        filter: Optional[dict] = None,\n        namespace: Optional[str] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Return pinecone documents most similar to query.\n        Args:\n            query: Text to look up documents similar to.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}843{"id": "b0bc89b0289b-3", "text": "Args:\n            query: Text to look up documents similar to.\n            k: Number of Documents to return. Defaults to 4.\n            filter: Dictionary of argument(s) to filter on metadata\n            namespace: Namespace to search in. Default will search in '' namespace.\n        Returns:\n            List of Documents most similar to the query and score for each\n        \"\"\"\n        docs_and_scores = self.similarity_search_with_score(\n            query, k=k, filter=filter, namespace=namespace, **kwargs\n        )\n        return [doc for doc, _ in docs_and_scores]\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        ids: Optional[List[str]] = None,\n        batch_size: int = 32,\n        text_key: str = \"text\",\n        index_name: Optional[str] = None,\n        namespace: Optional[str] = None,\n        **kwargs: Any,\n    ) -> Pinecone:\n        \"\"\"Construct Pinecone wrapper from raw documents.\n        This is a user friendly interface that:\n            1. Embeds documents.\n            2. Adds the documents to a provided Pinecone index\n        This is intended to be a quick way to get started.\n        Example:\n            .. code-block:: python\n                from langchain import Pinecone\n                from langchain.embeddings import OpenAIEmbeddings\n                import pinecone\n                # The environment should be the one specified next to the API key\n                # in your Pinecone console\n                pinecone.init(api_key=\"***\", environment=\"...\")\n                embeddings = OpenAIEmbeddings()\n                pinecone = Pinecone.from_texts(\n                    texts,\n                    embeddings,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}844{"id": "b0bc89b0289b-4", "text": "pinecone = Pinecone.from_texts(\n                    texts,\n                    embeddings,\n                    index_name=\"langchain-demo\"\n                )\n        \"\"\"\n        try:\n            import pinecone\n        except ImportError:\n            raise ValueError(\n                \"Could not import pinecone python package. \"\n                \"Please install it with `pip install pinecone-client`.\"\n            )\n        indexes = pinecone.list_indexes()  # checks if provided index exists\n        if index_name in indexes:\n            index = pinecone.Index(index_name)\n        elif len(indexes) == 0:\n            raise ValueError(\n                \"No active indexes found in your Pinecone project, \"\n                \"are you sure you're using the right API key and environment?\"\n            )\n        else:\n            raise ValueError(\n                f\"Index '{index_name}' not found in your Pinecone project. \"\n                f\"Did you mean one of the following indexes: {', '.join(indexes)}\"\n            )\n        for i in range(0, len(texts), batch_size):\n            # set end position of batch\n            i_end = min(i + batch_size, len(texts))\n            # get batch of texts and ids\n            lines_batch = texts[i:i_end]\n            # create ids if not provided\n            if ids:\n                ids_batch = ids[i:i_end]\n            else:\n                ids_batch = [str(uuid.uuid4()) for n in range(i, i_end)]\n            # create embeddings\n            embeds = embedding.embed_documents(lines_batch)\n            # prep metadata and upsert batch\n            if metadatas:\n                metadata = metadatas[i:i_end]\n            else:\n                metadata = [{} for _ in range(i, i_end)]\n            for j, line in enumerate(lines_batch):", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}845{"id": "b0bc89b0289b-5", "text": "for j, line in enumerate(lines_batch):\n                metadata[j][text_key] = line\n            to_upsert = zip(ids_batch, embeds, metadata)\n            # upsert to Pinecone\n            index.upsert(vectors=list(to_upsert), namespace=namespace)\n        return cls(index, embedding.embed_query, text_key, namespace)\n[docs]    @classmethod\n    def from_existing_index(\n        cls,\n        index_name: str,\n        embedding: Embeddings,\n        text_key: str = \"text\",\n        namespace: Optional[str] = None,\n    ) -> Pinecone:\n        \"\"\"Load pinecone vectorstore from index name.\"\"\"\n        try:\n            import pinecone\n        except ImportError:\n            raise ValueError(\n                \"Could not import pinecone python package. \"\n                \"Please install it with `pip install pinecone-client`.\"\n            )\n        return cls(\n            pinecone.Index(index_name), embedding.embed_query, text_key, namespace\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/pinecone.html"}846{"id": "82c350cade8f-0", "text": "Source code for langchain.vectorstores.myscale\n\"\"\"Wrapper around MyScale vector database.\"\"\"\nfrom __future__ import annotations\nimport json\nimport logging\nfrom hashlib import sha1\nfrom threading import Thread\nfrom typing import Any, Dict, Iterable, List, Optional, Tuple\nfrom pydantic import BaseSettings\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.base import VectorStore\nlogger = logging.getLogger()\ndef has_mul_sub_str(s: str, *args: Any) -> bool:\n    for a in args:\n        if a not in s:\n            return False\n    return True\n[docs]class MyScaleSettings(BaseSettings):\n    \"\"\"MyScale Client Configuration\n    Attribute:\n        myscale_host (str) : An URL to connect to MyScale backend.\n                             Defaults to 'localhost'.\n        myscale_port (int) : URL port to connect with HTTP. Defaults to 8443.\n        username (str) : Usernamed to login. Defaults to None.\n        password (str) : Password to login. Defaults to None.\n        index_type (str): index type string.\n        index_param (dict): index build parameter.\n        database (str) : Database name to find the table. Defaults to 'default'.\n        table (str) : Table name to operate on.\n                      Defaults to 'vector_table'.\n        metric (str) : Metric to compute distance,\n                       supported are ('l2', 'cosine', 'ip'). Defaults to 'cosine'.\n        column_map (Dict) : Column type map to project column name onto langchain\n                            semantics. Must have keys: `text`, `id`, `vector`,\n                            must be same size to number of columns. For example:\n                            .. code-block:: python\n                            {", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}847{"id": "82c350cade8f-1", "text": ".. code-block:: python\n                            {\n                                'id': 'text_id',\n                                'vector': 'text_embedding',\n                                'text': 'text_plain',\n                                'metadata': 'metadata_dictionary_in_json',\n                            }\n                            Defaults to identity map.\n    \"\"\"\n    host: str = \"localhost\"\n    port: int = 8443\n    username: Optional[str] = None\n    password: Optional[str] = None\n    index_type: str = \"IVFFLAT\"\n    index_param: Optional[Dict[str, str]] = None\n    column_map: Dict[str, str] = {\n        \"id\": \"id\",\n        \"text\": \"text\",\n        \"vector\": \"vector\",\n        \"metadata\": \"metadata\",\n    }\n    database: str = \"default\"\n    table: str = \"langchain\"\n    metric: str = \"cosine\"\n    def __getitem__(self, item: str) -> Any:\n        return getattr(self, item)\n    class Config:\n        env_file = \".env\"\n        env_prefix = \"myscale_\"\n        env_file_encoding = \"utf-8\"\n[docs]class MyScale(VectorStore):\n    \"\"\"Wrapper around MyScale vector database\n    You need a `clickhouse-connect` python package, and a valid account\n    to connect to MyScale.\n    MyScale can not only search with simple vector indexes,\n    it also supports complex query with multiple conditions,\n    constraints and even sub-queries.\n    For more information, please visit\n        [myscale official site](https://docs.myscale.com/en/overview/)\n    \"\"\"\n    def __init__(\n        self,\n        embedding: Embeddings,\n        config: Optional[MyScaleSettings] = None,\n        **kwargs: Any,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}848{"id": "82c350cade8f-2", "text": "config: Optional[MyScaleSettings] = None,\n        **kwargs: Any,\n    ) -> None:\n        \"\"\"MyScale Wrapper to LangChain\n        embedding_function (Embeddings):\n        config (MyScaleSettings): Configuration to MyScale Client\n        Other keyword arguments will pass into\n            [clickhouse-connect](https://docs.myscale.com/)\n        \"\"\"\n        try:\n            from clickhouse_connect import get_client\n        except ImportError:\n            raise ValueError(\n                \"Could not import clickhouse connect python package. \"\n                \"Please install it with `pip install clickhouse-connect`.\"\n            )\n        try:\n            from tqdm import tqdm\n            self.pgbar = tqdm\n        except ImportError:\n            # Just in case if tqdm is not installed\n            self.pgbar = lambda x: x\n        super().__init__()\n        if config is not None:\n            self.config = config\n        else:\n            self.config = MyScaleSettings()\n        assert self.config\n        assert self.config.host and self.config.port\n        assert (\n            self.config.column_map\n            and self.config.database\n            and self.config.table\n            and self.config.metric\n        )\n        for k in [\"id\", \"vector\", \"text\", \"metadata\"]:\n            assert k in self.config.column_map\n        assert self.config.metric in [\"ip\", \"cosine\", \"l2\"]\n        # initialize the schema\n        dim = len(embedding.embed_query(\"try this out\"))\n        index_params = (\n            \", \" + \",\".join([f\"'{k}={v}'\" for k, v in self.config.index_param.items()])\n            if self.config.index_param\n            else \"\"\n        )\n        schema_ = f\"\"\"\n            CREATE TABLE IF NOT EXISTS {self.config.database}.{self.config.table}(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}849{"id": "82c350cade8f-3", "text": "CREATE TABLE IF NOT EXISTS {self.config.database}.{self.config.table}(\n                {self.config.column_map['id']} String,\n                {self.config.column_map['text']} String,\n                {self.config.column_map['vector']} Array(Float32),\n                {self.config.column_map['metadata']} JSON,\n                CONSTRAINT cons_vec_len CHECK length(\\\n                    {self.config.column_map['vector']}) = {dim},\n                VECTOR INDEX vidx {self.config.column_map['vector']} \\\n                    TYPE {self.config.index_type}(\\\n                        'metric_type={self.config.metric}'{index_params})\n            ) ENGINE = MergeTree ORDER BY {self.config.column_map['id']}\n        \"\"\"\n        self.dim = dim\n        self.BS = \"\\\\\"\n        self.must_escape = (\"\\\\\", \"'\")\n        self.embedding_function = embedding.embed_query\n        self.dist_order = \"ASC\" if self.config.metric in [\"cosine\", \"l2\"] else \"DESC\"\n        # Create a connection to myscale\n        self.client = get_client(\n            host=self.config.host,\n            port=self.config.port,\n            username=self.config.username,\n            password=self.config.password,\n            **kwargs,\n        )\n        self.client.command(\"SET allow_experimental_object_type=1\")\n        self.client.command(schema_)\n[docs]    def escape_str(self, value: str) -> str:\n        return \"\".join(f\"{self.BS}{c}\" if c in self.must_escape else c for c in value)\n    def _build_istr(self, transac: Iterable, column_names: Iterable[str]) -> str:\n        ks = \",\".join(column_names)\n        _data = []\n        for n in transac:\n            n = \",\".join([f\"'{self.escape_str(str(_n))}'\" for _n in n])", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}850{"id": "82c350cade8f-4", "text": "_data.append(f\"({n})\")\n        i_str = f\"\"\"\n                INSERT INTO TABLE \n                    {self.config.database}.{self.config.table}({ks})\n                VALUES\n                {','.join(_data)}\n                \"\"\"\n        return i_str\n    def _insert(self, transac: Iterable, column_names: Iterable[str]) -> None:\n        _i_str = self._build_istr(transac, column_names)\n        self.client.command(_i_str)\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        batch_size: int = 32,\n        ids: Optional[Iterable[str]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Run more texts through the embeddings and add to the vectorstore.\n        Args:\n            texts: Iterable of strings to add to the vectorstore.\n            ids: Optional list of ids to associate with the texts.\n            batch_size: Batch size of insertion\n            metadata: Optional column data to be inserted\n        Returns:\n            List of ids from adding the texts into the vectorstore.\n        \"\"\"\n        # Embed and create the documents\n        ids = ids or [sha1(t.encode(\"utf-8\")).hexdigest() for t in texts]\n        colmap_ = self.config.column_map\n        transac = []\n        column_names = {\n            colmap_[\"id\"]: ids,\n            colmap_[\"text\"]: texts,\n            colmap_[\"vector\"]: map(self.embedding_function, texts),\n        }\n        metadatas = metadatas or [{} for _ in texts]\n        column_names[colmap_[\"metadata\"]] = map(json.dumps, metadatas)", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}851{"id": "82c350cade8f-5", "text": "column_names[colmap_[\"metadata\"]] = map(json.dumps, metadatas)\n        assert len(set(colmap_) - set(column_names)) >= 0\n        keys, values = zip(*column_names.items())\n        try:\n            t = None\n            for v in self.pgbar(\n                zip(*values), desc=\"Inserting data...\", total=len(metadatas)\n            ):\n                assert len(v[keys.index(self.config.column_map[\"vector\"])]) == self.dim\n                transac.append(v)\n                if len(transac) == batch_size:\n                    if t:\n                        t.join()\n                    t = Thread(target=self._insert, args=[transac, keys])\n                    t.start()\n                    transac = []\n            if len(transac) > 0:\n                if t:\n                    t.join()\n                self._insert(transac, keys)\n            return [i for i in ids]\n        except Exception as e:\n            logger.error(f\"\\033[91m\\033[1m{type(e)}\\033[0m \\033[95m{str(e)}\\033[0m\")\n            return []\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[Dict[Any, Any]]] = None,\n        config: Optional[MyScaleSettings] = None,\n        text_ids: Optional[Iterable[str]] = None,\n        batch_size: int = 32,\n        **kwargs: Any,\n    ) -> MyScale:\n        \"\"\"Create Myscale wrapper with existing texts\n        Args:\n            embedding_function (Embeddings): Function to extract text embedding\n            texts (Iterable[str]): List or tuple of strings to be added", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}852{"id": "82c350cade8f-6", "text": "texts (Iterable[str]): List or tuple of strings to be added\n            config (MyScaleSettings, Optional): Myscale configuration\n            text_ids (Optional[Iterable], optional): IDs for the texts.\n                                                     Defaults to None.\n            batch_size (int, optional): Batchsize when transmitting data to MyScale.\n                                        Defaults to 32.\n            metadata (List[dict], optional): metadata to texts. Defaults to None.\n            Other keyword arguments will pass into\n                [clickhouse-connect](https://clickhouse.com/docs/en/integrations/python#clickhouse-connect-driver-api)\n        Returns:\n            MyScale Index\n        \"\"\"\n        ctx = cls(embedding, config, **kwargs)\n        ctx.add_texts(texts, ids=text_ids, batch_size=batch_size, metadatas=metadatas)\n        return ctx\n    def __repr__(self) -> str:\n        \"\"\"Text representation for myscale, prints backends, username and schemas.\n            Easy to use with `str(Myscale())`\n        Returns:\n            repr: string to show connection info and data schema\n        \"\"\"\n        _repr = f\"\\033[92m\\033[1m{self.config.database}.{self.config.table} @ \"\n        _repr += f\"{self.config.host}:{self.config.port}\\033[0m\\n\\n\"\n        _repr += f\"\\033[1musername: {self.config.username}\\033[0m\\n\\nTable Schema:\\n\"\n        _repr += \"-\" * 51 + \"\\n\"\n        for r in self.client.query(\n            f\"DESC {self.config.database}.{self.config.table}\"\n        ).named_results():\n            _repr += (", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}853{"id": "82c350cade8f-7", "text": ").named_results():\n            _repr += (\n                f\"|\\033[94m{r['name']:24s}\\033[0m|\\033[96m{r['type']:24s}\\033[0m|\\n\"\n            )\n        _repr += \"-\" * 51 + \"\\n\"\n        return _repr\n    def _build_qstr(\n        self, q_emb: List[float], topk: int, where_str: Optional[str] = None\n    ) -> str:\n        q_emb_str = \",\".join(map(str, q_emb))\n        if where_str:\n            where_str = f\"PREWHERE {where_str}\"\n        else:\n            where_str = \"\"\n        q_str = f\"\"\"\n            SELECT {self.config.column_map['text']}, \n                {self.config.column_map['metadata']}, dist\n            FROM {self.config.database}.{self.config.table}\n            {where_str}\n            ORDER BY distance({self.config.column_map['vector']}, [{q_emb_str}]) \n                AS dist {self.dist_order}\n            LIMIT {topk}\n            \"\"\"\n        return q_str\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, where_str: Optional[str] = None, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"Perform a similarity search with MyScale\n        Args:\n            query (str): query string\n            k (int, optional): Top K neighbors to retrieve. Defaults to 4.\n            where_str (Optional[str], optional): where condition string.\n                                                 Defaults to None.\n            NOTE: Please do not let end-user to fill this and always be aware\n                  of SQL injection. When dealing with metadatas, remember to", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}854{"id": "82c350cade8f-8", "text": "of SQL injection. When dealing with metadatas, remember to\n                  use `{self.metadata_column}.attribute` instead of `attribute`\n                  alone. The default name for it is `metadata`.\n        Returns:\n            List[Document]: List of Documents\n        \"\"\"\n        return self.similarity_search_by_vector(\n            self.embedding_function(query), k, where_str, **kwargs\n        )\n[docs]    def similarity_search_by_vector(\n        self,\n        embedding: List[float],\n        k: int = 4,\n        where_str: Optional[str] = None,\n        **kwargs: Any,\n    ) -> List[Document]:\n        \"\"\"Perform a similarity search with MyScale by vectors\n        Args:\n            query (str): query string\n            k (int, optional): Top K neighbors to retrieve. Defaults to 4.\n            where_str (Optional[str], optional): where condition string.\n                                                 Defaults to None.\n            NOTE: Please do not let end-user to fill this and always be aware\n                  of SQL injection. When dealing with metadatas, remember to\n                  use `{self.metadata_column}.attribute` instead of `attribute`\n                  alone. The default name for it is `metadata`.\n        Returns:\n            List[Document]: List of (Document, similarity)\n        \"\"\"\n        q_str = self._build_qstr(embedding, k, where_str)\n        try:\n            return [\n                Document(\n                    page_content=r[self.config.column_map[\"text\"]],\n                    metadata=r[self.config.column_map[\"metadata\"]],\n                )\n                for r in self.client.query(q_str).named_results()\n            ]\n        except Exception as e:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}855{"id": "82c350cade8f-9", "text": "]\n        except Exception as e:\n            logger.error(f\"\\033[91m\\033[1m{type(e)}\\033[0m \\033[95m{str(e)}\\033[0m\")\n            return []\n[docs]    def similarity_search_with_relevance_scores(\n        self, query: str, k: int = 4, where_str: Optional[str] = None, **kwargs: Any\n    ) -> List[Tuple[Document, float]]:\n        \"\"\"Perform a similarity search with MyScale\n        Args:\n            query (str): query string\n            k (int, optional): Top K neighbors to retrieve. Defaults to 4.\n            where_str (Optional[str], optional): where condition string.\n                                                 Defaults to None.\n            NOTE: Please do not let end-user to fill this and always be aware\n                  of SQL injection. When dealing with metadatas, remember to\n                  use `{self.metadata_column}.attribute` instead of `attribute`\n                  alone. The default name for it is `metadata`.\n        Returns:\n            List[Document]: List of documents\n        \"\"\"\n        q_str = self._build_qstr(self.embedding_function(query), k, where_str)\n        try:\n            return [\n                (\n                    Document(\n                        page_content=r[self.config.column_map[\"text\"]],\n                        metadata=r[self.config.column_map[\"metadata\"]],\n                    ),\n                    r[\"dist\"],\n                )\n                for r in self.client.query(q_str).named_results()\n            ]\n        except Exception as e:\n            logger.error(f\"\\033[91m\\033[1m{type(e)}\\033[0m \\033[95m{str(e)}\\033[0m\")\n            return []\n[docs]    def drop(self) -> None:\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}856{"id": "82c350cade8f-10", "text": "return []\n[docs]    def drop(self) -> None:\n        \"\"\"\n        Helper function: Drop data\n        \"\"\"\n        self.client.command(\n            f\"DROP TABLE IF EXISTS {self.config.database}.{self.config.table}\"\n        )\n    @property\n    def metadata_column(self) -> str:\n        return self.config.column_map[\"metadata\"]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/myscale.html"}857{"id": "0539bf37ccae-0", "text": "Source code for langchain.vectorstores.tair\n\"\"\"Wrapper around Tair Vector.\"\"\"\nfrom __future__ import annotations\nimport json\nimport logging\nimport uuid\nfrom typing import Any, Iterable, List, Optional, Type\nfrom langchain.docstore.document import Document\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.utils import get_from_dict_or_env\nfrom langchain.vectorstores.base import VectorStore\nlogger = logging.getLogger(__name__)\ndef _uuid_key() -> str:\n    return uuid.uuid4().hex\n[docs]class Tair(VectorStore):\n    def __init__(\n        self,\n        embedding_function: Embeddings,\n        url: str,\n        index_name: str,\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        search_params: Optional[dict] = None,\n        **kwargs: Any,\n    ):\n        self.embedding_function = embedding_function\n        self.index_name = index_name\n        try:\n            from tair import Tair as TairClient\n        except ImportError:\n            raise ValueError(\n                \"Could not import tair python package. \"\n                \"Please install it with `pip install tair`.\"\n            )\n        try:\n            # connect to tair from url\n            client = TairClient.from_url(url, **kwargs)\n        except ValueError as e:\n            raise ValueError(f\"Tair failed to connect: {e}\")\n        self.client = client\n        self.content_key = content_key\n        self.metadata_key = metadata_key\n        self.search_params = search_params\n[docs]    def create_index_if_not_exist(\n        self,\n        dim: int,\n        distance_type: str,\n        index_type: str,\n        data_type: str,\n        **kwargs: Any,\n    ) -> bool:", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}858{"id": "0539bf37ccae-1", "text": "data_type: str,\n        **kwargs: Any,\n    ) -> bool:\n        index = self.client.tvs_get_index(self.index_name)\n        if index is not None:\n            logger.info(\"Index already exists\")\n            return False\n        self.client.tvs_create_index(\n            self.index_name,\n            dim,\n            distance_type,\n            index_type,\n            data_type,\n            **kwargs,\n        )\n        return True\n[docs]    def add_texts(\n        self,\n        texts: Iterable[str],\n        metadatas: Optional[List[dict]] = None,\n        **kwargs: Any,\n    ) -> List[str]:\n        \"\"\"Add texts data to an existing index.\"\"\"\n        ids = []\n        keys = kwargs.get(\"keys\", None)\n        # Write data to tair\n        pipeline = self.client.pipeline(transaction=False)\n        embeddings = self.embedding_function.embed_documents(list(texts))\n        for i, text in enumerate(texts):\n            # Use provided key otherwise use default key\n            key = keys[i] if keys else _uuid_key()\n            metadata = metadatas[i] if metadatas else {}\n            pipeline.tvs_hset(\n                self.index_name,\n                key,\n                embeddings[i],\n                False,\n                **{\n                    self.content_key: text,\n                    self.metadata_key: json.dumps(metadata),\n                },\n            )\n            ids.append(key)\n        pipeline.execute()\n        return ids\n[docs]    def similarity_search(\n        self, query: str, k: int = 4, **kwargs: Any\n    ) -> List[Document]:\n        \"\"\"\n        Returns the most similar indexed documents to the query text.\n        Args:\n            query (str): The query text for which to find similar documents.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}859{"id": "0539bf37ccae-2", "text": "Args:\n            query (str): The query text for which to find similar documents.\n            k (int): The number of documents to return. Default is 4.\n        Returns:\n            List[Document]: A list of documents that are most similar to the query text.\n        \"\"\"\n        # Creates embedding vector from user query\n        embedding = self.embedding_function.embed_query(query)\n        keys_and_scores = self.client.tvs_knnsearch(\n            self.index_name, k, embedding, False, None, **kwargs\n        )\n        pipeline = self.client.pipeline(transaction=False)\n        for key, _ in keys_and_scores:\n            pipeline.tvs_hmget(\n                self.index_name, key, self.metadata_key, self.content_key\n            )\n        docs = pipeline.execute()\n        return [\n            Document(\n                page_content=d[1],\n                metadata=json.loads(d[0]),\n            )\n            for d in docs\n        ]\n[docs]    @classmethod\n    def from_texts(\n        cls: Type[Tair],\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        index_name: str = \"langchain\",\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        **kwargs: Any,\n    ) -> Tair:\n        try:\n            from tair import tairvector\n        except ImportError:\n            raise ValueError(\n                \"Could not import tair python package. \"\n                \"Please install it with `pip install tair`.\"\n            )\n        url = get_from_dict_or_env(kwargs, \"tair_url\", \"TAIR_URL\")\n        if \"tair_url\" in kwargs:\n            kwargs.pop(\"tair_url\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}860{"id": "0539bf37ccae-3", "text": "if \"tair_url\" in kwargs:\n            kwargs.pop(\"tair_url\")\n        distance_type = tairvector.DistanceMetric.InnerProduct\n        if \"distance_type\" in kwargs:\n            distance_type = kwargs.pop(\"distance_typ\")\n        index_type = tairvector.IndexType.HNSW\n        if \"index_type\" in kwargs:\n            index_type = kwargs.pop(\"index_type\")\n        data_type = tairvector.DataType.Float32\n        if \"data_type\" in kwargs:\n            data_type = kwargs.pop(\"data_type\")\n        index_params = {}\n        if \"index_params\" in kwargs:\n            index_params = kwargs.pop(\"index_params\")\n        search_params = {}\n        if \"search_params\" in kwargs:\n            search_params = kwargs.pop(\"search_params\")\n        keys = None\n        if \"keys\" in kwargs:\n            keys = kwargs.pop(\"keys\")\n        try:\n            tair_vector_store = cls(\n                embedding,\n                url,\n                index_name,\n                content_key=content_key,\n                metadata_key=metadata_key,\n                search_params=search_params,\n                **kwargs,\n            )\n        except ValueError as e:\n            raise ValueError(f\"tair failed to connect: {e}\")\n        # Create embeddings for documents\n        embeddings = embedding.embed_documents(texts)\n        tair_vector_store.create_index_if_not_exist(\n            len(embeddings[0]),\n            distance_type,\n            index_type,\n            data_type,\n            **index_params,\n        )\n        tair_vector_store.add_texts(texts, metadatas, keys=keys)\n        return tair_vector_store\n[docs]    @classmethod\n    def from_documents(\n        cls,\n        documents: List[Document],\n        embedding: Embeddings,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}861{"id": "0539bf37ccae-4", "text": "cls,\n        documents: List[Document],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        index_name: str = \"langchain\",\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        **kwargs: Any,\n    ) -> Tair:\n        texts = [d.page_content for d in documents]\n        metadatas = [d.metadata for d in documents]\n        return cls.from_texts(\n            texts, embedding, metadatas, index_name, content_key, metadata_key, **kwargs\n        )\n[docs]    @staticmethod\n    def drop_index(\n        index_name: str = \"langchain\",\n        **kwargs: Any,\n    ) -> bool:\n        \"\"\"\n        Drop an existing index.\n        Args:\n            index_name (str): Name of the index to drop.\n        Returns:\n            bool: True if the index is dropped successfully.\n        \"\"\"\n        try:\n            from tair import Tair as TairClient\n        except ImportError:\n            raise ValueError(\n                \"Could not import tair python package. \"\n                \"Please install it with `pip install tair`.\"\n            )\n        url = get_from_dict_or_env(kwargs, \"tair_url\", \"TAIR_URL\")\n        try:\n            if \"tair_url\" in kwargs:\n                kwargs.pop(\"tair_url\")\n            client = TairClient.from_url(url=url, **kwargs)\n        except ValueError as e:\n            raise ValueError(f\"Tair connection error: {e}\")\n        # delete index\n        ret = client.tvs_del_index(index_name)\n        if ret == 0:\n            # index not exist\n            logger.info(\"Index does not exist\")\n            return False", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}862{"id": "0539bf37ccae-5", "text": "# index not exist\n            logger.info(\"Index does not exist\")\n            return False\n        return True\n[docs]    @classmethod\n    def from_existing_index(\n        cls,\n        embedding: Embeddings,\n        index_name: str = \"langchain\",\n        content_key: str = \"content\",\n        metadata_key: str = \"metadata\",\n        **kwargs: Any,\n    ) -> Tair:\n        \"\"\"Connect to an existing Tair index.\"\"\"\n        url = get_from_dict_or_env(kwargs, \"tair_url\", \"TAIR_URL\")\n        search_params = {}\n        if \"search_params\" in kwargs:\n            search_params = kwargs.pop(\"search_params\")\n        return cls(\n            embedding,\n            url,\n            index_name,\n            content_key=content_key,\n            metadata_key=metadata_key,\n            search_params=search_params,\n            **kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/tair.html"}863{"id": "020713c03d4a-0", "text": "Source code for langchain.vectorstores.docarray.hnsw\n\"\"\"Wrapper around Hnswlib store.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, List, Literal, Optional\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.docarray.base import (\n    DocArrayIndex,\n    _check_docarray_import,\n)\n[docs]class DocArrayHnswSearch(DocArrayIndex):\n    \"\"\"Wrapper around HnswLib storage.\n    To use it, you should have the ``docarray`` package with version >=0.32.0 installed.\n    You can install it with `pip install \"langchain[docarray]\"`.\n    \"\"\"\n[docs]    @classmethod\n    def from_params(\n        cls,\n        embedding: Embeddings,\n        work_dir: str,\n        n_dim: int,\n        dist_metric: Literal[\"cosine\", \"ip\", \"l2\"] = \"cosine\",\n        max_elements: int = 1024,\n        index: bool = True,\n        ef_construction: int = 200,\n        ef: int = 10,\n        M: int = 16,\n        allow_replace_deleted: bool = True,\n        num_threads: int = 1,\n        **kwargs: Any,\n    ) -> DocArrayHnswSearch:\n        \"\"\"Initialize DocArrayHnswSearch store.\n        Args:\n            embedding (Embeddings): Embedding function.\n            work_dir (str): path to the location where all the data will be stored.\n            n_dim (int): dimension of an embedding.\n            dist_metric (str): Distance metric for DocArrayHnswSearch can be one of:\n                \"cosine\", \"ip\", and \"l2\". Defaults to \"cosine\".", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/docarray/hnsw.html"}864{"id": "020713c03d4a-1", "text": "\"cosine\", \"ip\", and \"l2\". Defaults to \"cosine\".\n            max_elements (int): Maximum number of vectors that can be stored.\n                Defaults to 1024.\n            index (bool): Whether an index should be built for this field.\n                Defaults to True.\n            ef_construction (int): defines a construction time/accuracy trade-off.\n                Defaults to 200.\n            ef (int): parameter controlling query time/accuracy trade-off.\n                Defaults to 10.\n            M (int): parameter that defines the maximum number of outgoing\n                connections in the graph. Defaults to 16.\n            allow_replace_deleted (bool): Enables replacing of deleted elements\n                with new added ones. Defaults to True.\n            num_threads (int): Sets the number of cpu threads to use. Defaults to 1.\n            **kwargs: Other keyword arguments to be passed to the get_doc_cls method.\n        \"\"\"\n        _check_docarray_import()\n        from docarray.index import HnswDocumentIndex\n        doc_cls = cls._get_doc_cls(\n            dim=n_dim,\n            space=dist_metric,\n            max_elements=max_elements,\n            index=index,\n            ef_construction=ef_construction,\n            ef=ef,\n            M=M,\n            allow_replace_deleted=allow_replace_deleted,\n            num_threads=num_threads,\n            **kwargs,\n        )\n        doc_index = HnswDocumentIndex[doc_cls](work_dir=work_dir)  # type: ignore\n        return cls(doc_index, embedding)\n[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[dict]] = None,\n        work_dir: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/docarray/hnsw.html"}865{"id": "020713c03d4a-2", "text": "work_dir: Optional[str] = None,\n        n_dim: Optional[int] = None,\n        **kwargs: Any,\n    ) -> DocArrayHnswSearch:\n        \"\"\"Create an DocArrayHnswSearch store and insert data.\n        Args:\n            texts (List[str]): Text data.\n            embedding (Embeddings): Embedding function.\n            metadatas (Optional[List[dict]]): Metadata for each text if it exists.\n                Defaults to None.\n            work_dir (str): path to the location where all the data will be stored.\n            n_dim (int): dimension of an embedding.\n            **kwargs: Other keyword arguments to be passed to the __init__ method.\n        Returns:\n            DocArrayHnswSearch Vector Store\n        \"\"\"\n        if work_dir is None:\n            raise ValueError(\"`work_dir` parameter has not been set.\")\n        if n_dim is None:\n            raise ValueError(\"`n_dim` parameter has not been set.\")\n        store = cls.from_params(embedding, work_dir, n_dim, **kwargs)\n        store.add_texts(texts=texts, metadatas=metadatas)\n        return store\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/docarray/hnsw.html"}866{"id": "b4f401baf7cb-0", "text": "Source code for langchain.vectorstores.docarray.in_memory\n\"\"\"Wrapper around in-memory storage.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, Dict, List, Literal, Optional\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.vectorstores.docarray.base import (\n    DocArrayIndex,\n    _check_docarray_import,\n)\n[docs]class DocArrayInMemorySearch(DocArrayIndex):\n    \"\"\"Wrapper around in-memory storage for exact search.\n    To use it, you should have the ``docarray`` package with version >=0.32.0 installed.\n    You can install it with `pip install \"langchain[docarray]\"`.\n    \"\"\"\n[docs]    @classmethod\n    def from_params(\n        cls,\n        embedding: Embeddings,\n        metric: Literal[\n            \"cosine_sim\", \"euclidian_dist\", \"sgeuclidean_dist\"\n        ] = \"cosine_sim\",\n        **kwargs: Any,\n    ) -> DocArrayInMemorySearch:\n        \"\"\"Initialize DocArrayInMemorySearch store.\n        Args:\n            embedding (Embeddings): Embedding function.\n            metric (str): metric for exact nearest-neighbor search.\n                Can be one of: \"cosine_sim\", \"euclidean_dist\" and \"sqeuclidean_dist\".\n                Defaults to \"cosine_sim\".\n            **kwargs: Other keyword arguments to be passed to the get_doc_cls method.\n        \"\"\"\n        _check_docarray_import()\n        from docarray.index import InMemoryExactNNIndex\n        doc_cls = cls._get_doc_cls(space=metric, **kwargs)\n        doc_index = InMemoryExactNNIndex[doc_cls]()  # type: ignore\n        return cls(doc_index, embedding)\n[docs]    @classmethod\n    def from_texts(", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/docarray/in_memory.html"}867{"id": "b4f401baf7cb-1", "text": "[docs]    @classmethod\n    def from_texts(\n        cls,\n        texts: List[str],\n        embedding: Embeddings,\n        metadatas: Optional[List[Dict[Any, Any]]] = None,\n        **kwargs: Any,\n    ) -> DocArrayInMemorySearch:\n        \"\"\"Create an DocArrayInMemorySearch store and insert data.\n        Args:\n            texts (List[str]): Text data.\n            embedding (Embeddings): Embedding function.\n            metadatas (Optional[List[Dict[Any, Any]]]): Metadata for each text\n                if it exists. Defaults to None.\n            metric (str): metric for exact nearest-neighbor search.\n                Can be one of: \"cosine_sim\", \"euclidean_dist\" and \"sqeuclidean_dist\".\n                Defaults to \"cosine_sim\".\n        Returns:\n            DocArrayInMemorySearch Vector Store\n        \"\"\"\n        store = cls.from_params(embedding, **kwargs)\n        store.add_texts(texts=texts, metadatas=metadatas)\n        return store\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/vectorstores/docarray/in_memory.html"}868{"id": "4f87dd3da754-0", "text": "Source code for langchain.output_parsers.regex\nfrom __future__ import annotations\nimport re\nfrom typing import Dict, List, Optional\nfrom langchain.schema import BaseOutputParser\n[docs]class RegexParser(BaseOutputParser):\n    \"\"\"Class to parse the output into a dictionary.\"\"\"\n    regex: str\n    output_keys: List[str]\n    default_output_key: Optional[str] = None\n    @property\n    def _type(self) -> str:\n        \"\"\"Return the type key.\"\"\"\n        return \"regex_parser\"\n[docs]    def parse(self, text: str) -> Dict[str, str]:\n        \"\"\"Parse the output of an LLM call.\"\"\"\n        match = re.search(self.regex, text)\n        if match:\n            return {key: match.group(i + 1) for i, key in enumerate(self.output_keys)}\n        else:\n            if self.default_output_key is None:\n                raise ValueError(f\"Could not parse output: {text}\")\n            else:\n                return {\n                    key: text if key == self.default_output_key else \"\"\n                    for key in self.output_keys\n                }\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/regex.html"}869{"id": "888cc2e251b5-0", "text": "Source code for langchain.output_parsers.retry\nfrom __future__ import annotations\nfrom typing import TypeVar\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.llm import LLMChain\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.prompts.prompt import PromptTemplate\nfrom langchain.schema import (\n    BaseOutputParser,\n    OutputParserException,\n    PromptValue,\n)\nNAIVE_COMPLETION_RETRY = \"\"\"Prompt:\n{prompt}\nCompletion:\n{completion}\nAbove, the Completion did not satisfy the constraints given in the Prompt.\nPlease try again:\"\"\"\nNAIVE_COMPLETION_RETRY_WITH_ERROR = \"\"\"Prompt:\n{prompt}\nCompletion:\n{completion}\nAbove, the Completion did not satisfy the constraints given in the Prompt.\nDetails: {error}\nPlease try again:\"\"\"\nNAIVE_RETRY_PROMPT = PromptTemplate.from_template(NAIVE_COMPLETION_RETRY)\nNAIVE_RETRY_WITH_ERROR_PROMPT = PromptTemplate.from_template(\n    NAIVE_COMPLETION_RETRY_WITH_ERROR\n)\nT = TypeVar(\"T\")\n[docs]class RetryOutputParser(BaseOutputParser[T]):\n    \"\"\"Wraps a parser and tries to fix parsing errors.\n    Does this by passing the original prompt and the completion to another\n    LLM, and telling it the completion did not satisfy criteria in the prompt.\n    \"\"\"\n    parser: BaseOutputParser[T]\n    retry_chain: LLMChain\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        parser: BaseOutputParser[T],\n        prompt: BasePromptTemplate = NAIVE_RETRY_PROMPT,\n    ) -> RetryOutputParser[T]:\n        chain = LLMChain(llm=llm, prompt=prompt)", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/retry.html"}870{"id": "888cc2e251b5-1", "text": "chain = LLMChain(llm=llm, prompt=prompt)\n        return cls(parser=parser, retry_chain=chain)\n[docs]    def parse_with_prompt(self, completion: str, prompt_value: PromptValue) -> T:\n        try:\n            parsed_completion = self.parser.parse(completion)\n        except OutputParserException:\n            new_completion = self.retry_chain.run(\n                prompt=prompt_value.to_string(), completion=completion\n            )\n            parsed_completion = self.parser.parse(new_completion)\n        return parsed_completion\n[docs]    def parse(self, completion: str) -> T:\n        raise NotImplementedError(\n            \"This OutputParser can only be called by the `parse_with_prompt` method.\"\n        )\n[docs]    def get_format_instructions(self) -> str:\n        return self.parser.get_format_instructions()\n    @property\n    def _type(self) -> str:\n        return \"retry\"\n[docs]class RetryWithErrorOutputParser(BaseOutputParser[T]):\n    \"\"\"Wraps a parser and tries to fix parsing errors.\n    Does this by passing the original prompt, the completion, AND the error\n    that was raised to another language model and telling it that the completion\n    did not work, and raised the given error. Differs from RetryOutputParser\n    in that this implementation provides the error that was raised back to the\n    LLM, which in theory should give it more information on how to fix it.\n    \"\"\"\n    parser: BaseOutputParser[T]\n    retry_chain: LLMChain\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        parser: BaseOutputParser[T],\n        prompt: BasePromptTemplate = NAIVE_RETRY_WITH_ERROR_PROMPT,\n    ) -> RetryWithErrorOutputParser[T]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/retry.html"}871{"id": "888cc2e251b5-2", "text": ") -> RetryWithErrorOutputParser[T]:\n        chain = LLMChain(llm=llm, prompt=prompt)\n        return cls(parser=parser, retry_chain=chain)\n[docs]    def parse_with_prompt(self, completion: str, prompt_value: PromptValue) -> T:\n        try:\n            parsed_completion = self.parser.parse(completion)\n        except OutputParserException as e:\n            new_completion = self.retry_chain.run(\n                prompt=prompt_value.to_string(), completion=completion, error=repr(e)\n            )\n            parsed_completion = self.parser.parse(new_completion)\n        return parsed_completion\n[docs]    def parse(self, completion: str) -> T:\n        raise NotImplementedError(\n            \"This OutputParser can only be called by the `parse_with_prompt` method.\"\n        )\n[docs]    def get_format_instructions(self) -> str:\n        return self.parser.get_format_instructions()\n    @property\n    def _type(self) -> str:\n        return \"retry_with_error\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/retry.html"}872{"id": "b60878f36d50-0", "text": "Source code for langchain.output_parsers.rail_parser\nfrom __future__ import annotations\nfrom typing import Any, Dict\nfrom langchain.schema import BaseOutputParser\n[docs]class GuardrailsOutputParser(BaseOutputParser):\n    guard: Any\n    @property\n    def _type(self) -> str:\n        return \"guardrails\"\n[docs]    @classmethod\n    def from_rail(cls, rail_file: str, num_reasks: int = 1) -> GuardrailsOutputParser:\n        try:\n            from guardrails import Guard\n        except ImportError:\n            raise ValueError(\n                \"guardrails-ai package not installed. \"\n                \"Install it by running `pip install guardrails-ai`.\"\n            )\n        return cls(guard=Guard.from_rail(rail_file, num_reasks=num_reasks))\n[docs]    @classmethod\n    def from_rail_string(\n        cls, rail_str: str, num_reasks: int = 1\n    ) -> GuardrailsOutputParser:\n        try:\n            from guardrails import Guard\n        except ImportError:\n            raise ValueError(\n                \"guardrails-ai package not installed. \"\n                \"Install it by running `pip install guardrails-ai`.\"\n            )\n        return cls(guard=Guard.from_rail_string(rail_str, num_reasks=num_reasks))\n[docs]    def get_format_instructions(self) -> str:\n        return self.guard.raw_prompt.format_instructions\n[docs]    def parse(self, text: str) -> Dict:\n        return self.guard.parse(text)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/rail_parser.html"}873{"id": "08722766fa01-0", "text": "Source code for langchain.output_parsers.list\nfrom __future__ import annotations\nfrom abc import abstractmethod\nfrom typing import List\nfrom langchain.schema import BaseOutputParser\n[docs]class ListOutputParser(BaseOutputParser):\n    \"\"\"Class to parse the output of an LLM call to a list.\"\"\"\n    @property\n    def _type(self) -> str:\n        return \"list\"\n[docs]    @abstractmethod\n    def parse(self, text: str) -> List[str]:\n        \"\"\"Parse the output of an LLM call.\"\"\"\n[docs]class CommaSeparatedListOutputParser(ListOutputParser):\n    \"\"\"Parse out comma separated lists.\"\"\"\n[docs]    def get_format_instructions(self) -> str:\n        return (\n            \"Your response should be a list of comma separated values, \"\n            \"eg: `foo, bar, baz`\"\n        )\n[docs]    def parse(self, text: str) -> List[str]:\n        \"\"\"Parse the output of an LLM call.\"\"\"\n        return text.strip().split(\", \")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/list.html"}874{"id": "97b8489e14dc-0", "text": "Source code for langchain.output_parsers.structured\nfrom __future__ import annotations\nfrom typing import Any, List\nfrom pydantic import BaseModel\nfrom langchain.output_parsers.format_instructions import STRUCTURED_FORMAT_INSTRUCTIONS\nfrom langchain.output_parsers.json import parse_and_check_json_markdown\nfrom langchain.schema import BaseOutputParser\nline_template = '\\t\"{name}\": {type}  // {description}'\n[docs]class ResponseSchema(BaseModel):\n    name: str\n    description: str\ndef _get_sub_string(schema: ResponseSchema) -> str:\n    return line_template.format(\n        name=schema.name, description=schema.description, type=\"string\"\n    )\n[docs]class StructuredOutputParser(BaseOutputParser):\n    response_schemas: List[ResponseSchema]\n[docs]    @classmethod\n    def from_response_schemas(\n        cls, response_schemas: List[ResponseSchema]\n    ) -> StructuredOutputParser:\n        return cls(response_schemas=response_schemas)\n[docs]    def get_format_instructions(self) -> str:\n        schema_str = \"\\n\".join(\n            [_get_sub_string(schema) for schema in self.response_schemas]\n        )\n        return STRUCTURED_FORMAT_INSTRUCTIONS.format(format=schema_str)\n[docs]    def parse(self, text: str) -> Any:\n        expected_keys = [rs.name for rs in self.response_schemas]\n        return parse_and_check_json_markdown(text, expected_keys)\n    @property\n    def _type(self) -> str:\n        return \"structured\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/structured.html"}875{"id": "115fa238fc29-0", "text": "Source code for langchain.output_parsers.fix\nfrom __future__ import annotations\nfrom typing import TypeVar\nfrom langchain.base_language import BaseLanguageModel\nfrom langchain.chains.llm import LLMChain\nfrom langchain.output_parsers.prompts import NAIVE_FIX_PROMPT\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.schema import BaseOutputParser, OutputParserException\nT = TypeVar(\"T\")\n[docs]class OutputFixingParser(BaseOutputParser[T]):\n    \"\"\"Wraps a parser and tries to fix parsing errors.\"\"\"\n    parser: BaseOutputParser[T]\n    retry_chain: LLMChain\n[docs]    @classmethod\n    def from_llm(\n        cls,\n        llm: BaseLanguageModel,\n        parser: BaseOutputParser[T],\n        prompt: BasePromptTemplate = NAIVE_FIX_PROMPT,\n    ) -> OutputFixingParser[T]:\n        chain = LLMChain(llm=llm, prompt=prompt)\n        return cls(parser=parser, retry_chain=chain)\n[docs]    def parse(self, completion: str) -> T:\n        try:\n            parsed_completion = self.parser.parse(completion)\n        except OutputParserException as e:\n            new_completion = self.retry_chain.run(\n                instructions=self.parser.get_format_instructions(),\n                completion=completion,\n                error=repr(e),\n            )\n            parsed_completion = self.parser.parse(new_completion)\n        return parsed_completion\n[docs]    def get_format_instructions(self) -> str:\n        return self.parser.get_format_instructions()\n    @property\n    def _type(self) -> str:\n        return \"output_fixing\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/fix.html"}876{"id": "d3c528a86b2b-0", "text": "Source code for langchain.output_parsers.pydantic\nimport json\nimport re\nfrom typing import Type, TypeVar\nfrom pydantic import BaseModel, ValidationError\nfrom langchain.output_parsers.format_instructions import PYDANTIC_FORMAT_INSTRUCTIONS\nfrom langchain.schema import BaseOutputParser, OutputParserException\nT = TypeVar(\"T\", bound=BaseModel)\n[docs]class PydanticOutputParser(BaseOutputParser[T]):\n    pydantic_object: Type[T]\n[docs]    def parse(self, text: str) -> T:\n        try:\n            # Greedy search for 1st json candidate.\n            match = re.search(\n                r\"\\{.*\\}\", text.strip(), re.MULTILINE | re.IGNORECASE | re.DOTALL\n            )\n            json_str = \"\"\n            if match:\n                json_str = match.group()\n            json_object = json.loads(json_str, strict=False)\n            return self.pydantic_object.parse_obj(json_object)\n        except (json.JSONDecodeError, ValidationError) as e:\n            name = self.pydantic_object.__name__\n            msg = f\"Failed to parse {name} from completion {text}. Got: {e}\"\n            raise OutputParserException(msg)\n[docs]    def get_format_instructions(self) -> str:\n        schema = self.pydantic_object.schema()\n        # Remove extraneous fields.\n        reduced_schema = schema\n        if \"title\" in reduced_schema:\n            del reduced_schema[\"title\"]\n        if \"type\" in reduced_schema:\n            del reduced_schema[\"type\"]\n        # Ensure json in context is well-formed with double quotes.\n        schema_str = json.dumps(reduced_schema)\n        return PYDANTIC_FORMAT_INSTRUCTIONS.format(schema=schema_str)\n    @property\n    def _type(self) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/pydantic.html"}877{"id": "d3c528a86b2b-1", "text": "@property\n    def _type(self) -> str:\n        return \"pydantic\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/pydantic.html"}878{"id": "59ee8dcf0614-0", "text": "Source code for langchain.output_parsers.regex_dict\nfrom __future__ import annotations\nimport re\nfrom typing import Dict, Optional\nfrom langchain.schema import BaseOutputParser\n[docs]class RegexDictParser(BaseOutputParser):\n    \"\"\"Class to parse the output into a dictionary.\"\"\"\n    regex_pattern: str = r\"{}:\\s?([^.'\\n']*)\\.?\"  # : :meta private:\n    output_key_to_format: Dict[str, str]\n    no_update_value: Optional[str] = None\n    @property\n    def _type(self) -> str:\n        \"\"\"Return the type key.\"\"\"\n        return \"regex_dict_parser\"\n[docs]    def parse(self, text: str) -> Dict[str, str]:\n        \"\"\"Parse the output of an LLM call.\"\"\"\n        result = {}\n        for output_key, expected_format in self.output_key_to_format.items():\n            specific_regex = self.regex_pattern.format(re.escape(expected_format))\n            matches = re.findall(specific_regex, text)\n            if not matches:\n                raise ValueError(\n                    f\"No match found for output key: {output_key} with expected format \\\n                        {expected_format} on text {text}\"\n                )\n            elif len(matches) > 1:\n                raise ValueError(\n                    f\"Multiple matches found for output key: {output_key} with \\\n                        expected format {expected_format} on text {text}\"\n                )\n            elif (\n                self.no_update_value is not None and matches[0] == self.no_update_value\n            ):\n                continue\n            else:\n                result[output_key] = matches[0]\n        return result\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/output_parsers/regex_dict.html"}879{"id": "c11aa59eb29a-0", "text": "Source code for langchain.docstore.wikipedia\n\"\"\"Wrapper around wikipedia API.\"\"\"\nfrom typing import Union\nfrom langchain.docstore.base import Docstore\nfrom langchain.docstore.document import Document\n[docs]class Wikipedia(Docstore):\n    \"\"\"Wrapper around wikipedia API.\"\"\"\n    def __init__(self) -> None:\n        \"\"\"Check that wikipedia package is installed.\"\"\"\n        try:\n            import wikipedia  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"Could not import wikipedia python package. \"\n                \"Please install it with `pip install wikipedia`.\"\n            )\n[docs]    def search(self, search: str) -> Union[str, Document]:\n        \"\"\"Try to search for wiki page.\n        If page exists, return the page summary, and a PageWithLookups object.\n        If page does not exist, return similar entries.\n        \"\"\"\n        import wikipedia\n        try:\n            page_content = wikipedia.page(search).content\n            url = wikipedia.page(search).url\n            result: Union[str, Document] = Document(\n                page_content=page_content, metadata={\"page\": url}\n            )\n        except wikipedia.PageError:\n            result = f\"Could not find [{search}]. Similar: {wikipedia.search(search)}\"\n        except wikipedia.DisambiguationError:\n            result = f\"Could not find [{search}]. Similar: {wikipedia.search(search)}\"\n        return result\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/docstore/wikipedia.html"}880{"id": "91657d09c103-0", "text": "Source code for langchain.docstore.in_memory\n\"\"\"Simple in memory docstore in the form of a dict.\"\"\"\nfrom typing import Dict, Union\nfrom langchain.docstore.base import AddableMixin, Docstore\nfrom langchain.docstore.document import Document\n[docs]class InMemoryDocstore(Docstore, AddableMixin):\n    \"\"\"Simple in memory docstore in the form of a dict.\"\"\"\n    def __init__(self, _dict: Dict[str, Document]):\n        \"\"\"Initialize with dict.\"\"\"\n        self._dict = _dict\n[docs]    def add(self, texts: Dict[str, Document]) -> None:\n        \"\"\"Add texts to in memory dictionary.\"\"\"\n        overlapping = set(texts).intersection(self._dict)\n        if overlapping:\n            raise ValueError(f\"Tried to add ids that already exist: {overlapping}\")\n        self._dict = dict(self._dict, **texts)\n[docs]    def search(self, search: str) -> Union[str, Document]:\n        \"\"\"Search via direct lookup.\"\"\"\n        if search not in self._dict:\n            return f\"ID {search} not found.\"\n        else:\n            return self._dict[search]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/docstore/in_memory.html"}881{"id": "107631cf4e2d-0", "text": "Source code for langchain.prompts.base\n\"\"\"BasePrompt schema definition.\"\"\"\nfrom __future__ import annotations\nimport json\nfrom abc import ABC, abstractmethod\nfrom pathlib import Path\nfrom typing import Any, Callable, Dict, List, Mapping, Optional, Set, Union\nimport yaml\nfrom pydantic import BaseModel, Extra, Field, root_validator\nfrom langchain.formatting import formatter\nfrom langchain.schema import BaseMessage, BaseOutputParser, HumanMessage, PromptValue\ndef jinja2_formatter(template: str, **kwargs: Any) -> str:\n    \"\"\"Format a template using jinja2.\"\"\"\n    try:\n        from jinja2 import Template\n    except ImportError:\n        raise ImportError(\n            \"jinja2 not installed, which is needed to use the jinja2_formatter. \"\n            \"Please install it with `pip install jinja2`.\"\n        )\n    return Template(template).render(**kwargs)\ndef validate_jinja2(template: str, input_variables: List[str]) -> None:\n    input_variables_set = set(input_variables)\n    valid_variables = _get_jinja2_variables_from_template(template)\n    missing_variables = valid_variables - input_variables_set\n    extra_variables = input_variables_set - valid_variables\n    error_message = \"\"\n    if missing_variables:\n        error_message += f\"Missing variables: {missing_variables} \"\n    if extra_variables:\n        error_message += f\"Extra variables: {extra_variables}\"\n    if error_message:\n        raise KeyError(error_message.strip())\ndef _get_jinja2_variables_from_template(template: str) -> Set[str]:\n    try:\n        from jinja2 import Environment, meta\n    except ImportError:\n        raise ImportError(\n            \"jinja2 not installed, which is needed to use the jinja2_formatter. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/base.html"}882{"id": "107631cf4e2d-1", "text": "\"jinja2 not installed, which is needed to use the jinja2_formatter. \"\n            \"Please install it with `pip install jinja2`.\"\n        )\n    env = Environment()\n    ast = env.parse(template)\n    variables = meta.find_undeclared_variables(ast)\n    return variables\nDEFAULT_FORMATTER_MAPPING: Dict[str, Callable] = {\n    \"f-string\": formatter.format,\n    \"jinja2\": jinja2_formatter,\n}\nDEFAULT_VALIDATOR_MAPPING: Dict[str, Callable] = {\n    \"f-string\": formatter.validate_input_variables,\n    \"jinja2\": validate_jinja2,\n}\ndef check_valid_template(\n    template: str, template_format: str, input_variables: List[str]\n) -> None:\n    \"\"\"Check that template string is valid.\"\"\"\n    if template_format not in DEFAULT_FORMATTER_MAPPING:\n        valid_formats = list(DEFAULT_FORMATTER_MAPPING)\n        raise ValueError(\n            f\"Invalid template format. Got `{template_format}`;\"\n            f\" should be one of {valid_formats}\"\n        )\n    try:\n        validator_func = DEFAULT_VALIDATOR_MAPPING[template_format]\n        validator_func(template, input_variables)\n    except KeyError as e:\n        raise ValueError(\n            \"Invalid prompt schema; check for mismatched or missing input parameters. \"\n            + str(e)\n        )\nclass StringPromptValue(PromptValue):\n    text: str\n    def to_string(self) -> str:\n        \"\"\"Return prompt as string.\"\"\"\n        return self.text\n    def to_messages(self) -> List[BaseMessage]:\n        \"\"\"Return prompt as messages.\"\"\"\n        return [HumanMessage(content=self.text)]\n[docs]class BasePromptTemplate(BaseModel, ABC):\n    \"\"\"Base class for all prompt templates, returning a prompt.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/base.html"}883{"id": "107631cf4e2d-2", "text": "\"\"\"Base class for all prompt templates, returning a prompt.\"\"\"\n    input_variables: List[str]\n    \"\"\"A list of the names of the variables the prompt template expects.\"\"\"\n    output_parser: Optional[BaseOutputParser] = None\n    \"\"\"How to parse the output of calling an LLM on this formatted prompt.\"\"\"\n    partial_variables: Mapping[str, Union[str, Callable[[], str]]] = Field(\n        default_factory=dict\n    )\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n[docs]    @abstractmethod\n    def format_prompt(self, **kwargs: Any) -> PromptValue:\n        \"\"\"Create Chat Messages.\"\"\"\n    @root_validator()\n    def validate_variable_names(cls, values: Dict) -> Dict:\n        \"\"\"Validate variable names do not include restricted names.\"\"\"\n        if \"stop\" in values[\"input_variables\"]:\n            raise ValueError(\n                \"Cannot have an input variable named 'stop', as it is used internally,\"\n                \" please rename.\"\n            )\n        if \"stop\" in values[\"partial_variables\"]:\n            raise ValueError(\n                \"Cannot have an partial variable named 'stop', as it is used \"\n                \"internally, please rename.\"\n            )\n        overall = set(values[\"input_variables\"]).intersection(\n            values[\"partial_variables\"]\n        )\n        if overall:\n            raise ValueError(\n                f\"Found overlapping input and partial variables: {overall}\"\n            )\n        return values\n[docs]    def partial(self, **kwargs: Union[str, Callable[[], str]]) -> BasePromptTemplate:\n        \"\"\"Return a partial of the prompt template.\"\"\"\n        prompt_dict = self.__dict__.copy()\n        prompt_dict[\"input_variables\"] = list(\n            set(self.input_variables).difference(kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/base.html"}884{"id": "107631cf4e2d-3", "text": "prompt_dict[\"input_variables\"] = list(\n            set(self.input_variables).difference(kwargs)\n        )\n        prompt_dict[\"partial_variables\"] = {**self.partial_variables, **kwargs}\n        return type(self)(**prompt_dict)\n    def _merge_partial_and_user_variables(self, **kwargs: Any) -> Dict[str, Any]:\n        # Get partial params:\n        partial_kwargs = {\n            k: v if isinstance(v, str) else v()\n            for k, v in self.partial_variables.items()\n        }\n        return {**partial_kwargs, **kwargs}\n[docs]    @abstractmethod\n    def format(self, **kwargs: Any) -> str:\n        \"\"\"Format the prompt with the inputs.\n        Args:\n            kwargs: Any arguments to be passed to the prompt template.\n        Returns:\n            A formatted string.\n        Example:\n        .. code-block:: python\n            prompt.format(variable1=\"foo\")\n        \"\"\"\n    @property\n    def _prompt_type(self) -> str:\n        \"\"\"Return the prompt type key.\"\"\"\n        raise NotImplementedError\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return dictionary representation of prompt.\"\"\"\n        prompt_dict = super().dict(**kwargs)\n        prompt_dict[\"_type\"] = self._prompt_type\n        return prompt_dict\n[docs]    def save(self, file_path: Union[Path, str]) -> None:\n        \"\"\"Save the prompt.\n        Args:\n            file_path: Path to directory to save prompt to.\n        Example:\n        .. code-block:: python\n            prompt.save(file_path=\"path/prompt.yaml\")\n        \"\"\"\n        if self.partial_variables:\n            raise ValueError(\"Cannot save prompt with partial variables.\")\n        # Convert file to Path object.\n        if isinstance(file_path, str):", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/base.html"}885{"id": "107631cf4e2d-4", "text": "# Convert file to Path object.\n        if isinstance(file_path, str):\n            save_path = Path(file_path)\n        else:\n            save_path = file_path\n        directory_path = save_path.parent\n        directory_path.mkdir(parents=True, exist_ok=True)\n        # Fetch dictionary to save\n        prompt_dict = self.dict()\n        if save_path.suffix == \".json\":\n            with open(file_path, \"w\") as f:\n                json.dump(prompt_dict, f, indent=4)\n        elif save_path.suffix == \".yaml\":\n            with open(file_path, \"w\") as f:\n                yaml.dump(prompt_dict, f, default_flow_style=False)\n        else:\n            raise ValueError(f\"{save_path} must be json or yaml\")\n[docs]class StringPromptTemplate(BasePromptTemplate, ABC):\n    \"\"\"String prompt should expose the format method, returning a prompt.\"\"\"\n[docs]    def format_prompt(self, **kwargs: Any) -> PromptValue:\n        \"\"\"Create Chat Messages.\"\"\"\n        return StringPromptValue(text=self.format(**kwargs))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/base.html"}886{"id": "a9a11f902b27-0", "text": "Source code for langchain.prompts.few_shot\n\"\"\"Prompt template that contains few shot examples.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import Extra, root_validator\nfrom langchain.prompts.base import (\n    DEFAULT_FORMATTER_MAPPING,\n    StringPromptTemplate,\n    check_valid_template,\n)\nfrom langchain.prompts.example_selector.base import BaseExampleSelector\nfrom langchain.prompts.prompt import PromptTemplate\n[docs]class FewShotPromptTemplate(StringPromptTemplate):\n    \"\"\"Prompt template that contains few shot examples.\"\"\"\n    examples: Optional[List[dict]] = None\n    \"\"\"Examples to format into the prompt.\n    Either this or example_selector should be provided.\"\"\"\n    example_selector: Optional[BaseExampleSelector] = None\n    \"\"\"ExampleSelector to choose the examples to format into the prompt.\n    Either this or examples should be provided.\"\"\"\n    example_prompt: PromptTemplate\n    \"\"\"PromptTemplate used to format an individual example.\"\"\"\n    suffix: str\n    \"\"\"A prompt template string to put after the examples.\"\"\"\n    input_variables: List[str]\n    \"\"\"A list of the names of the variables the prompt template expects.\"\"\"\n    example_separator: str = \"\\n\\n\"\n    \"\"\"String separator used to join the prefix, the examples, and suffix.\"\"\"\n    prefix: str = \"\"\n    \"\"\"A prompt template string to put before the examples.\"\"\"\n    template_format: str = \"f-string\"\n    \"\"\"The format of the prompt template. Options are: 'f-string', 'jinja2'.\"\"\"\n    validate_template: bool = True\n    \"\"\"Whether or not to try validating the template.\"\"\"\n    @root_validator(pre=True)\n    def check_examples_and_selector(cls, values: Dict) -> Dict:\n        \"\"\"Check that one and only one of examples/example_selector are provided.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot.html"}887{"id": "a9a11f902b27-1", "text": "\"\"\"Check that one and only one of examples/example_selector are provided.\"\"\"\n        examples = values.get(\"examples\", None)\n        example_selector = values.get(\"example_selector\", None)\n        if examples and example_selector:\n            raise ValueError(\n                \"Only one of 'examples' and 'example_selector' should be provided\"\n            )\n        if examples is None and example_selector is None:\n            raise ValueError(\n                \"One of 'examples' and 'example_selector' should be provided\"\n            )\n        return values\n    @root_validator()\n    def template_is_valid(cls, values: Dict) -> Dict:\n        \"\"\"Check that prefix, suffix and input variables are consistent.\"\"\"\n        if values[\"validate_template\"]:\n            check_valid_template(\n                values[\"prefix\"] + values[\"suffix\"],\n                values[\"template_format\"],\n                values[\"input_variables\"] + list(values[\"partial_variables\"]),\n            )\n        return values\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    def _get_examples(self, **kwargs: Any) -> List[dict]:\n        if self.examples is not None:\n            return self.examples\n        elif self.example_selector is not None:\n            return self.example_selector.select_examples(kwargs)\n        else:\n            raise ValueError\n[docs]    def format(self, **kwargs: Any) -> str:\n        \"\"\"Format the prompt with the inputs.\n        Args:\n            kwargs: Any arguments to be passed to the prompt template.\n        Returns:\n            A formatted string.\n        Example:\n        .. code-block:: python\n            prompt.format(variable1=\"foo\")\n        \"\"\"\n        kwargs = self._merge_partial_and_user_variables(**kwargs)\n        # Get the examples to use.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot.html"}888{"id": "a9a11f902b27-2", "text": "# Get the examples to use.\n        examples = self._get_examples(**kwargs)\n        examples = [\n            {k: e[k] for k in self.example_prompt.input_variables} for e in examples\n        ]\n        # Format the examples.\n        example_strings = [\n            self.example_prompt.format(**example) for example in examples\n        ]\n        # Create the overall template.\n        pieces = [self.prefix, *example_strings, self.suffix]\n        template = self.example_separator.join([piece for piece in pieces if piece])\n        # Format the template with the input variables.\n        return DEFAULT_FORMATTER_MAPPING[self.template_format](template, **kwargs)\n    @property\n    def _prompt_type(self) -> str:\n        \"\"\"Return the prompt type key.\"\"\"\n        return \"few_shot\"\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return a dictionary of the prompt.\"\"\"\n        if self.example_selector:\n            raise ValueError(\"Saving an example selector is not currently supported\")\n        return super().dict(**kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot.html"}889{"id": "a505ffca0e47-0", "text": "Source code for langchain.prompts.prompt\n\"\"\"Prompt schema definition.\"\"\"\nfrom __future__ import annotations\nfrom pathlib import Path\nfrom string import Formatter\nfrom typing import Any, Dict, List, Union\nfrom pydantic import Extra, root_validator\nfrom langchain.prompts.base import (\n    DEFAULT_FORMATTER_MAPPING,\n    StringPromptTemplate,\n    _get_jinja2_variables_from_template,\n    check_valid_template,\n)\n[docs]class PromptTemplate(StringPromptTemplate):\n    \"\"\"Schema to represent a prompt for an LLM.\n    Example:\n        .. code-block:: python\n            from langchain import PromptTemplate\n            prompt = PromptTemplate(input_variables=[\"foo\"], template=\"Say {foo}\")\n    \"\"\"\n    input_variables: List[str]\n    \"\"\"A list of the names of the variables the prompt template expects.\"\"\"\n    template: str\n    \"\"\"The prompt template.\"\"\"\n    template_format: str = \"f-string\"\n    \"\"\"The format of the prompt template. Options are: 'f-string', 'jinja2'.\"\"\"\n    validate_template: bool = True\n    \"\"\"Whether or not to try validating the template.\"\"\"\n    @property\n    def _prompt_type(self) -> str:\n        \"\"\"Return the prompt type key.\"\"\"\n        return \"prompt\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    def format(self, **kwargs: Any) -> str:\n        \"\"\"Format the prompt with the inputs.\n        Args:\n            kwargs: Any arguments to be passed to the prompt template.\n        Returns:\n            A formatted string.\n        Example:\n        .. code-block:: python\n            prompt.format(variable1=\"foo\")\n        \"\"\"\n        kwargs = self._merge_partial_and_user_variables(**kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/prompt.html"}890{"id": "a505ffca0e47-1", "text": "\"\"\"\n        kwargs = self._merge_partial_and_user_variables(**kwargs)\n        return DEFAULT_FORMATTER_MAPPING[self.template_format](self.template, **kwargs)\n    @root_validator()\n    def template_is_valid(cls, values: Dict) -> Dict:\n        \"\"\"Check that template and input variables are consistent.\"\"\"\n        if values[\"validate_template\"]:\n            all_inputs = values[\"input_variables\"] + list(values[\"partial_variables\"])\n            check_valid_template(\n                values[\"template\"], values[\"template_format\"], all_inputs\n            )\n        return values\n[docs]    @classmethod\n    def from_examples(\n        cls,\n        examples: List[str],\n        suffix: str,\n        input_variables: List[str],\n        example_separator: str = \"\\n\\n\",\n        prefix: str = \"\",\n        **kwargs: Any,\n    ) -> PromptTemplate:\n        \"\"\"Take examples in list format with prefix and suffix to create a prompt.\n        Intended to be used as a way to dynamically create a prompt from examples.\n        Args:\n            examples: List of examples to use in the prompt.\n            suffix: String to go after the list of examples. Should generally\n                set up the user's input.\n            input_variables: A list of variable names the final prompt template\n                will expect.\n            example_separator: The separator to use in between examples. Defaults\n                to two new line characters.\n            prefix: String that should go before any examples. Generally includes\n                examples. Default to an empty string.\n        Returns:\n            The final prompt generated.\n        \"\"\"\n        template = example_separator.join([prefix, *examples, suffix])\n        return cls(input_variables=input_variables, template=template, **kwargs)\n[docs]    @classmethod\n    def from_file(", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/prompt.html"}891{"id": "a505ffca0e47-2", "text": "[docs]    @classmethod\n    def from_file(\n        cls, template_file: Union[str, Path], input_variables: List[str], **kwargs: Any\n    ) -> PromptTemplate:\n        \"\"\"Load a prompt from a file.\n        Args:\n            template_file: The path to the file containing the prompt template.\n            input_variables: A list of variable names the final prompt template\n                will expect.\n        Returns:\n            The prompt loaded from the file.\n        \"\"\"\n        with open(str(template_file), \"r\") as f:\n            template = f.read()\n        return cls(input_variables=input_variables, template=template, **kwargs)\n[docs]    @classmethod\n    def from_template(cls, template: str, **kwargs: Any) -> PromptTemplate:\n        \"\"\"Load a prompt template from a template.\"\"\"\n        if \"template_format\" in kwargs and kwargs[\"template_format\"] == \"jinja2\":\n            # Get the variables for the template\n            input_variables = _get_jinja2_variables_from_template(template)\n        else:\n            input_variables = {\n                v for _, v, _, _ in Formatter().parse(template) if v is not None\n            }\n        if \"partial_variables\" in kwargs:\n            partial_variables = kwargs[\"partial_variables\"]\n            input_variables = {\n                var for var in input_variables if var not in partial_variables\n            }\n        return cls(\n            input_variables=list(sorted(input_variables)), template=template, **kwargs\n        )\n# For backwards compatibility.\nPrompt = PromptTemplate\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/prompt.html"}892{"id": "0a5b34af04e6-0", "text": "Source code for langchain.prompts.loading\n\"\"\"Load prompts from disk.\"\"\"\nimport importlib\nimport json\nimport logging\nfrom pathlib import Path\nfrom typing import Union\nimport yaml\nfrom langchain.output_parsers.regex import RegexParser\nfrom langchain.prompts.base import BasePromptTemplate\nfrom langchain.prompts.few_shot import FewShotPromptTemplate\nfrom langchain.prompts.prompt import PromptTemplate\nfrom langchain.utilities.loading import try_load_from_hub\nURL_BASE = \"https://raw.githubusercontent.com/hwchase17/langchain-hub/master/prompts/\"\nlogger = logging.getLogger(__name__)\ndef load_prompt_from_config(config: dict) -> BasePromptTemplate:\n    \"\"\"Load prompt from Config Dict.\"\"\"\n    if \"_type\" not in config:\n        logger.warning(\"No `_type` key found, defaulting to `prompt`.\")\n    config_type = config.pop(\"_type\", \"prompt\")\n    if config_type not in type_to_loader_dict:\n        raise ValueError(f\"Loading {config_type} prompt not supported\")\n    prompt_loader = type_to_loader_dict[config_type]\n    return prompt_loader(config)\ndef _load_template(var_name: str, config: dict) -> dict:\n    \"\"\"Load template from disk if applicable.\"\"\"\n    # Check if template_path exists in config.\n    if f\"{var_name}_path\" in config:\n        # If it does, make sure template variable doesn't also exist.\n        if var_name in config:\n            raise ValueError(\n                f\"Both `{var_name}_path` and `{var_name}` cannot be provided.\"\n            )\n        # Pop the template path from the config.\n        template_path = Path(config.pop(f\"{var_name}_path\"))\n        # Load the template.\n        if template_path.suffix == \".txt\":\n            with open(template_path) as f:", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/loading.html"}893{"id": "0a5b34af04e6-1", "text": "if template_path.suffix == \".txt\":\n            with open(template_path) as f:\n                template = f.read()\n        else:\n            raise ValueError\n        # Set the template variable to the extracted variable.\n        config[var_name] = template\n    return config\ndef _load_examples(config: dict) -> dict:\n    \"\"\"Load examples if necessary.\"\"\"\n    if isinstance(config[\"examples\"], list):\n        pass\n    elif isinstance(config[\"examples\"], str):\n        with open(config[\"examples\"]) as f:\n            if config[\"examples\"].endswith(\".json\"):\n                examples = json.load(f)\n            elif config[\"examples\"].endswith((\".yaml\", \".yml\")):\n                examples = yaml.safe_load(f)\n            else:\n                raise ValueError(\n                    \"Invalid file format. Only json or yaml formats are supported.\"\n                )\n        config[\"examples\"] = examples\n    else:\n        raise ValueError(\"Invalid examples format. Only list or string are supported.\")\n    return config\ndef _load_output_parser(config: dict) -> dict:\n    \"\"\"Load output parser.\"\"\"\n    if \"output_parser\" in config and config[\"output_parser\"]:\n        _config = config.pop(\"output_parser\")\n        output_parser_type = _config.pop(\"_type\")\n        if output_parser_type == \"regex_parser\":\n            output_parser = RegexParser(**_config)\n        else:\n            raise ValueError(f\"Unsupported output parser {output_parser_type}\")\n        config[\"output_parser\"] = output_parser\n    return config\ndef _load_few_shot_prompt(config: dict) -> FewShotPromptTemplate:\n    \"\"\"Load the few shot prompt from the config.\"\"\"\n    # Load the suffix and prefix templates.\n    config = _load_template(\"suffix\", config)\n    config = _load_template(\"prefix\", config)", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/loading.html"}894{"id": "0a5b34af04e6-2", "text": "config = _load_template(\"prefix\", config)\n    # Load the example prompt.\n    if \"example_prompt_path\" in config:\n        if \"example_prompt\" in config:\n            raise ValueError(\n                \"Only one of example_prompt and example_prompt_path should \"\n                \"be specified.\"\n            )\n        config[\"example_prompt\"] = load_prompt(config.pop(\"example_prompt_path\"))\n    else:\n        config[\"example_prompt\"] = load_prompt_from_config(config[\"example_prompt\"])\n    # Load the examples.\n    config = _load_examples(config)\n    config = _load_output_parser(config)\n    return FewShotPromptTemplate(**config)\ndef _load_prompt(config: dict) -> PromptTemplate:\n    \"\"\"Load the prompt template from config.\"\"\"\n    # Load the template from disk if necessary.\n    config = _load_template(\"template\", config)\n    config = _load_output_parser(config)\n    return PromptTemplate(**config)\n[docs]def load_prompt(path: Union[str, Path]) -> BasePromptTemplate:\n    \"\"\"Unified method for loading a prompt from LangChainHub or local fs.\"\"\"\n    if hub_result := try_load_from_hub(\n        path, _load_prompt_from_file, \"prompts\", {\"py\", \"json\", \"yaml\"}\n    ):\n        return hub_result\n    else:\n        return _load_prompt_from_file(path)\ndef _load_prompt_from_file(file: Union[str, Path]) -> BasePromptTemplate:\n    \"\"\"Load prompt from file.\"\"\"\n    # Convert file to Path object.\n    if isinstance(file, str):\n        file_path = Path(file)\n    else:\n        file_path = file\n    # Load from either json or yaml.\n    if file_path.suffix == \".json\":\n        with open(file_path) as f:\n            config = json.load(f)", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/loading.html"}895{"id": "0a5b34af04e6-3", "text": "with open(file_path) as f:\n            config = json.load(f)\n    elif file_path.suffix == \".yaml\":\n        with open(file_path, \"r\") as f:\n            config = yaml.safe_load(f)\n    elif file_path.suffix == \".py\":\n        spec = importlib.util.spec_from_loader(\n            \"prompt\", loader=None, origin=str(file_path)\n        )\n        if spec is None:\n            raise ValueError(\"could not load spec\")\n        helper = importlib.util.module_from_spec(spec)\n        with open(file_path, \"rb\") as f:\n            exec(f.read(), helper.__dict__)\n        if not isinstance(helper.PROMPT, BasePromptTemplate):\n            raise ValueError(\"Did not get object of type BasePromptTemplate.\")\n        return helper.PROMPT\n    else:\n        raise ValueError(f\"Got unsupported file type {file_path.suffix}\")\n    # Load the prompt from the config now.\n    return load_prompt_from_config(config)\ntype_to_loader_dict = {\n    \"prompt\": _load_prompt,\n    \"few_shot\": _load_few_shot_prompt,\n    # \"few_shot_with_templates\": _load_few_shot_with_templates_prompt,\n}\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/loading.html"}896{"id": "388d1b99b9c6-0", "text": "Source code for langchain.prompts.few_shot_with_templates\n\"\"\"Prompt template that contains few shot examples.\"\"\"\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import Extra, root_validator\nfrom langchain.prompts.base import DEFAULT_FORMATTER_MAPPING, StringPromptTemplate\nfrom langchain.prompts.example_selector.base import BaseExampleSelector\nfrom langchain.prompts.prompt import PromptTemplate\n[docs]class FewShotPromptWithTemplates(StringPromptTemplate):\n    \"\"\"Prompt template that contains few shot examples.\"\"\"\n    examples: Optional[List[dict]] = None\n    \"\"\"Examples to format into the prompt.\n    Either this or example_selector should be provided.\"\"\"\n    example_selector: Optional[BaseExampleSelector] = None\n    \"\"\"ExampleSelector to choose the examples to format into the prompt.\n    Either this or examples should be provided.\"\"\"\n    example_prompt: PromptTemplate\n    \"\"\"PromptTemplate used to format an individual example.\"\"\"\n    suffix: StringPromptTemplate\n    \"\"\"A PromptTemplate to put after the examples.\"\"\"\n    input_variables: List[str]\n    \"\"\"A list of the names of the variables the prompt template expects.\"\"\"\n    example_separator: str = \"\\n\\n\"\n    \"\"\"String separator used to join the prefix, the examples, and suffix.\"\"\"\n    prefix: Optional[StringPromptTemplate] = None\n    \"\"\"A PromptTemplate to put before the examples.\"\"\"\n    template_format: str = \"f-string\"\n    \"\"\"The format of the prompt template. Options are: 'f-string', 'jinja2'.\"\"\"\n    validate_template: bool = True\n    \"\"\"Whether or not to try validating the template.\"\"\"\n    @root_validator(pre=True)\n    def check_examples_and_selector(cls, values: Dict) -> Dict:\n        \"\"\"Check that one and only one of examples/example_selector are provided.\"\"\"\n        examples = values.get(\"examples\", None)", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot_with_templates.html"}897{"id": "388d1b99b9c6-1", "text": "examples = values.get(\"examples\", None)\n        example_selector = values.get(\"example_selector\", None)\n        if examples and example_selector:\n            raise ValueError(\n                \"Only one of 'examples' and 'example_selector' should be provided\"\n            )\n        if examples is None and example_selector is None:\n            raise ValueError(\n                \"One of 'examples' and 'example_selector' should be provided\"\n            )\n        return values\n    @root_validator()\n    def template_is_valid(cls, values: Dict) -> Dict:\n        \"\"\"Check that prefix, suffix and input variables are consistent.\"\"\"\n        if values[\"validate_template\"]:\n            input_variables = values[\"input_variables\"]\n            expected_input_variables = set(values[\"suffix\"].input_variables)\n            expected_input_variables |= set(values[\"partial_variables\"])\n            if values[\"prefix\"] is not None:\n                expected_input_variables |= set(values[\"prefix\"].input_variables)\n            missing_vars = expected_input_variables.difference(input_variables)\n            if missing_vars:\n                raise ValueError(\n                    f\"Got input_variables={input_variables}, but based on \"\n                    f\"prefix/suffix expected {expected_input_variables}\"\n                )\n        return values\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n    def _get_examples(self, **kwargs: Any) -> List[dict]:\n        if self.examples is not None:\n            return self.examples\n        elif self.example_selector is not None:\n            return self.example_selector.select_examples(kwargs)\n        else:\n            raise ValueError\n[docs]    def format(self, **kwargs: Any) -> str:\n        \"\"\"Format the prompt with the inputs.\n        Args:\n            kwargs: Any arguments to be passed to the prompt template.\n        Returns:", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot_with_templates.html"}898{"id": "388d1b99b9c6-2", "text": "kwargs: Any arguments to be passed to the prompt template.\n        Returns:\n            A formatted string.\n        Example:\n        .. code-block:: python\n            prompt.format(variable1=\"foo\")\n        \"\"\"\n        kwargs = self._merge_partial_and_user_variables(**kwargs)\n        # Get the examples to use.\n        examples = self._get_examples(**kwargs)\n        # Format the examples.\n        example_strings = [\n            self.example_prompt.format(**example) for example in examples\n        ]\n        # Create the overall prefix.\n        if self.prefix is None:\n            prefix = \"\"\n        else:\n            prefix_kwargs = {\n                k: v for k, v in kwargs.items() if k in self.prefix.input_variables\n            }\n            for k in prefix_kwargs.keys():\n                kwargs.pop(k)\n            prefix = self.prefix.format(**prefix_kwargs)\n        # Create the overall suffix\n        suffix_kwargs = {\n            k: v for k, v in kwargs.items() if k in self.suffix.input_variables\n        }\n        for k in suffix_kwargs.keys():\n            kwargs.pop(k)\n        suffix = self.suffix.format(\n            **suffix_kwargs,\n        )\n        pieces = [prefix, *example_strings, suffix]\n        template = self.example_separator.join([piece for piece in pieces if piece])\n        # Format the template with the input variables.\n        return DEFAULT_FORMATTER_MAPPING[self.template_format](template, **kwargs)\n    @property\n    def _prompt_type(self) -> str:\n        \"\"\"Return the prompt type key.\"\"\"\n        return \"few_shot_with_templates\"\n[docs]    def dict(self, **kwargs: Any) -> Dict:\n        \"\"\"Return a dictionary of the prompt.\"\"\"\n        if self.example_selector:\n            raise ValueError(\"Saving an example selector is not currently supported\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot_with_templates.html"}899{"id": "388d1b99b9c6-3", "text": "if self.example_selector:\n            raise ValueError(\"Saving an example selector is not currently supported\")\n        return super().dict(**kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/few_shot_with_templates.html"}900{"id": "8d79cdffffbc-0", "text": "Source code for langchain.prompts.chat\n\"\"\"Chat prompt template.\"\"\"\nfrom __future__ import annotations\nfrom abc import ABC, abstractmethod\nfrom pathlib import Path\nfrom typing import Any, Callable, List, Sequence, Tuple, Type, TypeVar, Union\nfrom pydantic import BaseModel, Field\nfrom langchain.memory.buffer import get_buffer_string\nfrom langchain.prompts.base import BasePromptTemplate, StringPromptTemplate\nfrom langchain.prompts.prompt import PromptTemplate\nfrom langchain.schema import (\n    AIMessage,\n    BaseMessage,\n    ChatMessage,\n    HumanMessage,\n    PromptValue,\n    SystemMessage,\n)\nclass BaseMessagePromptTemplate(BaseModel, ABC):\n    @abstractmethod\n    def format_messages(self, **kwargs: Any) -> List[BaseMessage]:\n        \"\"\"To messages.\"\"\"\n    @property\n    @abstractmethod\n    def input_variables(self) -> List[str]:\n        \"\"\"Input variables for this prompt template.\"\"\"\n[docs]class MessagesPlaceholder(BaseMessagePromptTemplate):\n    \"\"\"Prompt template that assumes variable is already list of messages.\"\"\"\n    variable_name: str\n[docs]    def format_messages(self, **kwargs: Any) -> List[BaseMessage]:\n        \"\"\"To a BaseMessage.\"\"\"\n        value = kwargs[self.variable_name]\n        if not isinstance(value, list):\n            raise ValueError(\n                f\"variable {self.variable_name} should be a list of base messages, \"\n                f\"got {value}\"\n            )\n        for v in value:\n            if not isinstance(v, BaseMessage):\n                raise ValueError(\n                    f\"variable {self.variable_name} should be a list of base messages,\"\n                    f\" got {value}\"\n                )\n        return value\n    @property\n    def input_variables(self) -> List[str]:\n        \"\"\"Input variables for this prompt template.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/chat.html"}901{"id": "8d79cdffffbc-1", "text": "def input_variables(self) -> List[str]:\n        \"\"\"Input variables for this prompt template.\"\"\"\n        return [self.variable_name]\nMessagePromptTemplateT = TypeVar(\n    \"MessagePromptTemplateT\", bound=\"BaseStringMessagePromptTemplate\"\n)\nclass BaseStringMessagePromptTemplate(BaseMessagePromptTemplate, ABC):\n    prompt: StringPromptTemplate\n    additional_kwargs: dict = Field(default_factory=dict)\n    @classmethod\n    def from_template(\n        cls: Type[MessagePromptTemplateT], template: str, **kwargs: Any\n    ) -> MessagePromptTemplateT:\n        prompt = PromptTemplate.from_template(template)\n        return cls(prompt=prompt, **kwargs)\n    @classmethod\n    def from_template_file(\n        cls: Type[MessagePromptTemplateT],\n        template_file: Union[str, Path],\n        input_variables: List[str],\n        **kwargs: Any,\n    ) -> MessagePromptTemplateT:\n        prompt = PromptTemplate.from_file(template_file, input_variables)\n        return cls(prompt=prompt, **kwargs)\n    @abstractmethod\n    def format(self, **kwargs: Any) -> BaseMessage:\n        \"\"\"To a BaseMessage.\"\"\"\n    def format_messages(self, **kwargs: Any) -> List[BaseMessage]:\n        return [self.format(**kwargs)]\n    @property\n    def input_variables(self) -> List[str]:\n        return self.prompt.input_variables\nclass ChatMessagePromptTemplate(BaseStringMessagePromptTemplate):\n    role: str\n    def format(self, **kwargs: Any) -> BaseMessage:\n        text = self.prompt.format(**kwargs)\n        return ChatMessage(\n            content=text, role=self.role, additional_kwargs=self.additional_kwargs\n        )\nclass HumanMessagePromptTemplate(BaseStringMessagePromptTemplate):\n    def format(self, **kwargs: Any) -> BaseMessage:", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/chat.html"}902{"id": "8d79cdffffbc-2", "text": "def format(self, **kwargs: Any) -> BaseMessage:\n        text = self.prompt.format(**kwargs)\n        return HumanMessage(content=text, additional_kwargs=self.additional_kwargs)\nclass AIMessagePromptTemplate(BaseStringMessagePromptTemplate):\n    def format(self, **kwargs: Any) -> BaseMessage:\n        text = self.prompt.format(**kwargs)\n        return AIMessage(content=text, additional_kwargs=self.additional_kwargs)\nclass SystemMessagePromptTemplate(BaseStringMessagePromptTemplate):\n    def format(self, **kwargs: Any) -> BaseMessage:\n        text = self.prompt.format(**kwargs)\n        return SystemMessage(content=text, additional_kwargs=self.additional_kwargs)\nclass ChatPromptValue(PromptValue):\n    messages: List[BaseMessage]\n    def to_string(self) -> str:\n        \"\"\"Return prompt as string.\"\"\"\n        return get_buffer_string(self.messages)\n    def to_messages(self) -> List[BaseMessage]:\n        \"\"\"Return prompt as messages.\"\"\"\n        return self.messages\n[docs]class BaseChatPromptTemplate(BasePromptTemplate, ABC):\n[docs]    def format(self, **kwargs: Any) -> str:\n        return self.format_prompt(**kwargs).to_string()\n[docs]    def format_prompt(self, **kwargs: Any) -> PromptValue:\n        messages = self.format_messages(**kwargs)\n        return ChatPromptValue(messages=messages)\n[docs]    @abstractmethod\n    def format_messages(self, **kwargs: Any) -> List[BaseMessage]:\n        \"\"\"Format kwargs into a list of messages.\"\"\"\n[docs]class ChatPromptTemplate(BaseChatPromptTemplate, ABC):\n    input_variables: List[str]\n    messages: List[Union[BaseMessagePromptTemplate, BaseMessage]]\n    @classmethod\n    def from_template(cls, template: str, **kwargs: Any) -> ChatPromptTemplate:", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/chat.html"}903{"id": "8d79cdffffbc-3", "text": "def from_template(cls, template: str, **kwargs: Any) -> ChatPromptTemplate:\n        prompt_template = PromptTemplate.from_template(template, **kwargs)\n        message = HumanMessagePromptTemplate(prompt=prompt_template)\n        return cls.from_messages([message])\n    @classmethod\n    def from_role_strings(\n        cls, string_messages: List[Tuple[str, str]]\n    ) -> ChatPromptTemplate:\n        messages = [\n            ChatMessagePromptTemplate(\n                prompt=PromptTemplate.from_template(template), role=role\n            )\n            for role, template in string_messages\n        ]\n        return cls.from_messages(messages)\n    @classmethod\n    def from_strings(\n        cls, string_messages: List[Tuple[Type[BaseMessagePromptTemplate], str]]\n    ) -> ChatPromptTemplate:\n        messages = [\n            role(prompt=PromptTemplate.from_template(template))\n            for role, template in string_messages\n        ]\n        return cls.from_messages(messages)\n    @classmethod\n    def from_messages(\n        cls, messages: Sequence[Union[BaseMessagePromptTemplate, BaseMessage]]\n    ) -> ChatPromptTemplate:\n        input_vars = set()\n        for message in messages:\n            if isinstance(message, BaseMessagePromptTemplate):\n                input_vars.update(message.input_variables)\n        return cls(input_variables=list(input_vars), messages=messages)\n[docs]    def format(self, **kwargs: Any) -> str:\n        return self.format_prompt(**kwargs).to_string()\n[docs]    def format_messages(self, **kwargs: Any) -> List[BaseMessage]:\n        kwargs = self._merge_partial_and_user_variables(**kwargs)\n        result = []\n        for message_template in self.messages:\n            if isinstance(message_template, BaseMessage):\n                result.extend([message_template])", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/chat.html"}904{"id": "8d79cdffffbc-4", "text": "if isinstance(message_template, BaseMessage):\n                result.extend([message_template])\n            elif isinstance(message_template, BaseMessagePromptTemplate):\n                rel_params = {\n                    k: v\n                    for k, v in kwargs.items()\n                    if k in message_template.input_variables\n                }\n                message = message_template.format_messages(**rel_params)\n                result.extend(message)\n            else:\n                raise ValueError(f\"Unexpected input: {message_template}\")\n        return result\n[docs]    def partial(self, **kwargs: Union[str, Callable[[], str]]) -> BasePromptTemplate:\n        raise NotImplementedError\n    @property\n    def _prompt_type(self) -> str:\n        raise NotImplementedError\n[docs]    def save(self, file_path: Union[Path, str]) -> None:\n        raise NotImplementedError\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/chat.html"}905{"id": "feb4b7b40235-0", "text": "Source code for langchain.prompts.example_selector.semantic_similarity\n\"\"\"Example selector that selects examples based on SemanticSimilarity.\"\"\"\nfrom __future__ import annotations\nfrom typing import Any, Dict, List, Optional, Type\nfrom pydantic import BaseModel, Extra\nfrom langchain.embeddings.base import Embeddings\nfrom langchain.prompts.example_selector.base import BaseExampleSelector\nfrom langchain.vectorstores.base import VectorStore\ndef sorted_values(values: Dict[str, str]) -> List[Any]:\n    \"\"\"Return a list of values in dict sorted by key.\"\"\"\n    return [values[val] for val in sorted(values)]\n[docs]class SemanticSimilarityExampleSelector(BaseExampleSelector, BaseModel):\n    \"\"\"Example selector that selects examples based on SemanticSimilarity.\"\"\"\n    vectorstore: VectorStore\n    \"\"\"VectorStore than contains information about examples.\"\"\"\n    k: int = 4\n    \"\"\"Number of examples to select.\"\"\"\n    example_keys: Optional[List[str]] = None\n    \"\"\"Optional keys to filter examples to.\"\"\"\n    input_keys: Optional[List[str]] = None\n    \"\"\"Optional keys to filter input to. If provided, the search is based on\n    the input variables instead of all variables.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n        arbitrary_types_allowed = True\n[docs]    def add_example(self, example: Dict[str, str]) -> str:\n        \"\"\"Add new example to vectorstore.\"\"\"\n        if self.input_keys:\n            string_example = \" \".join(\n                sorted_values({key: example[key] for key in self.input_keys})\n            )\n        else:\n            string_example = \" \".join(sorted_values(example))\n        ids = self.vectorstore.add_texts([string_example], metadatas=[example])\n        return ids[0]", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/semantic_similarity.html"}906{"id": "feb4b7b40235-1", "text": "return ids[0]\n[docs]    def select_examples(self, input_variables: Dict[str, str]) -> List[dict]:\n        \"\"\"Select which examples to use based on semantic similarity.\"\"\"\n        # Get the docs with the highest similarity.\n        if self.input_keys:\n            input_variables = {key: input_variables[key] for key in self.input_keys}\n        query = \" \".join(sorted_values(input_variables))\n        example_docs = self.vectorstore.similarity_search(query, k=self.k)\n        # Get the examples from the metadata.\n        # This assumes that examples are stored in metadata.\n        examples = [dict(e.metadata) for e in example_docs]\n        # If example keys are provided, filter examples to those keys.\n        if self.example_keys:\n            examples = [{k: eg[k] for k in self.example_keys} for eg in examples]\n        return examples\n[docs]    @classmethod\n    def from_examples(\n        cls,\n        examples: List[dict],\n        embeddings: Embeddings,\n        vectorstore_cls: Type[VectorStore],\n        k: int = 4,\n        input_keys: Optional[List[str]] = None,\n        **vectorstore_cls_kwargs: Any,\n    ) -> SemanticSimilarityExampleSelector:\n        \"\"\"Create k-shot example selector using example list and embeddings.\n        Reshuffles examples dynamically based on query similarity.\n        Args:\n            examples: List of examples to use in the prompt.\n            embeddings: An initialized embedding API interface, e.g. OpenAIEmbeddings().\n            vectorstore_cls: A vector store DB interface class, e.g. FAISS.\n            k: Number of examples to select\n            input_keys: If provided, the search is based on the input variables\n                instead of all variables.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/semantic_similarity.html"}907{"id": "feb4b7b40235-2", "text": "instead of all variables.\n            vectorstore_cls_kwargs: optional kwargs containing url for vector store\n        Returns:\n            The ExampleSelector instantiated, backed by a vector store.\n        \"\"\"\n        if input_keys:\n            string_examples = [\n                \" \".join(sorted_values({k: eg[k] for k in input_keys}))\n                for eg in examples\n            ]\n        else:\n            string_examples = [\" \".join(sorted_values(eg)) for eg in examples]\n        vectorstore = vectorstore_cls.from_texts(\n            string_examples, embeddings, metadatas=examples, **vectorstore_cls_kwargs\n        )\n        return cls(vectorstore=vectorstore, k=k, input_keys=input_keys)\n[docs]class MaxMarginalRelevanceExampleSelector(SemanticSimilarityExampleSelector):\n    \"\"\"ExampleSelector that selects examples based on Max Marginal Relevance.\n    This was shown to improve performance in this paper:\n    https://arxiv.org/pdf/2211.13892.pdf\n    \"\"\"\n    fetch_k: int = 20\n    \"\"\"Number of examples to fetch to rerank.\"\"\"\n[docs]    def select_examples(self, input_variables: Dict[str, str]) -> List[dict]:\n        \"\"\"Select which examples to use based on semantic similarity.\"\"\"\n        # Get the docs with the highest similarity.\n        if self.input_keys:\n            input_variables = {key: input_variables[key] for key in self.input_keys}\n        query = \" \".join(sorted_values(input_variables))\n        example_docs = self.vectorstore.max_marginal_relevance_search(\n            query, k=self.k, fetch_k=self.fetch_k\n        )\n        # Get the examples from the metadata.\n        # This assumes that examples are stored in metadata.\n        examples = [dict(e.metadata) for e in example_docs]", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/semantic_similarity.html"}908{"id": "feb4b7b40235-3", "text": "examples = [dict(e.metadata) for e in example_docs]\n        # If example keys are provided, filter examples to those keys.\n        if self.example_keys:\n            examples = [{k: eg[k] for k in self.example_keys} for eg in examples]\n        return examples\n[docs]    @classmethod\n    def from_examples(\n        cls,\n        examples: List[dict],\n        embeddings: Embeddings,\n        vectorstore_cls: Type[VectorStore],\n        k: int = 4,\n        input_keys: Optional[List[str]] = None,\n        fetch_k: int = 20,\n        **vectorstore_cls_kwargs: Any,\n    ) -> MaxMarginalRelevanceExampleSelector:\n        \"\"\"Create k-shot example selector using example list and embeddings.\n        Reshuffles examples dynamically based on query similarity.\n        Args:\n            examples: List of examples to use in the prompt.\n            embeddings: An iniialized embedding API interface, e.g. OpenAIEmbeddings().\n            vectorstore_cls: A vector store DB interface class, e.g. FAISS.\n            k: Number of examples to select\n            input_keys: If provided, the search is based on the input variables\n                instead of all variables.\n            vectorstore_cls_kwargs: optional kwargs containing url for vector store\n        Returns:\n            The ExampleSelector instantiated, backed by a vector store.\n        \"\"\"\n        if input_keys:\n            string_examples = [\n                \" \".join(sorted_values({k: eg[k] for k in input_keys}))\n                for eg in examples\n            ]\n        else:\n            string_examples = [\" \".join(sorted_values(eg)) for eg in examples]\n        vectorstore = vectorstore_cls.from_texts(\n            string_examples, embeddings, metadatas=examples, **vectorstore_cls_kwargs\n        )", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/semantic_similarity.html"}909{"id": "feb4b7b40235-4", "text": "string_examples, embeddings, metadatas=examples, **vectorstore_cls_kwargs\n        )\n        return cls(vectorstore=vectorstore, k=k, fetch_k=fetch_k, input_keys=input_keys)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/semantic_similarity.html"}910{"id": "14a3be0a79ee-0", "text": "Source code for langchain.prompts.example_selector.length_based\n\"\"\"Select examples based on length.\"\"\"\nimport re\nfrom typing import Callable, Dict, List\nfrom pydantic import BaseModel, validator\nfrom langchain.prompts.example_selector.base import BaseExampleSelector\nfrom langchain.prompts.prompt import PromptTemplate\ndef _get_length_based(text: str) -> int:\n    return len(re.split(\"\\n| \", text))\n[docs]class LengthBasedExampleSelector(BaseExampleSelector, BaseModel):\n    \"\"\"Select examples based on length.\"\"\"\n    examples: List[dict]\n    \"\"\"A list of the examples that the prompt template expects.\"\"\"\n    example_prompt: PromptTemplate\n    \"\"\"Prompt template used to format the examples.\"\"\"\n    get_text_length: Callable[[str], int] = _get_length_based\n    \"\"\"Function to measure prompt length. Defaults to word count.\"\"\"\n    max_length: int = 2048\n    \"\"\"Max length for the prompt, beyond which examples are cut.\"\"\"\n    example_text_lengths: List[int] = []  #: :meta private:\n[docs]    def add_example(self, example: Dict[str, str]) -> None:\n        \"\"\"Add new example to list.\"\"\"\n        self.examples.append(example)\n        string_example = self.example_prompt.format(**example)\n        self.example_text_lengths.append(self.get_text_length(string_example))\n    @validator(\"example_text_lengths\", always=True)\n    def calculate_example_text_lengths(cls, v: List[int], values: Dict) -> List[int]:\n        \"\"\"Calculate text lengths if they don't exist.\"\"\"\n        # Check if text lengths were passed in\n        if v:\n            return v\n        # If they were not, calculate them\n        example_prompt = values[\"example_prompt\"]\n        get_text_length = values[\"get_text_length\"]", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/length_based.html"}911{"id": "14a3be0a79ee-1", "text": "get_text_length = values[\"get_text_length\"]\n        string_examples = [example_prompt.format(**eg) for eg in values[\"examples\"]]\n        return [get_text_length(eg) for eg in string_examples]\n[docs]    def select_examples(self, input_variables: Dict[str, str]) -> List[dict]:\n        \"\"\"Select which examples to use based on the input lengths.\"\"\"\n        inputs = \" \".join(input_variables.values())\n        remaining_length = self.max_length - self.get_text_length(inputs)\n        i = 0\n        examples = []\n        while remaining_length > 0 and i < len(self.examples):\n            new_length = remaining_length - self.example_text_lengths[i]\n            if new_length < 0:\n                break\n            else:\n                examples.append(self.examples[i])\n                remaining_length = new_length\n            i += 1\n        return examples\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/prompts/example_selector/length_based.html"}912{"id": "0ce1a28e3706-0", "text": "Source code for langchain.document_loaders.airbyte_json\n\"\"\"Loader that loads local airbyte json files.\"\"\"\nimport json\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utils import stringify_dict\n[docs]class AirbyteJSONLoader(BaseLoader):\n    \"\"\"Loader that loads local airbyte json files.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path. This should start with '/tmp/airbyte_local/'.\"\"\"\n        self.file_path = file_path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        text = \"\"\n        for line in open(self.file_path, \"r\"):\n            data = json.loads(line)[\"_airbyte_data\"]\n            text += stringify_dict(data)\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/airbyte_json.html"}913{"id": "3575be28a5a2-0", "text": "Source code for langchain.document_loaders.diffbot\n\"\"\"Loader that uses Diffbot to load webpages in text format.\"\"\"\nimport logging\nfrom typing import Any, List\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class DiffbotLoader(BaseLoader):\n    \"\"\"Loader that loads Diffbot file json.\"\"\"\n    def __init__(\n        self, api_token: str, urls: List[str], continue_on_failure: bool = True\n    ):\n        \"\"\"Initialize with API token, ids, and key.\"\"\"\n        self.api_token = api_token\n        self.urls = urls\n        self.continue_on_failure = continue_on_failure\n    def _diffbot_api_url(self, diffbot_api: str) -> str:\n        return f\"https://api.diffbot.com/v3/{diffbot_api}\"\n    def _get_diffbot_data(self, url: str) -> Any:\n        \"\"\"Get Diffbot file from Diffbot REST API.\"\"\"\n        # TODO: Add support for other Diffbot APIs\n        diffbot_url = self._diffbot_api_url(\"article\")\n        params = {\n            \"token\": self.api_token,\n            \"url\": url,\n        }\n        response = requests.get(diffbot_url, params=params, timeout=10)\n        # TODO: handle non-ok errors\n        return response.json() if response.ok else {}\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Extract text from Diffbot on all the URLs and return Document instances\"\"\"\n        docs: List[Document] = list()\n        for url in self.urls:\n            try:\n                data = self._get_diffbot_data(url)\n                text = data[\"objects\"][0][\"text\"] if \"objects\" in data else \"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/diffbot.html"}914{"id": "3575be28a5a2-1", "text": "text = data[\"objects\"][0][\"text\"] if \"objects\" in data else \"\"\n                metadata = {\"source\": url}\n                docs.append(Document(page_content=text, metadata=metadata))\n            except Exception as e:\n                if self.continue_on_failure:\n                    logger.error(f\"Error fetching or processing {url}, exception: {e}\")\n                else:\n                    raise e\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/diffbot.html"}915{"id": "c074fe91410c-0", "text": "Source code for langchain.document_loaders.modern_treasury\n\"\"\"Loader that fetches data from Modern Treasury\"\"\"\nimport json\nimport urllib.request\nfrom base64 import b64encode\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utils import get_from_env, stringify_value\nMODERN_TREASURY_ENDPOINTS = {\n    \"payment_orders\": \"https://app.moderntreasury.com/api/payment_orders\",\n    \"expected_payments\": \"https://app.moderntreasury.com/api/expected_payments\",\n    \"returns\": \"https://app.moderntreasury.com/api/returns\",\n    \"incoming_payment_details\": \"https://app.moderntreasury.com/api/\\\nincoming_payment_details\",\n    \"counterparties\": \"https://app.moderntreasury.com/api/counterparties\",\n    \"internal_accounts\": \"https://app.moderntreasury.com/api/internal_accounts\",\n    \"external_accounts\": \"https://app.moderntreasury.com/api/external_accounts\",\n    \"transactions\": \"https://app.moderntreasury.com/api/transactions\",\n    \"ledgers\": \"https://app.moderntreasury.com/api/ledgers\",\n    \"ledger_accounts\": \"https://app.moderntreasury.com/api/ledger_accounts\",\n    \"ledger_transactions\": \"https://app.moderntreasury.com/api/ledger_transactions\",\n    \"events\": \"https://app.moderntreasury.com/api/events\",\n    \"invoices\": \"https://app.moderntreasury.com/api/invoices\",\n}\n[docs]class ModernTreasuryLoader(BaseLoader):\n    def __init__(\n        self,\n        resource: str,\n        organization_id: Optional[str] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/modern_treasury.html"}916{"id": "c074fe91410c-1", "text": "self,\n        resource: str,\n        organization_id: Optional[str] = None,\n        api_key: Optional[str] = None,\n    ) -> None:\n        self.resource = resource\n        organization_id = organization_id or get_from_env(\n            \"organization_id\", \"MODERN_TREASURY_ORGANIZATION_ID\"\n        )\n        api_key = api_key or get_from_env(\"api_key\", \"MODERN_TREASURY_API_KEY\")\n        credentials = f\"{organization_id}:{api_key}\".encode(\"utf-8\")\n        basic_auth_token = b64encode(credentials).decode(\"utf-8\")\n        self.headers = {\"Authorization\": f\"Basic {basic_auth_token}\"}\n    def _make_request(self, url: str) -> List[Document]:\n        request = urllib.request.Request(url, headers=self.headers)\n        with urllib.request.urlopen(request) as response:\n            json_data = json.loads(response.read().decode())\n            text = stringify_value(json_data)\n            metadata = {\"source\": url}\n            return [Document(page_content=text, metadata=metadata)]\n    def _get_resource(self) -> List[Document]:\n        endpoint = MODERN_TREASURY_ENDPOINTS.get(self.resource)\n        if endpoint is None:\n            return []\n        return self._make_request(endpoint)\n[docs]    def load(self) -> List[Document]:\n        return self._get_resource()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/modern_treasury.html"}917{"id": "17454bfe4875-0", "text": "Source code for langchain.document_loaders.bigquery\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class BigQueryLoader(BaseLoader):\n    \"\"\"Loads a query result from BigQuery into a list of documents.\n    Each document represents one row of the result. The `page_content_columns`\n    are written into the `page_content` of the document. The `metadata_columns`\n    are written into the `metadata` of the document. By default, all columns\n    are written into the `page_content` and none into the `metadata`.\n    \"\"\"\n    def __init__(\n        self,\n        query: str,\n        project: Optional[str] = None,\n        page_content_columns: Optional[List[str]] = None,\n        metadata_columns: Optional[List[str]] = None,\n    ):\n        self.query = query\n        self.project = project\n        self.page_content_columns = page_content_columns\n        self.metadata_columns = metadata_columns\n[docs]    def load(self) -> List[Document]:\n        try:\n            from google.cloud import bigquery\n        except ImportError as ex:\n            raise ValueError(\n                \"Could not import google-cloud-bigquery python package. \"\n                \"Please install it with `pip install google-cloud-bigquery`.\"\n            ) from ex\n        bq_client = bigquery.Client(self.project)\n        query_result = bq_client.query(self.query).result()\n        docs: List[Document] = []\n        page_content_columns = self.page_content_columns\n        metadata_columns = self.metadata_columns\n        if page_content_columns is None:\n            page_content_columns = [column.name for column in query_result.schema]\n        if metadata_columns is None:\n            metadata_columns = []\n        for row in query_result:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bigquery.html"}918{"id": "17454bfe4875-1", "text": "metadata_columns = []\n        for row in query_result:\n            page_content = \"\\n\".join(\n                f\"{k}: {v}\" for k, v in row.items() if k in page_content_columns\n            )\n            metadata = {k: v for k, v in row.items() if k in metadata_columns}\n            doc = Document(page_content=page_content, metadata=metadata)\n            docs.append(doc)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bigquery.html"}919{"id": "b4976e43b00a-0", "text": "Source code for langchain.document_loaders.readthedocs\n\"\"\"Loader that loads ReadTheDocs documentation directory dump.\"\"\"\nfrom pathlib import Path\nfrom typing import Any, List, Optional, Tuple, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class ReadTheDocsLoader(BaseLoader):\n    \"\"\"Loader that loads ReadTheDocs documentation directory dump.\"\"\"\n    def __init__(\n        self,\n        path: Union[str, Path],\n        encoding: Optional[str] = None,\n        errors: Optional[str] = None,\n        custom_html_tag: Optional[Tuple[str, dict]] = None,\n        **kwargs: Optional[Any]\n    ):\n        \"\"\"\n        Initialize ReadTheDocsLoader\n        The loader loops over all files under `path` and extract the actual content of\n        the files by retrieving main html tags. Default main html tags include\n        `<main id=\"main-content>`, <`div role=\"main>`, and `<article role=\"main\">`. You\n        can also define your own html tags by passing custom_html_tag, e.g.\n        `(\"div\", \"class=main\")`. The loader iterates html tags with the order of\n        custom html tags (if exists) and default html tags. If any of the tags is not\n        empty, the loop will break and retrieve the content out of that tag.\n        Args:\n            path: The location of pulled readthedocs folder.\n            encoding: The encoding with which to open the documents.\n            errors: Specifies how encoding and decoding errors are to be handled\u2014this\n                cannot be used in binary mode.\n            custom_html_tag: Optional custom html tag to retrieve the content from\n                files.\n        \"\"\"\n        try:\n            from bs4 import BeautifulSoup\n        except ImportError:\n            raise ImportError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/readthedocs.html"}920{"id": "b4976e43b00a-1", "text": "try:\n            from bs4 import BeautifulSoup\n        except ImportError:\n            raise ImportError(\n                \"Could not import python packages. \"\n                \"Please install it with `pip install beautifulsoup4`. \"\n            )\n        try:\n            _ = BeautifulSoup(\n                \"<html><body>Parser builder library test.</body></html>\", **kwargs\n            )\n        except Exception as e:\n            raise ValueError(\"Parsing kwargs do not appear valid\") from e\n        self.file_path = Path(path)\n        self.encoding = encoding\n        self.errors = errors\n        self.custom_html_tag = custom_html_tag\n        self.bs_kwargs = kwargs\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        docs = []\n        for p in self.file_path.rglob(\"*\"):\n            if p.is_dir():\n                continue\n            with open(p, encoding=self.encoding, errors=self.errors) as f:\n                text = self._clean_data(f.read())\n            metadata = {\"source\": str(p)}\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\n    def _clean_data(self, data: str) -> str:\n        from bs4 import BeautifulSoup\n        soup = BeautifulSoup(data, **self.bs_kwargs)\n        # default tags\n        html_tags = [\n            (\"div\", {\"role\": \"main\"}),\n            (\"main\", {\"id\": \"main-content\"}),\n        ]\n        if self.custom_html_tag is not None:\n            html_tags.append(self.custom_html_tag)\n        text = None\n        # reversed order. check the custom one first\n        for tag, attrs in html_tags[::-1]:\n            text = soup.find(tag, attrs)\n            # if found, break\n            if text is not None:\n                break\n        if text is not None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/readthedocs.html"}921{"id": "b4976e43b00a-2", "text": "if text is not None:\n                break\n        if text is not None:\n            text = text.get_text()\n        else:\n            text = \"\"\n        # trim empty lines\n        return \"\\n\".join([t for t in text.split(\"\\n\") if t])\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/readthedocs.html"}922{"id": "3fd1f44a2bb9-0", "text": "Source code for langchain.document_loaders.chatgpt\n\"\"\"Load conversations from ChatGPT data export\"\"\"\nimport datetime\nimport json\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\ndef concatenate_rows(message: dict, title: str) -> str:\n    if not message:\n        return \"\"\n    sender = message[\"author\"][\"role\"] if message[\"author\"] else \"unknown\"\n    text = message[\"content\"][\"parts\"][0]\n    date = datetime.datetime.fromtimestamp(message[\"create_time\"]).strftime(\n        \"%Y-%m-%d %H:%M:%S\"\n    )\n    return f\"{title} - {sender} on {date}: {text}\\n\\n\"\n[docs]class ChatGPTLoader(BaseLoader):\n    \"\"\"Loader that loads conversations from exported ChatGPT data.\"\"\"\n    def __init__(self, log_file: str, num_logs: int = -1):\n        self.log_file = log_file\n        self.num_logs = num_logs\n[docs]    def load(self) -> List[Document]:\n        with open(self.log_file, encoding=\"utf8\") as f:\n            data = json.load(f)[: self.num_logs] if self.num_logs else json.load(f)\n        documents = []\n        for d in data:\n            title = d[\"title\"]\n            messages = d[\"mapping\"]\n            text = \"\".join(\n                [\n                    concatenate_rows(messages[key][\"message\"], title)\n                    for idx, key in enumerate(messages)\n                    if not (\n                        idx == 0\n                        and messages[key][\"message\"][\"author\"][\"role\"] == \"system\"\n                    )\n                ]\n            )\n            metadata = {\"source\": str(self.log_file)}\n            documents.append(Document(page_content=text, metadata=metadata))", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/chatgpt.html"}923{"id": "3fd1f44a2bb9-1", "text": "documents.append(Document(page_content=text, metadata=metadata))\n        return documents\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/chatgpt.html"}924{"id": "2279b13cc068-0", "text": "Source code for langchain.document_loaders.powerpoint\n\"\"\"Loader that loads powerpoint files.\"\"\"\nimport os\nfrom typing import List\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class UnstructuredPowerPointLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load powerpoint files.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.__version__ import __version__ as __unstructured_version__\n        from unstructured.file_utils.filetype import FileType, detect_filetype\n        unstructured_version = tuple(\n            [int(x) for x in __unstructured_version__.split(\".\")]\n        )\n        # NOTE(MthwRobinson) - magic will raise an import error if the libmagic\n        # system dependency isn't installed. If it's not installed, we'll just\n        # check the file extension\n        try:\n            import magic  # noqa: F401\n            is_ppt = detect_filetype(self.file_path) == FileType.PPT\n        except ImportError:\n            _, extension = os.path.splitext(str(self.file_path))\n            is_ppt = extension == \".ppt\"\n        if is_ppt and unstructured_version < (0, 4, 11):\n            raise ValueError(\n                f\"You are on unstructured version {__unstructured_version__}. \"\n                \"Partitioning .ppt files is only supported in unstructured>=0.4.11. \"\n                \"Please upgrade the unstructured package and try again.\"\n            )\n        if is_ppt:\n            from unstructured.partition.ppt import partition_ppt\n            return partition_ppt(filename=self.file_path, **self.unstructured_kwargs)\n        else:\n            from unstructured.partition.pptx import partition_pptx\n            return partition_pptx(filename=self.file_path, **self.unstructured_kwargs)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/powerpoint.html"}925{"id": "2279b13cc068-1", "text": "return partition_pptx(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/powerpoint.html"}926{"id": "3d04f0b23ffd-0", "text": "Source code for langchain.document_loaders.epub\n\"\"\"Loader that loads EPub files.\"\"\"\nfrom typing import List\nfrom langchain.document_loaders.unstructured import (\n    UnstructuredFileLoader,\n    satisfies_min_unstructured_version,\n)\n[docs]class UnstructuredEPubLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load epub files.\"\"\"\n    def _get_elements(self) -> List:\n        min_unstructured_version = \"0.5.4\"\n        if not satisfies_min_unstructured_version(min_unstructured_version):\n            raise ValueError(\n                \"Partitioning epub files is only supported in \"\n                f\"unstructured>={min_unstructured_version}.\"\n            )\n        from unstructured.partition.epub import partition_epub\n        return partition_epub(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/epub.html"}927{"id": "2e54435ae86f-0", "text": "Source code for langchain.document_loaders.python\nimport tokenize\nfrom langchain.document_loaders.text import TextLoader\n[docs]class PythonLoader(TextLoader):\n    \"\"\"\n    Load Python files, respecting any non-default encoding if specified.\n    \"\"\"\n    def __init__(self, file_path: str):\n        with open(file_path, \"rb\") as f:\n            encoding, _ = tokenize.detect_encoding(f.readline)\n        super().__init__(file_path=file_path, encoding=encoding)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/python.html"}928{"id": "c6c9a1681019-0", "text": "Source code for langchain.document_loaders.azure_blob_storage_file\n\"\"\"Loading logic for loading documents from an Azure Blob Storage file.\"\"\"\nimport os\nimport tempfile\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class AzureBlobStorageFileLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from Azure Blob Storage.\"\"\"\n    def __init__(self, conn_str: str, container: str, blob_name: str):\n        \"\"\"Initialize with connection string, container and blob name.\"\"\"\n        self.conn_str = conn_str\n        self.container = container\n        self.blob = blob_name\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            from azure.storage.blob import BlobClient\n        except ImportError as exc:\n            raise ValueError(\n                \"Could not import azure storage blob python package. \"\n                \"Please install it with `pip install azure-storage-blob`.\"\n            ) from exc\n        client = BlobClient.from_connection_string(\n            conn_str=self.conn_str, container_name=self.container, blob_name=self.blob\n        )\n        with tempfile.TemporaryDirectory() as temp_dir:\n            file_path = f\"{temp_dir}/{self.container}/{self.blob}\"\n            os.makedirs(os.path.dirname(file_path), exist_ok=True)\n            with open(f\"{file_path}\", \"wb\") as file:\n                blob_data = client.download_blob()\n                blob_data.readinto(file)\n            loader = UnstructuredFileLoader(file_path)\n            return loader.load()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/azure_blob_storage_file.html"}929{"id": "789a80bdf7ba-0", "text": "Source code for langchain.document_loaders.git\nimport os\nfrom typing import Callable, List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class GitLoader(BaseLoader):\n    \"\"\"Loads files from a Git repository into a list of documents.\n    Repository can be local on disk available at `repo_path`,\n    or remote at `clone_url` that will be cloned to `repo_path`.\n    Currently supports only text files.\n    Each document represents one file in the repository. The `path` points to\n    the local Git repository, and the `branch` specifies the branch to load\n    files from. By default, it loads from the `main` branch.\n    \"\"\"\n    def __init__(\n        self,\n        repo_path: str,\n        clone_url: Optional[str] = None,\n        branch: Optional[str] = \"main\",\n        file_filter: Optional[Callable[[str], bool]] = None,\n    ):\n        self.repo_path = repo_path\n        self.clone_url = clone_url\n        self.branch = branch\n        self.file_filter = file_filter\n[docs]    def load(self) -> List[Document]:\n        try:\n            from git import Blob, Repo  # type: ignore\n        except ImportError as ex:\n            raise ImportError(\n                \"Could not import git python package. \"\n                \"Please install it with `pip install GitPython`.\"\n            ) from ex\n        if not os.path.exists(self.repo_path) and self.clone_url is None:\n            raise ValueError(f\"Path {self.repo_path} does not exist\")\n        elif self.clone_url:\n            repo = Repo.clone_from(self.clone_url, self.repo_path)\n            repo.git.checkout(self.branch)\n        else:\n            repo = Repo(self.repo_path)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/git.html"}930{"id": "789a80bdf7ba-1", "text": "else:\n            repo = Repo(self.repo_path)\n            repo.git.checkout(self.branch)\n        docs: List[Document] = []\n        for item in repo.tree().traverse():\n            if not isinstance(item, Blob):\n                continue\n            file_path = os.path.join(self.repo_path, item.path)\n            ignored_files = repo.ignored([file_path])  # type: ignore\n            if len(ignored_files):\n                continue\n            # uses filter to skip files\n            if self.file_filter and not self.file_filter(file_path):\n                continue\n            rel_file_path = os.path.relpath(file_path, self.repo_path)\n            try:\n                with open(file_path, \"rb\") as f:\n                    content = f.read()\n                    file_type = os.path.splitext(item.name)[1]\n                    # loads only text files\n                    try:\n                        text_content = content.decode(\"utf-8\")\n                    except UnicodeDecodeError:\n                        continue\n                    metadata = {\n                        \"source\": rel_file_path,\n                        \"file_path\": rel_file_path,\n                        \"file_name\": item.name,\n                        \"file_type\": file_type,\n                    }\n                    doc = Document(page_content=text_content, metadata=metadata)\n                    docs.append(doc)\n            except Exception as e:\n                print(f\"Error reading file {file_path}: {e}\")\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/git.html"}931{"id": "c814b73f8f6a-0", "text": "Source code for langchain.document_loaders.text\nimport logging\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.helpers import detect_file_encodings\nlogger = logging.getLogger(__name__)\n[docs]class TextLoader(BaseLoader):\n    \"\"\"Load text files.\n    Args:\n        file_path: Path to the file to load.\n        encoding: File encoding to use. If `None`, the file will be loaded\n        with the default system encoding.\n        autodetect_encoding: Whether to try to autodetect the file encoding\n            if the specified encoding fails.\n    \"\"\"\n    def __init__(\n        self,\n        file_path: str,\n        encoding: Optional[str] = None,\n        autodetect_encoding: bool = False,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        self.encoding = encoding\n        self.autodetect_encoding = autodetect_encoding\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load from file path.\"\"\"\n        text = \"\"\n        try:\n            with open(self.file_path, encoding=self.encoding) as f:\n                text = f.read()\n        except UnicodeDecodeError as e:\n            if self.autodetect_encoding:\n                detected_encodings = detect_file_encodings(self.file_path)\n                for encoding in detected_encodings:\n                    logger.debug(\"Trying encoding: \", encoding.encoding)\n                    try:\n                        with open(self.file_path, encoding=encoding.encoding) as f:\n                            text = f.read()\n                        break\n                    except UnicodeDecodeError:\n                        continue\n            else:\n                raise RuntimeError(f\"Error loading {self.file_path}\") from e\n        except Exception as e:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/text.html"}932{"id": "c814b73f8f6a-1", "text": "except Exception as e:\n            raise RuntimeError(f\"Error loading {self.file_path}\") from e\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/text.html"}933{"id": "89345ca41efa-0", "text": "Source code for langchain.document_loaders.image\n\"\"\"Loader that loads image files.\"\"\"\nfrom typing import List\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class UnstructuredImageLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load image files, such as PNGs and JPGs.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.partition.image import partition_image\n        return partition_image(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/image.html"}934{"id": "aa8bbc1faffc-0", "text": "Source code for langchain.document_loaders.srt\n\"\"\"Loader for .srt (subtitle) files.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class SRTLoader(BaseLoader):\n    \"\"\"Loader for .srt (subtitle) files.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            import pysrt  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"package `pysrt` not found, please install it with `pip install pysrt`\"\n            )\n        self.file_path = file_path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load using pysrt file.\"\"\"\n        import pysrt\n        parsed_info = pysrt.open(self.file_path)\n        text = \" \".join([t.text for t in parsed_info])\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/srt.html"}935{"id": "2612ec6a7fdb-0", "text": "Source code for langchain.document_loaders.url_selenium\n\"\"\"Loader that uses Selenium to load a page, then uses unstructured to load the html.\n\"\"\"\nimport logging\nfrom typing import TYPE_CHECKING, List, Literal, Optional, Union\nif TYPE_CHECKING:\n    from selenium.webdriver import Chrome, Firefox\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class SeleniumURLLoader(BaseLoader):\n    \"\"\"Loader that uses Selenium and to load a page and unstructured to load the html.\n    This is useful for loading pages that require javascript to render.\n    Attributes:\n        urls (List[str]): List of URLs to load.\n        continue_on_failure (bool): If True, continue loading other URLs on failure.\n        browser (str): The browser to use, either 'chrome' or 'firefox'.\n        binary_location (Optional[str]): The location of the browser binary.\n        executable_path (Optional[str]): The path to the browser executable.\n        headless (bool): If True, the browser will run in headless mode.\n        arguments [List[str]]: List of arguments to pass to the browser.\n    \"\"\"\n    def __init__(\n        self,\n        urls: List[str],\n        continue_on_failure: bool = True,\n        browser: Literal[\"chrome\", \"firefox\"] = \"chrome\",\n        binary_location: Optional[str] = None,\n        executable_path: Optional[str] = None,\n        headless: bool = True,\n        arguments: List[str] = [],\n    ):\n        \"\"\"Load a list of URLs using Selenium and unstructured.\"\"\"\n        try:\n            import selenium  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"selenium package not found, please install it with \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url_selenium.html"}936{"id": "2612ec6a7fdb-1", "text": "raise ImportError(\n                \"selenium package not found, please install it with \"\n                \"`pip install selenium`\"\n            )\n        try:\n            import unstructured  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"unstructured package not found, please install it with \"\n                \"`pip install unstructured`\"\n            )\n        self.urls = urls\n        self.continue_on_failure = continue_on_failure\n        self.browser = browser\n        self.binary_location = binary_location\n        self.executable_path = executable_path\n        self.headless = headless\n        self.arguments = arguments\n    def _get_driver(self) -> Union[\"Chrome\", \"Firefox\"]:\n        \"\"\"Create and return a WebDriver instance based on the specified browser.\n        Raises:\n            ValueError: If an invalid browser is specified.\n        Returns:\n            Union[Chrome, Firefox]: A WebDriver instance for the specified browser.\n        \"\"\"\n        if self.browser.lower() == \"chrome\":\n            from selenium.webdriver import Chrome\n            from selenium.webdriver.chrome.options import Options as ChromeOptions\n            chrome_options = ChromeOptions()\n            for arg in self.arguments:\n                chrome_options.add_argument(arg)\n            if self.headless:\n                chrome_options.add_argument(\"--headless\")\n                chrome_options.add_argument(\"--no-sandbox\")\n            if self.binary_location is not None:\n                chrome_options.binary_location = self.binary_location\n            if self.executable_path is None:\n                return Chrome(options=chrome_options)\n            return Chrome(executable_path=self.executable_path, options=chrome_options)\n        elif self.browser.lower() == \"firefox\":\n            from selenium.webdriver import Firefox\n            from selenium.webdriver.firefox.options import Options as FirefoxOptions\n            firefox_options = FirefoxOptions()\n            for arg in self.arguments:\n                firefox_options.add_argument(arg)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url_selenium.html"}937{"id": "2612ec6a7fdb-2", "text": "for arg in self.arguments:\n                firefox_options.add_argument(arg)\n            if self.headless:\n                firefox_options.add_argument(\"--headless\")\n            if self.binary_location is not None:\n                firefox_options.binary_location = self.binary_location\n            if self.executable_path is None:\n                return Firefox(options=firefox_options)\n            return Firefox(\n                executable_path=self.executable_path, options=firefox_options\n            )\n        else:\n            raise ValueError(\"Invalid browser specified. Use 'chrome' or 'firefox'.\")\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load the specified URLs using Selenium and create Document instances.\n        Returns:\n            List[Document]: A list of Document instances with loaded content.\n        \"\"\"\n        from unstructured.partition.html import partition_html\n        docs: List[Document] = list()\n        driver = self._get_driver()\n        for url in self.urls:\n            try:\n                driver.get(url)\n                page_content = driver.page_source\n                elements = partition_html(text=page_content)\n                text = \"\\n\\n\".join([str(el) for el in elements])\n                metadata = {\"source\": url}\n                docs.append(Document(page_content=text, metadata=metadata))\n            except Exception as e:\n                if self.continue_on_failure:\n                    logger.error(f\"Error fetching or processing {url}, exception: {e}\")\n                else:\n                    raise e\n        driver.quit()\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url_selenium.html"}938{"id": "0bd90c5f763e-0", "text": "Source code for langchain.document_loaders.notion\n\"\"\"Loader that loads Notion directory dump.\"\"\"\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class NotionDirectoryLoader(BaseLoader):\n    \"\"\"Loader that loads Notion directory dump.\"\"\"\n    def __init__(self, path: str):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        ps = list(Path(self.file_path).glob(\"**/*.md\"))\n        docs = []\n        for p in ps:\n            with open(p) as f:\n                text = f.read()\n            metadata = {\"source\": str(p)}\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notion.html"}939{"id": "ae90d2a10d24-0", "text": "Source code for langchain.document_loaders.evernote\n\"\"\"Load documents from Evernote.\nhttps://gist.github.com/foxmask/7b29c43a161e001ff04afdb2f181e31c\n\"\"\"\nimport hashlib\nimport logging\nfrom base64 import b64decode\nfrom time import strptime\nfrom typing import Any, Dict, Iterator, List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class EverNoteLoader(BaseLoader):\n    \"\"\"EverNote Loader.\n    Loads an EverNote notebook export file e.g. my_notebook.enex into Documents.\n    Instructions on producing this file can be found at\n    https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML\n    Currently only the plain text in the note is extracted and stored as the contents\n    of the Document, any non content metadata (e.g. 'author', 'created', 'updated' etc.\n    but not 'content-raw' or 'resource') tags on the note will be extracted and stored\n    as metadata on the Document.\n    Args:\n        file_path (str): The path to the notebook export with a .enex extension\n        load_single_document (bool): Whether or not to concatenate the content of all\n            notes into a single long Document.\n        If this is set to True (default) then the only metadata on the document will be\n            the 'source' which contains the file name of the export.\n    \"\"\"  # noqa: E501\n    def __init__(self, file_path: str, load_single_document: bool = True):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        self.load_single_document = load_single_document", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/evernote.html"}940{"id": "ae90d2a10d24-1", "text": "self.file_path = file_path\n        self.load_single_document = load_single_document\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents from EverNote export file.\"\"\"\n        documents = [\n            Document(\n                page_content=note[\"content\"],\n                metadata={\n                    **{\n                        key: value\n                        for key, value in note.items()\n                        if key not in [\"content\", \"content-raw\", \"resource\"]\n                    },\n                    **{\"source\": self.file_path},\n                },\n            )\n            for note in self._parse_note_xml(self.file_path)\n            if note.get(\"content\") is not None\n        ]\n        if not self.load_single_document:\n            return documents\n        return [\n            Document(\n                page_content=\"\".join([document.page_content for document in documents]),\n                metadata={\"source\": self.file_path},\n            )\n        ]\n    @staticmethod\n    def _parse_content(content: str) -> str:\n        try:\n            import html2text\n            return html2text.html2text(content).strip()\n        except ImportError as e:\n            logging.error(\n                \"Could not import `html2text`. Although it is not a required package \"\n                \"to use Langchain, using the EverNote loader requires `html2text`. \"\n                \"Please install `html2text` via `pip install html2text` and try again.\"\n            )\n            raise e\n    @staticmethod\n    def _parse_resource(resource: list) -> dict:\n        rsc_dict: Dict[str, Any] = {}\n        for elem in resource:\n            if elem.tag == \"data\":\n                # Sometimes elem.text is None\n                rsc_dict[elem.tag] = b64decode(elem.text) if elem.text else b\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/evernote.html"}941{"id": "ae90d2a10d24-2", "text": "rsc_dict[\"hash\"] = hashlib.md5(rsc_dict[elem.tag]).hexdigest()\n            else:\n                rsc_dict[elem.tag] = elem.text\n        return rsc_dict\n    @staticmethod\n    def _parse_note(note: List, prefix: Optional[str] = None) -> dict:\n        note_dict: Dict[str, Any] = {}\n        resources = []\n        def add_prefix(element_tag: str) -> str:\n            if prefix is None:\n                return element_tag\n            return f\"{prefix}.{element_tag}\"\n        for elem in note:\n            if elem.tag == \"content\":\n                note_dict[elem.tag] = EverNoteLoader._parse_content(elem.text)\n                # A copy of original content\n                note_dict[\"content-raw\"] = elem.text\n            elif elem.tag == \"resource\":\n                resources.append(EverNoteLoader._parse_resource(elem))\n            elif elem.tag == \"created\" or elem.tag == \"updated\":\n                note_dict[elem.tag] = strptime(elem.text, \"%Y%m%dT%H%M%SZ\")\n            elif elem.tag == \"note-attributes\":\n                additional_attributes = EverNoteLoader._parse_note(\n                    elem, elem.tag\n                )  # Recursively enter the note-attributes tag\n                note_dict.update(additional_attributes)\n            else:\n                note_dict[elem.tag] = elem.text\n        if len(resources) > 0:\n            note_dict[\"resource\"] = resources\n        return {add_prefix(key): value for key, value in note_dict.items()}\n    @staticmethod\n    def _parse_note_xml(xml_file: str) -> Iterator[Dict[str, Any]]:\n        \"\"\"Parse Evernote xml.\"\"\"\n        # Without huge_tree set to True, parser may complain about huge text node", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/evernote.html"}942{"id": "ae90d2a10d24-3", "text": "# Without huge_tree set to True, parser may complain about huge text node\n        # Try to recover, because there may be \"&nbsp;\", which will cause\n        # \"XMLSyntaxError: Entity 'nbsp' not defined\"\n        try:\n            from lxml import etree\n        except ImportError as e:\n            logging.error(\n                \"Could not import `lxml`. Although it is not a required package to use \"\n                \"Langchain, using the EverNote loader requires `lxml`. Please install \"\n                \"`lxml` via `pip install lxml` and try again.\"\n            )\n            raise e\n        context = etree.iterparse(\n            xml_file, encoding=\"utf-8\", strip_cdata=False, huge_tree=True, recover=True\n        )\n        for action, elem in context:\n            if elem.tag == \"note\":\n                yield EverNoteLoader._parse_note(elem)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/evernote.html"}943{"id": "5a51b17b78cf-0", "text": "Source code for langchain.document_loaders.reddit\n\"\"\"Reddit document loader.\"\"\"\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, Iterable, List, Optional, Sequence\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nif TYPE_CHECKING:\n    import praw\ndef _dependable_praw_import() -> praw:\n    try:\n        import praw\n    except ImportError:\n        raise ValueError(\n            \"praw package not found, please install it with `pip install praw`\"\n        )\n    return praw\n[docs]class RedditPostsLoader(BaseLoader):\n    \"\"\"Reddit posts loader.\n    Read posts on a subreddit.\n    First you need to go to\n    https://www.reddit.com/prefs/apps/\n    and create your application\n    \"\"\"\n    def __init__(\n        self,\n        client_id: str,\n        client_secret: str,\n        user_agent: str,\n        search_queries: Sequence[str],\n        mode: str,\n        categories: Sequence[str] = [\"new\"],\n        number_posts: Optional[int] = 10,\n    ):\n        self.client_id = client_id\n        self.client_secret = client_secret\n        self.user_agent = user_agent\n        self.search_queries = search_queries\n        self.mode = mode\n        self.categories = categories\n        self.number_posts = number_posts\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load reddits.\"\"\"\n        praw = _dependable_praw_import()\n        reddit = praw.Reddit(\n            client_id=self.client_id,\n            client_secret=self.client_secret,\n            user_agent=self.user_agent,\n        )\n        results: List[Document] = []\n        if self.mode == \"subreddit\":\n            for search_query in self.search_queries:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/reddit.html"}944{"id": "5a51b17b78cf-1", "text": "if self.mode == \"subreddit\":\n            for search_query in self.search_queries:\n                for category in self.categories:\n                    docs = self._subreddit_posts_loader(\n                        search_query=search_query, category=category, reddit=reddit\n                    )\n                    results.extend(docs)\n        elif self.mode == \"username\":\n            for search_query in self.search_queries:\n                for category in self.categories:\n                    docs = self._user_posts_loader(\n                        search_query=search_query, category=category, reddit=reddit\n                    )\n                    results.extend(docs)\n        else:\n            raise ValueError(\n                \"mode not correct, please enter 'username' or 'subreddit' as mode\"\n            )\n        return results\n    def _subreddit_posts_loader(\n        self, search_query: str, category: str, reddit: praw.reddit.Reddit\n    ) -> Iterable[Document]:\n        subreddit = reddit.subreddit(search_query)\n        method = getattr(subreddit, category)\n        cat_posts = method(limit=self.number_posts)\n        \"\"\"Format reddit posts into a string.\"\"\"\n        for post in cat_posts:\n            metadata = {\n                \"post_subreddit\": post.subreddit_name_prefixed,\n                \"post_category\": category,\n                \"post_title\": post.title,\n                \"post_score\": post.score,\n                \"post_id\": post.id,\n                \"post_url\": post.url,\n                \"post_author\": post.author,\n            }\n            yield Document(\n                page_content=post.selftext,\n                metadata=metadata,\n            )\n    def _user_posts_loader(\n        self, search_query: str, category: str, reddit: praw.reddit.Reddit\n    ) -> Iterable[Document]:\n        user = reddit.redditor(search_query)\n        method = getattr(user.submissions, category)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/reddit.html"}945{"id": "5a51b17b78cf-2", "text": "method = getattr(user.submissions, category)\n        cat_posts = method(limit=self.number_posts)\n        \"\"\"Format reddit posts into a string.\"\"\"\n        for post in cat_posts:\n            metadata = {\n                \"post_subreddit\": post.subreddit_name_prefixed,\n                \"post_category\": category,\n                \"post_title\": post.title,\n                \"post_score\": post.score,\n                \"post_id\": post.id,\n                \"post_url\": post.url,\n                \"post_author\": post.author,\n            }\n            yield Document(\n                page_content=post.selftext,\n                metadata=metadata,\n            )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/reddit.html"}946{"id": "0c0a425cd978-0", "text": "Source code for langchain.document_loaders.url_playwright\n\"\"\"Loader that uses Playwright to load a page, then uses unstructured to load the html.\n\"\"\"\nimport logging\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class PlaywrightURLLoader(BaseLoader):\n    \"\"\"Loader that uses Playwright and to load a page and unstructured to load the html.\n    This is useful for loading pages that require javascript to render.\n    Attributes:\n        urls (List[str]): List of URLs to load.\n        continue_on_failure (bool): If True, continue loading other URLs on failure.\n        headless (bool): If True, the browser will run in headless mode.\n    \"\"\"\n    def __init__(\n        self,\n        urls: List[str],\n        continue_on_failure: bool = True,\n        headless: bool = True,\n        remove_selectors: Optional[List[str]] = None,\n    ):\n        \"\"\"Load a list of URLs using Playwright and unstructured.\"\"\"\n        try:\n            import playwright  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"playwright package not found, please install it with \"\n                \"`pip install playwright`\"\n            )\n        try:\n            import unstructured  # noqa:F401\n        except ImportError:\n            raise ValueError(\n                \"unstructured package not found, please install it with \"\n                \"`pip install unstructured`\"\n            )\n        self.urls = urls\n        self.continue_on_failure = continue_on_failure\n        self.headless = headless\n        self.remove_selectors = remove_selectors\n[docs]    def load(self) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url_playwright.html"}947{"id": "0c0a425cd978-1", "text": "[docs]    def load(self) -> List[Document]:\n        \"\"\"Load the specified URLs using Playwright and create Document instances.\n        Returns:\n            List[Document]: A list of Document instances with loaded content.\n        \"\"\"\n        from playwright.sync_api import sync_playwright\n        from unstructured.partition.html import partition_html\n        docs: List[Document] = list()\n        with sync_playwright() as p:\n            browser = p.chromium.launch(headless=self.headless)\n            for url in self.urls:\n                try:\n                    page = browser.new_page()\n                    page.goto(url)\n                    for selector in self.remove_selectors or []:\n                        elements = page.locator(selector).all()\n                        for element in elements:\n                            if element.is_visible():\n                                element.evaluate(\"element => element.remove()\")\n                    page_source = page.content()\n                    elements = partition_html(text=page_source)\n                    text = \"\\n\\n\".join([str(el) for el in elements])\n                    metadata = {\"source\": url}\n                    docs.append(Document(page_content=text, metadata=metadata))\n                except Exception as e:\n                    if self.continue_on_failure:\n                        logger.error(\n                            f\"Error fetching or processing {url}, exception: {e}\"\n                        )\n                    else:\n                        raise e\n            browser.close()\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url_playwright.html"}948{"id": "bd65e097d569-0", "text": "Source code for langchain.document_loaders.mastodon\n\"\"\"Mastodon document loader.\"\"\"\nfrom __future__ import annotations\nimport os\nfrom typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Sequence\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nif TYPE_CHECKING:\n    import mastodon\ndef _dependable_mastodon_import() -> mastodon:\n    try:\n        import mastodon\n    except ImportError:\n        raise ValueError(\n            \"Mastodon.py package not found, \"\n            \"please install it with `pip install Mastodon.py`\"\n        )\n    return mastodon\n[docs]class MastodonTootsLoader(BaseLoader):\n    \"\"\"Mastodon toots loader.\"\"\"\n    def __init__(\n        self,\n        mastodon_accounts: Sequence[str],\n        number_toots: Optional[int] = 100,\n        exclude_replies: bool = False,\n        access_token: Optional[str] = None,\n        api_base_url: str = \"https://mastodon.social\",\n    ):\n        \"\"\"Instantiate Mastodon toots loader.\n        Args:\n            mastodon_accounts: The list of Mastodon accounts to query.\n            number_toots: How many toots to pull for each account.\n            exclude_replies: Whether to exclude reply toots from the load.\n            access_token: An access token if toots are loaded as a Mastodon app. Can\n                also be specified via the environment variables \"MASTODON_ACCESS_TOKEN\".\n            api_base_url: A Mastodon API base URL to talk to, if not using the default.\n        \"\"\"\n        mastodon = _dependable_mastodon_import()\n        access_token = access_token or os.environ.get(\"MASTODON_ACCESS_TOKEN\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/mastodon.html"}949{"id": "bd65e097d569-1", "text": "access_token = access_token or os.environ.get(\"MASTODON_ACCESS_TOKEN\")\n        self.api = mastodon.Mastodon(\n            access_token=access_token, api_base_url=api_base_url\n        )\n        self.mastodon_accounts = mastodon_accounts\n        self.number_toots = number_toots\n        self.exclude_replies = exclude_replies\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load toots into documents.\"\"\"\n        results: List[Document] = []\n        for account in self.mastodon_accounts:\n            user = self.api.account_lookup(account)\n            toots = self.api.account_statuses(\n                user.id,\n                only_media=False,\n                pinned=False,\n                exclude_replies=self.exclude_replies,\n                exclude_reblogs=True,\n                limit=self.number_toots,\n            )\n            docs = self._format_toots(toots, user)\n            results.extend(docs)\n        return results\n    def _format_toots(\n        self, toots: List[Dict[str, Any]], user_info: dict\n    ) -> Iterable[Document]:\n        \"\"\"Format toots into documents.\n        Adding user info, and selected toot fields into the metadata.\n        \"\"\"\n        for toot in toots:\n            metadata = {\n                \"created_at\": toot[\"created_at\"],\n                \"user_info\": user_info,\n                \"is_reply\": toot[\"in_reply_to_id\"] is not None,\n            }\n            yield Document(\n                page_content=toot[\"content\"],\n                metadata=metadata,\n            )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/mastodon.html"}950{"id": "1dd99e8124a6-0", "text": "Source code for langchain.document_loaders.twitter\n\"\"\"Twitter document loader.\"\"\"\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Sequence, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nif TYPE_CHECKING:\n    import tweepy\n    from tweepy import OAuth2BearerHandler, OAuthHandler\ndef _dependable_tweepy_import() -> tweepy:\n    try:\n        import tweepy\n    except ImportError:\n        raise ImportError(\n            \"tweepy package not found, please install it with `pip install tweepy`\"\n        )\n    return tweepy\n[docs]class TwitterTweetLoader(BaseLoader):\n    \"\"\"Twitter tweets loader.\n    Read tweets of user twitter handle.\n    First you need to go to\n    `https://developer.twitter.com/en/docs/twitter-api\n    /getting-started/getting-access-to-the-twitter-api`\n    to get your token. And create a v2 version of the app.\n    \"\"\"\n    def __init__(\n        self,\n        auth_handler: Union[OAuthHandler, OAuth2BearerHandler],\n        twitter_users: Sequence[str],\n        number_tweets: Optional[int] = 100,\n    ):\n        self.auth = auth_handler\n        self.twitter_users = twitter_users\n        self.number_tweets = number_tweets\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load tweets.\"\"\"\n        tweepy = _dependable_tweepy_import()\n        api = tweepy.API(self.auth, parser=tweepy.parsers.JSONParser())\n        results: List[Document] = []\n        for username in self.twitter_users:\n            tweets = api.user_timeline(screen_name=username, count=self.number_tweets)\n            user = api.get_user(screen_name=username)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/twitter.html"}951{"id": "1dd99e8124a6-1", "text": "user = api.get_user(screen_name=username)\n            docs = self._format_tweets(tweets, user)\n            results.extend(docs)\n        return results\n    def _format_tweets(\n        self, tweets: List[Dict[str, Any]], user_info: dict\n    ) -> Iterable[Document]:\n        \"\"\"Format tweets into a string.\"\"\"\n        for tweet in tweets:\n            metadata = {\n                \"created_at\": tweet[\"created_at\"],\n                \"user_info\": user_info,\n            }\n            yield Document(\n                page_content=tweet[\"text\"],\n                metadata=metadata,\n            )\n[docs]    @classmethod\n    def from_bearer_token(\n        cls,\n        oauth2_bearer_token: str,\n        twitter_users: Sequence[str],\n        number_tweets: Optional[int] = 100,\n    ) -> TwitterTweetLoader:\n        \"\"\"Create a TwitterTweetLoader from OAuth2 bearer token.\"\"\"\n        tweepy = _dependable_tweepy_import()\n        auth = tweepy.OAuth2BearerHandler(oauth2_bearer_token)\n        return cls(\n            auth_handler=auth,\n            twitter_users=twitter_users,\n            number_tweets=number_tweets,\n        )\n[docs]    @classmethod\n    def from_secrets(\n        cls,\n        access_token: str,\n        access_token_secret: str,\n        consumer_key: str,\n        consumer_secret: str,\n        twitter_users: Sequence[str],\n        number_tweets: Optional[int] = 100,\n    ) -> TwitterTweetLoader:\n        \"\"\"Create a TwitterTweetLoader from access tokens and secrets.\"\"\"\n        tweepy = _dependable_tweepy_import()\n        auth = tweepy.OAuthHandler(\n            access_token=access_token,\n            access_token_secret=access_token_secret,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/twitter.html"}952{"id": "1dd99e8124a6-2", "text": "access_token=access_token,\n            access_token_secret=access_token_secret,\n            consumer_key=consumer_key,\n            consumer_secret=consumer_secret,\n        )\n        return cls(\n            auth_handler=auth,\n            twitter_users=twitter_users,\n            number_tweets=number_tweets,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/twitter.html"}953{"id": "19c89a759be6-0", "text": "Source code for langchain.document_loaders.gcs_directory\n\"\"\"Loading logic for loading documents from an GCS directory.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.gcs_file import GCSFileLoader\n[docs]class GCSDirectoryLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from GCS.\"\"\"\n    def __init__(self, project_name: str, bucket: str, prefix: str = \"\"):\n        \"\"\"Initialize with bucket and key name.\"\"\"\n        self.project_name = project_name\n        self.bucket = bucket\n        self.prefix = prefix\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            from google.cloud import storage\n        except ImportError:\n            raise ValueError(\n                \"Could not import google-cloud-storage python package. \"\n                \"Please install it with `pip install google-cloud-storage`.\"\n            )\n        client = storage.Client(project=self.project_name)\n        docs = []\n        for blob in client.list_blobs(self.bucket, prefix=self.prefix):\n            # we shall just skip directories since GCSFileLoader creates\n            # intermediate directories on the fly\n            if blob.name.endswith(\"/\"):\n                continue\n            loader = GCSFileLoader(self.project_name, self.bucket, blob.name)\n            docs.extend(loader.load())\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/gcs_directory.html"}954{"id": "7abd6c001e4e-0", "text": "Source code for langchain.document_loaders.html\n\"\"\"Loader that uses unstructured to load HTML files.\"\"\"\nfrom typing import List\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class UnstructuredHTMLLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load HTML files.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.partition.html import partition_html\n        return partition_html(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/html.html"}955{"id": "877eef4d6824-0", "text": "Source code for langchain.document_loaders.confluence\n\"\"\"Load Data from a Confluence Space\"\"\"\nimport logging\nfrom io import BytesIO\nfrom typing import Any, Callable, List, Optional, Union\nfrom tenacity import (\n    before_sleep_log,\n    retry,\n    stop_after_attempt,\n    wait_exponential,\n)\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class ConfluenceLoader(BaseLoader):\n    \"\"\"\n    Load Confluence pages. Port of https://llamahub.ai/l/confluence\n    This currently supports both username/api_key and Oauth2 login.\n    Specify a list page_ids and/or space_key to load in the corresponding pages into\n    Document objects, if both are specified the union of both sets will be returned.\n    You can also specify a boolean `include_attachments` to include attachments, this\n    is set to False by default, if set to True all attachments will be downloaded and\n    ConfluenceReader will extract the text from the attachments and add it to the\n    Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG,\n    SVG, Word and Excel.\n    Hint: space_key and page_id can both be found in the URL of a page in Confluence\n    - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>\n    Example:\n        .. code-block:: python\n            from langchain.document_loaders import ConfluenceLoader\n            loader = ConfluenceLoader(\n                url=\"https://yoursite.atlassian.com/wiki\",\n                username=\"me\",\n                api_key=\"12345\"\n            )\n            documents = loader.load(space_key=\"SPACE\",limit=50)\n    :param url: _description_", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}956{"id": "877eef4d6824-1", "text": ":param url: _description_\n    :type url: str\n    :param api_key: _description_, defaults to None\n    :type api_key: str, optional\n    :param username: _description_, defaults to None\n    :type username: str, optional\n    :param oauth2: _description_, defaults to {}\n    :type oauth2: dict, optional\n    :param cloud: _description_, defaults to True\n    :type cloud: bool, optional\n    :param number_of_retries: How many times to retry, defaults to 3\n    :type number_of_retries: Optional[int], optional\n    :param min_retry_seconds: defaults to 2\n    :type min_retry_seconds: Optional[int], optional\n    :param max_retry_seconds:  defaults to 10\n    :type max_retry_seconds: Optional[int], optional\n    :param confluence_kwargs: additional kwargs to initialize confluence with\n    :type confluence_kwargs: dict, optional\n    :raises ValueError: Errors while validating input\n    :raises ImportError: Required dependencies not installed.\n    \"\"\"\n    def __init__(\n        self,\n        url: str,\n        api_key: Optional[str] = None,\n        username: Optional[str] = None,\n        oauth2: Optional[dict] = None,\n        cloud: Optional[bool] = True,\n        number_of_retries: Optional[int] = 3,\n        min_retry_seconds: Optional[int] = 2,\n        max_retry_seconds: Optional[int] = 10,\n        confluence_kwargs: Optional[dict] = None,\n    ):\n        confluence_kwargs = confluence_kwargs or {}\n        errors = ConfluenceLoader.validate_init_args(url, api_key, username, oauth2)\n        if errors:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}957{"id": "877eef4d6824-2", "text": "if errors:\n            raise ValueError(f\"Error(s) while validating input: {errors}\")\n        self.base_url = url\n        self.number_of_retries = number_of_retries\n        self.min_retry_seconds = min_retry_seconds\n        self.max_retry_seconds = max_retry_seconds\n        try:\n            from atlassian import Confluence  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`atlassian` package not found, please run \"\n                \"`pip install atlassian-python-api`\"\n            )\n        if oauth2:\n            self.confluence = Confluence(\n                url=url, oauth2=oauth2, cloud=cloud, **confluence_kwargs\n            )\n        else:\n            self.confluence = Confluence(\n                url=url,\n                username=username,\n                password=api_key,\n                cloud=cloud,\n                **confluence_kwargs,\n            )\n[docs]    @staticmethod\n    def validate_init_args(\n        url: Optional[str] = None,\n        api_key: Optional[str] = None,\n        username: Optional[str] = None,\n        oauth2: Optional[dict] = None,\n    ) -> Union[List, None]:\n        \"\"\"Validates proper combinations of init arguments\"\"\"\n        errors = []\n        if url is None:\n            errors.append(\"Must provide `base_url`\")\n        if (api_key and not username) or (username and not api_key):\n            errors.append(\n                \"If one of `api_key` or `username` is provided, \"\n                \"the other must be as well.\"\n            )\n        if (api_key or username) and oauth2:\n            errors.append(\n                \"Cannot provide a value for `api_key` and/or \"\n                \"`username` and provide a value for `oauth2`\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}958{"id": "877eef4d6824-3", "text": "\"`username` and provide a value for `oauth2`\"\n            )\n        if oauth2 and oauth2.keys() != [\n            \"access_token\",\n            \"access_token_secret\",\n            \"consumer_key\",\n            \"key_cert\",\n        ]:\n            errors.append(\n                \"You have either ommited require keys or added extra \"\n                \"keys to the oauth2 dictionary. key values should be \"\n                \"`['access_token', 'access_token_secret', 'consumer_key', 'key_cert']`\"\n            )\n        if errors:\n            return errors\n        return None\n[docs]    def load(\n        self,\n        space_key: Optional[str] = None,\n        page_ids: Optional[List[str]] = None,\n        label: Optional[str] = None,\n        cql: Optional[str] = None,\n        include_restricted_content: bool = False,\n        include_archived_content: bool = False,\n        include_attachments: bool = False,\n        include_comments: bool = False,\n        limit: Optional[int] = 50,\n        max_pages: Optional[int] = 1000,\n    ) -> List[Document]:\n        \"\"\"\n        :param space_key: Space key retrieved from a confluence URL, defaults to None\n        :type space_key: Optional[str], optional\n        :param page_ids: List of specific page IDs to load, defaults to None\n        :type page_ids: Optional[List[str]], optional\n        :param label: Get all pages with this label, defaults to None\n        :type label: Optional[str], optional\n        :param cql: CQL Expression, defaults to None\n        :type cql: Optional[str], optional\n        :param include_restricted_content: defaults to False\n        :type include_restricted_content: bool, optional", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}959{"id": "877eef4d6824-4", "text": ":type include_restricted_content: bool, optional\n        :param include_archived_content: Whether to include archived content,\n                                         defaults to False\n        :type include_archived_content: bool, optional\n        :param include_attachments: defaults to False\n        :type include_attachments: bool, optional\n        :param include_comments: defaults to False\n        :type include_comments: bool, optional\n        :param limit: Maximum number of pages to retrieve per request, defaults to 50\n        :type limit: int, optional\n        :param max_pages: Maximum number of pages to retrieve in total, defaults 1000\n        :type max_pages: int, optional\n        :raises ValueError: _description_\n        :raises ImportError: _description_\n        :return: _description_\n        :rtype: List[Document]\n        \"\"\"\n        if not space_key and not page_ids and not label and not cql:\n            raise ValueError(\n                \"Must specify at least one among `space_key`, `page_ids`, \"\n                \"`label`, `cql` parameters.\"\n            )\n        docs = []\n        if space_key:\n            pages = self.paginate_request(\n                self.confluence.get_all_pages_from_space,\n                space=space_key,\n                limit=limit,\n                max_pages=max_pages,\n                status=\"any\" if include_archived_content else \"current\",\n                expand=\"body.storage.value\",\n            )\n            docs += self.process_pages(\n                pages, include_restricted_content, include_attachments, include_comments\n            )\n        if label:\n            pages = self.paginate_request(\n                self.confluence.get_all_pages_by_label,\n                label=label,\n                limit=limit,\n                max_pages=max_pages,\n            )\n            ids_by_label = [page[\"id\"] for page in pages]", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}960{"id": "877eef4d6824-5", "text": ")\n            ids_by_label = [page[\"id\"] for page in pages]\n            if page_ids:\n                page_ids = list(set(page_ids + ids_by_label))\n            else:\n                page_ids = list(set(ids_by_label))\n        if cql:\n            pages = self.paginate_request(\n                self.confluence.cql,\n                cql=cql,\n                limit=limit,\n                max_pages=max_pages,\n                include_archived_spaces=include_archived_content,\n                expand=\"body.storage.value\",\n            )\n            docs += self.process_pages(\n                pages, include_restricted_content, include_attachments, include_comments\n            )\n        if page_ids:\n            for page_id in page_ids:\n                get_page = retry(\n                    reraise=True,\n                    stop=stop_after_attempt(\n                        self.number_of_retries  # type: ignore[arg-type]\n                    ),\n                    wait=wait_exponential(\n                        multiplier=1,  # type: ignore[arg-type]\n                        min=self.min_retry_seconds,  # type: ignore[arg-type]\n                        max=self.max_retry_seconds,  # type: ignore[arg-type]\n                    ),\n                    before_sleep=before_sleep_log(logger, logging.WARNING),\n                )(self.confluence.get_page_by_id)\n                page = get_page(page_id=page_id, expand=\"body.storage.value\")\n                if not include_restricted_content and not self.is_public_page(page):\n                    continue\n                doc = self.process_page(page, include_attachments, include_comments)\n                docs.append(doc)\n        return docs\n[docs]    def paginate_request(self, retrieval_method: Callable, **kwargs: Any) -> List:\n        \"\"\"Paginate the various methods to retrieve groups of pages.\n        Unfortunately, due to page size, sometimes the Confluence API", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}961{"id": "877eef4d6824-6", "text": "Unfortunately, due to page size, sometimes the Confluence API\n        doesn't match the limit value. If `limit` is  >100 confluence\n        seems to cap the response to 100. Also, due to the Atlassian Python\n        package, we don't get the \"next\" values from the \"_links\" key because\n        they only return the value from the results key. So here, the pagination\n        starts from 0 and goes until the max_pages, getting the `limit` number\n        of pages with each request. We have to manually check if there\n        are more docs based on the length of the returned list of pages, rather than\n        just checking for the presence of a `next` key in the response like this page\n        would have you do:\n        https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/\n        :param retrieval_method: Function used to retrieve docs\n        :type retrieval_method: callable\n        :return: List of documents\n        :rtype: List\n        \"\"\"\n        max_pages = kwargs.pop(\"max_pages\")\n        docs: List[dict] = []\n        while len(docs) < max_pages:\n            get_pages = retry(\n                reraise=True,\n                stop=stop_after_attempt(\n                    self.number_of_retries  # type: ignore[arg-type]\n                ),\n                wait=wait_exponential(\n                    multiplier=1,\n                    min=self.min_retry_seconds,  # type: ignore[arg-type]\n                    max=self.max_retry_seconds,  # type: ignore[arg-type]\n                ),\n                before_sleep=before_sleep_log(logger, logging.WARNING),\n            )(retrieval_method)\n            batch = get_pages(**kwargs, start=len(docs))\n            if not batch:\n                break\n            docs.extend(batch)\n        return docs[:max_pages]", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}962{"id": "877eef4d6824-7", "text": "break\n            docs.extend(batch)\n        return docs[:max_pages]\n[docs]    def is_public_page(self, page: dict) -> bool:\n        \"\"\"Check if a page is publicly accessible.\"\"\"\n        restrictions = self.confluence.get_all_restrictions_for_content(page[\"id\"])\n        return (\n            page[\"status\"] == \"current\"\n            and not restrictions[\"read\"][\"restrictions\"][\"user\"][\"results\"]\n            and not restrictions[\"read\"][\"restrictions\"][\"group\"][\"results\"]\n        )\n[docs]    def process_pages(\n        self,\n        pages: List[dict],\n        include_restricted_content: bool,\n        include_attachments: bool,\n        include_comments: bool,\n    ) -> List[Document]:\n        \"\"\"Process a list of pages into a list of documents.\"\"\"\n        docs = []\n        for page in pages:\n            if not include_restricted_content and not self.is_public_page(page):\n                continue\n            doc = self.process_page(page, include_attachments, include_comments)\n            docs.append(doc)\n        return docs\n[docs]    def process_page(\n        self,\n        page: dict,\n        include_attachments: bool,\n        include_comments: bool,\n    ) -> Document:\n        try:\n            from bs4 import BeautifulSoup  # type: ignore\n        except ImportError:\n            raise ImportError(\n                \"`beautifulsoup4` package not found, please run \"\n                \"`pip install beautifulsoup4`\"\n            )\n        if include_attachments:\n            attachment_texts = self.process_attachment(page[\"id\"])\n        else:\n            attachment_texts = []\n        text = BeautifulSoup(\n            page[\"body\"][\"storage\"][\"value\"], \"lxml\"\n        ).get_text() + \"\".join(attachment_texts)\n        if include_comments:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}963{"id": "877eef4d6824-8", "text": ").get_text() + \"\".join(attachment_texts)\n        if include_comments:\n            comments = self.confluence.get_page_comments(\n                page[\"id\"], expand=\"body.view.value\", depth=\"all\"\n            )[\"results\"]\n            comment_texts = [\n                BeautifulSoup(comment[\"body\"][\"view\"][\"value\"], \"lxml\").get_text()\n                for comment in comments\n            ]\n            text = text + \"\".join(comment_texts)\n        return Document(\n            page_content=text,\n            metadata={\n                \"title\": page[\"title\"],\n                \"id\": page[\"id\"],\n                \"source\": self.base_url.strip(\"/\") + page[\"_links\"][\"webui\"],\n            },\n        )\n[docs]    def process_attachment(self, page_id: str) -> List[str]:\n        try:\n            from PIL import Image  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`Pillow` package not found, \" \"please run `pip install Pillow`\"\n            )\n        # depending on setup you may also need to set the correct path for\n        # poppler and tesseract\n        attachments = self.confluence.get_attachments_from_content(page_id)[\"results\"]\n        texts = []\n        for attachment in attachments:\n            media_type = attachment[\"metadata\"][\"mediaType\"]\n            absolute_url = self.base_url + attachment[\"_links\"][\"download\"]\n            title = attachment[\"title\"]\n            if media_type == \"application/pdf\":\n                text = title + self.process_pdf(absolute_url)\n            elif (\n                media_type == \"image/png\"\n                or media_type == \"image/jpg\"\n                or media_type == \"image/jpeg\"\n            ):\n                text = title + self.process_image(absolute_url)\n            elif (\n                media_type == \"application/vnd.openxmlformats-officedocument\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}964{"id": "877eef4d6824-9", "text": "elif (\n                media_type == \"application/vnd.openxmlformats-officedocument\"\n                \".wordprocessingml.document\"\n            ):\n                text = title + self.process_doc(absolute_url)\n            elif media_type == \"application/vnd.ms-excel\":\n                text = title + self.process_xls(absolute_url)\n            elif media_type == \"image/svg+xml\":\n                text = title + self.process_svg(absolute_url)\n            else:\n                continue\n            texts.append(text)\n        return texts\n[docs]    def process_pdf(self, link: str) -> str:\n        try:\n            import pytesseract  # noqa: F401\n            from pdf2image import convert_from_bytes  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`pytesseract` or `pdf2image` package not found, \"\n                \"please run `pip install pytesseract pdf2image`\"\n            )\n        response = self.confluence.request(path=link, absolute=True)\n        text = \"\"\n        if (\n            response.status_code != 200\n            or response.content == b\"\"\n            or response.content is None\n        ):\n            return text\n        try:\n            images = convert_from_bytes(response.content)\n        except ValueError:\n            return text\n        for i, image in enumerate(images):\n            image_text = pytesseract.image_to_string(image)\n            text += f\"Page {i + 1}:\\n{image_text}\\n\\n\"\n        return text\n[docs]    def process_image(self, link: str) -> str:\n        try:\n            import pytesseract  # noqa: F401\n            from PIL import Image  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`pytesseract` or `Pillow` package not found, \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}965{"id": "877eef4d6824-10", "text": "\"`pytesseract` or `Pillow` package not found, \"\n                \"please run `pip install pytesseract Pillow`\"\n            )\n        response = self.confluence.request(path=link, absolute=True)\n        text = \"\"\n        if (\n            response.status_code != 200\n            or response.content == b\"\"\n            or response.content is None\n        ):\n            return text\n        try:\n            image = Image.open(BytesIO(response.content))\n        except OSError:\n            return text\n        return pytesseract.image_to_string(image)\n[docs]    def process_doc(self, link: str) -> str:\n        try:\n            import docx2txt  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`docx2txt` package not found, please run `pip install docx2txt`\"\n            )\n        response = self.confluence.request(path=link, absolute=True)\n        text = \"\"\n        if (\n            response.status_code != 200\n            or response.content == b\"\"\n            or response.content is None\n        ):\n            return text\n        file_data = BytesIO(response.content)\n        return docx2txt.process(file_data)\n[docs]    def process_xls(self, link: str) -> str:\n        try:\n            import xlrd  # noqa: F401\n        except ImportError:\n            raise ImportError(\"`xlrd` package not found, please run `pip install xlrd`\")\n        response = self.confluence.request(path=link, absolute=True)\n        text = \"\"\n        if (\n            response.status_code != 200\n            or response.content == b\"\"\n            or response.content is None\n        ):\n            return text\n        workbook = xlrd.open_workbook(file_contents=response.content)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}966{"id": "877eef4d6824-11", "text": "):\n            return text\n        workbook = xlrd.open_workbook(file_contents=response.content)\n        for sheet in workbook.sheets():\n            text += f\"{sheet.name}:\\n\"\n            for row in range(sheet.nrows):\n                for col in range(sheet.ncols):\n                    text += f\"{sheet.cell_value(row, col)}\\t\"\n                text += \"\\n\"\n            text += \"\\n\"\n        return text\n[docs]    def process_svg(self, link: str) -> str:\n        try:\n            import pytesseract  # noqa: F401\n            from PIL import Image  # noqa: F401\n            from reportlab.graphics import renderPM  # noqa: F401\n            from svglib.svglib import svg2rlg  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`pytesseract`, `Pillow`, `reportlab` or `svglib` package not found, \"\n                \"please run `pip install pytesseract Pillow reportlab svglib`\"\n            )\n        response = self.confluence.request(path=link, absolute=True)\n        text = \"\"\n        if (\n            response.status_code != 200\n            or response.content == b\"\"\n            or response.content is None\n        ):\n            return text\n        drawing = svg2rlg(BytesIO(response.content))\n        img_data = BytesIO()\n        renderPM.drawToFile(drawing, img_data, fmt=\"PNG\")\n        img_data.seek(0)\n        image = Image.open(img_data)\n        return pytesseract.image_to_string(image)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/confluence.html"}967{"id": "44eb65c25dcf-0", "text": "Source code for langchain.document_loaders.slack_directory\n\"\"\"Loader for documents from a Slack export.\"\"\"\nimport json\nimport zipfile\nfrom pathlib import Path\nfrom typing import Dict, List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class SlackDirectoryLoader(BaseLoader):\n    \"\"\"Loader for loading documents from a Slack directory dump.\"\"\"\n    def __init__(self, zip_path: str, workspace_url: Optional[str] = None):\n        \"\"\"Initialize the SlackDirectoryLoader.\n        Args:\n            zip_path (str): The path to the Slack directory dump zip file.\n            workspace_url (Optional[str]): The Slack workspace URL.\n              Including the URL will turn\n              sources into links. Defaults to None.\n        \"\"\"\n        self.zip_path = Path(zip_path)\n        self.workspace_url = workspace_url\n        self.channel_id_map = self._get_channel_id_map(self.zip_path)\n    @staticmethod\n    def _get_channel_id_map(zip_path: Path) -> Dict[str, str]:\n        \"\"\"Get a dictionary mapping channel names to their respective IDs.\"\"\"\n        with zipfile.ZipFile(zip_path, \"r\") as zip_file:\n            try:\n                with zip_file.open(\"channels.json\", \"r\") as f:\n                    channels = json.load(f)\n                return {channel[\"name\"]: channel[\"id\"] for channel in channels}\n            except KeyError:\n                return {}\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load and return documents from the Slack directory dump.\"\"\"\n        docs = []\n        with zipfile.ZipFile(self.zip_path, \"r\") as zip_file:\n            for channel_path in zip_file.namelist():\n                channel_name = Path(channel_path).parent.name\n                if not channel_name:\n                    continue", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/slack_directory.html"}968{"id": "44eb65c25dcf-1", "text": "channel_name = Path(channel_path).parent.name\n                if not channel_name:\n                    continue\n                if channel_path.endswith(\".json\"):\n                    messages = self._read_json(zip_file, channel_path)\n                    for message in messages:\n                        document = self._convert_message_to_document(\n                            message, channel_name\n                        )\n                        docs.append(document)\n        return docs\n    def _read_json(self, zip_file: zipfile.ZipFile, file_path: str) -> List[dict]:\n        \"\"\"Read JSON data from a zip subfile.\"\"\"\n        with zip_file.open(file_path, \"r\") as f:\n            data = json.load(f)\n        return data\n    def _convert_message_to_document(\n        self, message: dict, channel_name: str\n    ) -> Document:\n        \"\"\"\n        Convert a message to a Document object.\n        Args:\n            message (dict): A message in the form of a dictionary.\n            channel_name (str): The name of the channel the message belongs to.\n        Returns:\n            Document: A Document object representing the message.\n        \"\"\"\n        text = message.get(\"text\", \"\")\n        metadata = self._get_message_metadata(message, channel_name)\n        return Document(\n            page_content=text,\n            metadata=metadata,\n        )\n    def _get_message_metadata(self, message: dict, channel_name: str) -> dict:\n        \"\"\"Create and return metadata for a given message and channel.\"\"\"\n        timestamp = message.get(\"ts\", \"\")\n        user = message.get(\"user\", \"\")\n        source = self._get_message_source(channel_name, user, timestamp)\n        return {\n            \"source\": source,\n            \"channel\": channel_name,\n            \"timestamp\": timestamp,\n            \"user\": user,\n        }", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/slack_directory.html"}969{"id": "44eb65c25dcf-2", "text": "\"timestamp\": timestamp,\n            \"user\": user,\n        }\n    def _get_message_source(self, channel_name: str, user: str, timestamp: str) -> str:\n        \"\"\"\n        Get the message source as a string.\n        Args:\n            channel_name (str): The name of the channel the message belongs to.\n            user (str): The user ID who sent the message.\n            timestamp (str): The timestamp of the message.\n        Returns:\n            str: The message source.\n        \"\"\"\n        if self.workspace_url:\n            channel_id = self.channel_id_map.get(channel_name, \"\")\n            return (\n                f\"{self.workspace_url}/archives/{channel_id}\"\n                + f\"/p{timestamp.replace('.', '')}\"\n            )\n        else:\n            return f\"{channel_name} - {user} - {timestamp}\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/slack_directory.html"}970{"id": "1ae701e0c6dd-0", "text": "Source code for langchain.document_loaders.hn\n\"\"\"Loader that loads HN.\"\"\"\nfrom typing import Any, List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class HNLoader(WebBaseLoader):\n    \"\"\"Load Hacker News data from either main page results or the comments page.\"\"\"\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Get important HN webpage information.\n        Components are:\n            - title\n            - content\n            - source url,\n            - time of post\n            - author of the post\n            - number of comments\n            - rank of the post\n        \"\"\"\n        soup_info = self.scrape()\n        if \"item\" in self.web_path:\n            return self.load_comments(soup_info)\n        else:\n            return self.load_results(soup_info)\n[docs]    def load_comments(self, soup_info: Any) -> List[Document]:\n        \"\"\"Load comments from a HN post.\"\"\"\n        comments = soup_info.select(\"tr[class='athing comtr']\")\n        title = soup_info.select_one(\"tr[id='pagespace']\").get(\"title\")\n        return [\n            Document(\n                page_content=comment.text.strip(),\n                metadata={\"source\": self.web_path, \"title\": title},\n            )\n            for comment in comments\n        ]\n[docs]    def load_results(self, soup: Any) -> List[Document]:\n        \"\"\"Load items from an HN page.\"\"\"\n        items = soup.select(\"tr[class='athing']\")\n        documents = []\n        for lineItem in items:\n            ranking = lineItem.select_one(\"span[class='rank']\").text\n            link = lineItem.find(\"span\", {\"class\": \"titleline\"}).find(\"a\").get(\"href\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/hn.html"}971{"id": "1ae701e0c6dd-1", "text": "title = lineItem.find(\"span\", {\"class\": \"titleline\"}).text.strip()\n            metadata = {\n                \"source\": self.web_path,\n                \"title\": title,\n                \"link\": link,\n                \"ranking\": ranking,\n            }\n            documents.append(\n                Document(\n                    page_content=title, link=link, ranking=ranking, metadata=metadata\n                )\n            )\n        return documents\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/hn.html"}972{"id": "6ecd3b76900c-0", "text": "Source code for langchain.document_loaders.gutenberg\n\"\"\"Loader that loads .txt web files.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class GutenbergLoader(BaseLoader):\n    \"\"\"Loader that uses urllib to load .txt web files.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        if not file_path.startswith(\"https://www.gutenberg.org\"):\n            raise ValueError(\"file path must start with 'https://www.gutenberg.org'\")\n        if not file_path.endswith(\".txt\"):\n            raise ValueError(\"file path must end with '.txt'\")\n        self.file_path = file_path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        from urllib.request import urlopen\n        elements = urlopen(self.file_path)\n        text = \"\\n\\n\".join([str(el.decode(\"utf-8-sig\")) for el in elements])\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/gutenberg.html"}973{"id": "e2b7015d5df3-0", "text": "Source code for langchain.document_loaders.email\n\"\"\"Loader that loads email files.\"\"\"\nimport os\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.unstructured import (\n    UnstructuredFileLoader,\n    satisfies_min_unstructured_version,\n)\n[docs]class UnstructuredEmailLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load email files.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.file_utils.filetype import FileType, detect_filetype\n        filetype = detect_filetype(self.file_path)\n        if filetype == FileType.EML:\n            from unstructured.partition.email import partition_email\n            return partition_email(filename=self.file_path, **self.unstructured_kwargs)\n        elif satisfies_min_unstructured_version(\"0.5.8\") and filetype == FileType.MSG:\n            from unstructured.partition.msg import partition_msg\n            return partition_msg(filename=self.file_path, **self.unstructured_kwargs)\n        else:\n            raise ValueError(\n                f\"Filetype {filetype} is not supported in UnstructuredEmailLoader.\"\n            )\n[docs]class OutlookMessageLoader(BaseLoader):\n    \"\"\"\n    Loader that loads Outlook Message files using extract_msg.\n    https://github.com/TeamMsgExtractor/msg-extractor\n    \"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        if not os.path.isfile(self.file_path):\n            raise ValueError(\"File path %s is not a valid file\" % self.file_path)\n        try:\n            import extract_msg  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"extract_msg is not installed. Please install it with \"\n                \"`pip install extract_msg`\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/email.html"}974{"id": "e2b7015d5df3-1", "text": "\"`pip install extract_msg`\"\n            )\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load data into document objects.\"\"\"\n        import extract_msg\n        msg = extract_msg.Message(self.file_path)\n        return [\n            Document(\n                page_content=msg.body,\n                metadata={\n                    \"subject\": msg.subject,\n                    \"sender\": msg.sender,\n                    \"date\": msg.date,\n                },\n            )\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/email.html"}975{"id": "25fc0ff9ffe2-0", "text": "Source code for langchain.document_loaders.sitemap\n\"\"\"Loader that fetches a sitemap and loads those URLs.\"\"\"\nimport itertools\nimport re\nfrom typing import Any, Callable, Generator, Iterable, List, Optional\nfrom langchain.document_loaders.web_base import WebBaseLoader\nfrom langchain.schema import Document\ndef _default_parsing_function(content: Any) -> str:\n    return str(content.get_text())\ndef _default_meta_function(meta: dict, _content: Any) -> dict:\n    return {\"source\": meta[\"loc\"], **meta}\ndef _batch_block(iterable: Iterable, size: int) -> Generator[List[dict], None, None]:\n    it = iter(iterable)\n    while item := list(itertools.islice(it, size)):\n        yield item\n[docs]class SitemapLoader(WebBaseLoader):\n    \"\"\"Loader that fetches a sitemap and loads those URLs.\"\"\"\n    def __init__(\n        self,\n        web_path: str,\n        filter_urls: Optional[List[str]] = None,\n        parsing_function: Optional[Callable] = None,\n        blocksize: Optional[int] = None,\n        blocknum: int = 0,\n        meta_function: Optional[Callable] = None,\n        is_local: bool = False,\n    ):\n        \"\"\"Initialize with webpage path and optional filter URLs.\n        Args:\n            web_path: url of the sitemap. can also be a local path\n            filter_urls: list of strings or regexes that will be applied to filter the\n                urls that are parsed and loaded\n            parsing_function: Function to parse bs4.Soup output\n            blocksize: number of sitemap locations per block\n            blocknum: the number of the block that should be loaded - zero indexed\n            meta_function: Function to parse bs4.Soup output for metadata", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/sitemap.html"}976{"id": "25fc0ff9ffe2-1", "text": "meta_function: Function to parse bs4.Soup output for metadata\n                remember when setting this method to also copy metadata[\"loc\"]\n                to metadata[\"source\"] if you are using this field\n            is_local: whether the sitemap is a local file\n        \"\"\"\n        if blocksize is not None and blocksize < 1:\n            raise ValueError(\"Sitemap blocksize should be at least 1\")\n        if blocknum < 0:\n            raise ValueError(\"Sitemap blocknum can not be lower then 0\")\n        try:\n            import lxml  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"lxml package not found, please install it with \" \"`pip install lxml`\"\n            )\n        super().__init__(web_path)\n        self.filter_urls = filter_urls\n        self.parsing_function = parsing_function or _default_parsing_function\n        self.meta_function = meta_function or _default_meta_function\n        self.blocksize = blocksize\n        self.blocknum = blocknum\n        self.is_local = is_local\n[docs]    def parse_sitemap(self, soup: Any) -> List[dict]:\n        \"\"\"Parse sitemap xml and load into a list of dicts.\"\"\"\n        els = []\n        for url in soup.find_all(\"url\"):\n            loc = url.find(\"loc\")\n            if not loc:\n                continue\n            if self.filter_urls and not any(\n                re.match(r, loc.text) for r in self.filter_urls\n            ):\n                continue\n            els.append(\n                {\n                    tag: prop.text\n                    for tag in [\"loc\", \"lastmod\", \"changefreq\", \"priority\"]\n                    if (prop := url.find(tag))\n                }\n            )\n        for sitemap in soup.find_all(\"sitemap\"):", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/sitemap.html"}977{"id": "25fc0ff9ffe2-2", "text": "}\n            )\n        for sitemap in soup.find_all(\"sitemap\"):\n            loc = sitemap.find(\"loc\")\n            if not loc:\n                continue\n            soup_child = self.scrape_all([loc.text], \"xml\")[0]\n            els.extend(self.parse_sitemap(soup_child))\n        return els\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load sitemap.\"\"\"\n        if self.is_local:\n            try:\n                import bs4\n            except ImportError:\n                raise ImportError(\n                    \"beautifulsoup4 package not found, please install it\"\n                    \" with `pip install beautifulsoup4`\"\n                )\n            fp = open(self.web_path)\n            soup = bs4.BeautifulSoup(fp, \"xml\")\n        else:\n            soup = self.scrape(\"xml\")\n        els = self.parse_sitemap(soup)\n        if self.blocksize is not None:\n            elblocks = list(_batch_block(els, self.blocksize))\n            blockcount = len(elblocks)\n            if blockcount - 1 < self.blocknum:\n                raise ValueError(\n                    \"Selected sitemap does not contain enough blocks for given blocknum\"\n                )\n            else:\n                els = elblocks[self.blocknum]\n        results = self.scrape_all([el[\"loc\"].strip() for el in els if \"loc\" in el])\n        return [\n            Document(\n                page_content=self.parsing_function(results[i]),\n                metadata=self.meta_function(els[i], results[i]),\n            )\n            for i in range(len(results))\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/sitemap.html"}978{"id": "979baa0476ca-0", "text": "Source code for langchain.document_loaders.word_document\n\"\"\"Loader that loads word documents.\"\"\"\nimport os\nimport tempfile\nfrom abc import ABC\nfrom typing import List\nfrom urllib.parse import urlparse\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class Docx2txtLoader(BaseLoader, ABC):\n    \"\"\"Loads a DOCX with docx2txt and chunks at character level.\n    Defaults to check for local file, but if the file is a web path, it will download it\n    to a temporary file, and use that, then clean up the temporary file after completion\n    \"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        if \"~\" in self.file_path:\n            self.file_path = os.path.expanduser(self.file_path)\n        # If the file is a web path, download it to a temporary file, and use that\n        if not os.path.isfile(self.file_path) and self._is_valid_url(self.file_path):\n            r = requests.get(self.file_path)\n            if r.status_code != 200:\n                raise ValueError(\n                    \"Check the url of your file; returned status code %s\"\n                    % r.status_code\n                )\n            self.web_path = self.file_path\n            self.temp_file = tempfile.NamedTemporaryFile()\n            self.temp_file.write(r.content)\n            self.file_path = self.temp_file.name\n        elif not os.path.isfile(self.file_path):\n            raise ValueError(\"File path %s is not a valid file or url\" % self.file_path)\n    def __del__(self) -> None:\n        if hasattr(self, \"temp_file\"):\n            self.temp_file.close()", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/word_document.html"}979{"id": "979baa0476ca-1", "text": "if hasattr(self, \"temp_file\"):\n            self.temp_file.close()\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load given path as single page.\"\"\"\n        import docx2txt\n        return [\n            Document(\n                page_content=docx2txt.process(self.file_path),\n                metadata={\"source\": self.file_path},\n            )\n        ]\n    @staticmethod\n    def _is_valid_url(url: str) -> bool:\n        \"\"\"Check if the url is valid.\"\"\"\n        parsed = urlparse(url)\n        return bool(parsed.netloc) and bool(parsed.scheme)\n[docs]class UnstructuredWordDocumentLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load word documents.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.__version__ import __version__ as __unstructured_version__\n        from unstructured.file_utils.filetype import FileType, detect_filetype\n        unstructured_version = tuple(\n            [int(x) for x in __unstructured_version__.split(\".\")]\n        )\n        # NOTE(MthwRobinson) - magic will raise an import error if the libmagic\n        # system dependency isn't installed. If it's not installed, we'll just\n        # check the file extension\n        try:\n            import magic  # noqa: F401\n            is_doc = detect_filetype(self.file_path) == FileType.DOC\n        except ImportError:\n            _, extension = os.path.splitext(str(self.file_path))\n            is_doc = extension == \".doc\"\n        if is_doc and unstructured_version < (0, 4, 11):\n            raise ValueError(\n                f\"You are on unstructured version {__unstructured_version__}. \"\n                \"Partitioning .doc files is only supported in unstructured>=0.4.11. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/word_document.html"}980{"id": "979baa0476ca-2", "text": "\"Please upgrade the unstructured package and try again.\"\n            )\n        if is_doc:\n            from unstructured.partition.doc import partition_doc\n            return partition_doc(filename=self.file_path, **self.unstructured_kwargs)\n        else:\n            from unstructured.partition.docx import partition_docx\n            return partition_docx(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/word_document.html"}981{"id": "a333e55eddc8-0", "text": "Source code for langchain.document_loaders.dataframe\n\"\"\"Load from Dataframe object\"\"\"\nfrom typing import Any, List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class DataFrameLoader(BaseLoader):\n    \"\"\"Load Pandas DataFrames.\"\"\"\n    def __init__(self, data_frame: Any, page_content_column: str = \"text\"):\n        \"\"\"Initialize with dataframe object.\"\"\"\n        import pandas as pd\n        if not isinstance(data_frame, pd.DataFrame):\n            raise ValueError(\n                f\"Expected data_frame to be a pd.DataFrame, got {type(data_frame)}\"\n            )\n        self.data_frame = data_frame\n        self.page_content_column = page_content_column\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load from the dataframe.\"\"\"\n        result = []\n        # For very large dataframes, this needs to yield instead of building a list\n        # but that would require chaging return type to a generator for BaseLoader\n        # and all its subclasses, which is a bigger refactor. Marking as future TODO.\n        # This change will allow us to extend this to Spark and Dask dataframes.\n        for _, row in self.data_frame.iterrows():\n            text = row[self.page_content_column]\n            metadata = row.to_dict()\n            metadata.pop(self.page_content_column)\n            result.append(Document(page_content=text, metadata=metadata))\n        return result\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/dataframe.html"}982{"id": "b715c76ccdde-0", "text": "Source code for langchain.document_loaders.web_base\n\"\"\"Web base loader class.\"\"\"\nimport asyncio\nimport logging\nimport warnings\nfrom typing import Any, List, Optional, Union\nimport aiohttp\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\ndefault_header_template = {\n    \"User-Agent\": \"\",\n    \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*\"\n    \";q=0.8\",\n    \"Accept-Language\": \"en-US,en;q=0.5\",\n    \"Referer\": \"https://www.google.com/\",\n    \"DNT\": \"1\",\n    \"Connection\": \"keep-alive\",\n    \"Upgrade-Insecure-Requests\": \"1\",\n}\ndef _build_metadata(soup: Any, url: str) -> dict:\n    \"\"\"Build metadata from BeautifulSoup output.\"\"\"\n    metadata = {\"source\": url}\n    if title := soup.find(\"title\"):\n        metadata[\"title\"] = title.get_text()\n    if description := soup.find(\"meta\", attrs={\"name\": \"description\"}):\n        metadata[\"description\"] = description.get(\"content\", None)\n    if html := soup.find(\"html\"):\n        metadata[\"language\"] = html.get(\"lang\", None)\n    return metadata\n[docs]class WebBaseLoader(BaseLoader):\n    \"\"\"Loader that uses urllib and beautiful soup to load webpages.\"\"\"\n    web_paths: List[str]\n    requests_per_second: int = 2\n    \"\"\"Max number of concurrent requests to make.\"\"\"\n    default_parser: str = \"html.parser\"\n    \"\"\"Default parser to use for BeautifulSoup.\"\"\"\n    def __init__(\n        self, web_path: Union[str, List[str]], header_template: Optional[dict] = None", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/web_base.html"}983{"id": "b715c76ccdde-1", "text": "):\n        \"\"\"Initialize with webpage path.\"\"\"\n        # TODO: Deprecate web_path in favor of web_paths, and remove this\n        # left like this because there are a number of loaders that expect single\n        # urls\n        if isinstance(web_path, str):\n            self.web_paths = [web_path]\n        elif isinstance(web_path, List):\n            self.web_paths = web_path\n        self.session = requests.Session()\n        try:\n            import bs4  # noqa:F401\n        except ImportError:\n            raise ValueError(\n                \"bs4 package not found, please install it with \" \"`pip install bs4`\"\n            )\n        headers = header_template or default_header_template\n        if not headers.get(\"User-Agent\"):\n            try:\n                from fake_useragent import UserAgent\n                headers[\"User-Agent\"] = UserAgent().random\n            except ImportError:\n                logger.info(\n                    \"fake_useragent not found, using default user agent.\"\n                    \"To get a realistic header for requests, \"\n                    \"`pip install fake_useragent`.\"\n                )\n        self.session.headers = dict(headers)\n    @property\n    def web_path(self) -> str:\n        if len(self.web_paths) > 1:\n            raise ValueError(\"Multiple webpaths found.\")\n        return self.web_paths[0]\n    async def _fetch(\n        self, url: str, retries: int = 3, cooldown: int = 2, backoff: float = 1.5\n    ) -> str:\n        async with aiohttp.ClientSession() as session:\n            for i in range(retries):\n                try:\n                    async with session.get(\n                        url, headers=self.session.headers\n                    ) as response:\n                        return await response.text()\n                except aiohttp.ClientConnectionError as e:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/web_base.html"}984{"id": "b715c76ccdde-2", "text": "return await response.text()\n                except aiohttp.ClientConnectionError as e:\n                    if i == retries - 1:\n                        raise\n                    else:\n                        logger.warning(\n                            f\"Error fetching {url} with attempt \"\n                            f\"{i + 1}/{retries}: {e}. Retrying...\"\n                        )\n                        await asyncio.sleep(cooldown * backoff**i)\n        raise ValueError(\"retry count exceeded\")\n    async def _fetch_with_rate_limit(\n        self, url: str, semaphore: asyncio.Semaphore\n    ) -> str:\n        async with semaphore:\n            return await self._fetch(url)\n[docs]    async def fetch_all(self, urls: List[str]) -> Any:\n        \"\"\"Fetch all urls concurrently with rate limiting.\"\"\"\n        semaphore = asyncio.Semaphore(self.requests_per_second)\n        tasks = []\n        for url in urls:\n            task = asyncio.ensure_future(self._fetch_with_rate_limit(url, semaphore))\n            tasks.append(task)\n        try:\n            from tqdm.asyncio import tqdm_asyncio\n            return await tqdm_asyncio.gather(\n                *tasks, desc=\"Fetching pages\", ascii=True, mininterval=1\n            )\n        except ImportError:\n            warnings.warn(\"For better logging of progress, `pip install tqdm`\")\n            return await asyncio.gather(*tasks)\n    @staticmethod\n    def _check_parser(parser: str) -> None:\n        \"\"\"Check that parser is valid for bs4.\"\"\"\n        valid_parsers = [\"html.parser\", \"lxml\", \"xml\", \"lxml-xml\", \"html5lib\"]\n        if parser not in valid_parsers:\n            raise ValueError(\n                \"`parser` must be one of \" + \", \".join(valid_parsers) + \".\"\n            )", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/web_base.html"}985{"id": "b715c76ccdde-3", "text": ")\n[docs]    def scrape_all(self, urls: List[str], parser: Union[str, None] = None) -> List[Any]:\n        \"\"\"Fetch all urls, then return soups for all results.\"\"\"\n        from bs4 import BeautifulSoup\n        results = asyncio.run(self.fetch_all(urls))\n        final_results = []\n        for i, result in enumerate(results):\n            url = urls[i]\n            if parser is None:\n                if url.endswith(\".xml\"):\n                    parser = \"xml\"\n                else:\n                    parser = self.default_parser\n                self._check_parser(parser)\n            final_results.append(BeautifulSoup(result, parser))\n        return final_results\n    def _scrape(self, url: str, parser: Union[str, None] = None) -> Any:\n        from bs4 import BeautifulSoup\n        if parser is None:\n            if url.endswith(\".xml\"):\n                parser = \"xml\"\n            else:\n                parser = self.default_parser\n        self._check_parser(parser)\n        html_doc = self.session.get(url)\n        html_doc.encoding = html_doc.apparent_encoding\n        return BeautifulSoup(html_doc.text, parser)\n[docs]    def scrape(self, parser: Union[str, None] = None) -> Any:\n        \"\"\"Scrape data from webpage and return it in BeautifulSoup format.\"\"\"\n        if parser is None:\n            parser = self.default_parser\n        return self._scrape(self.web_path, parser)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load text from the url(s) in web_path.\"\"\"\n        docs = []\n        for path in self.web_paths:\n            soup = self._scrape(path)\n            text = soup.get_text()\n            metadata = _build_metadata(soup, path)\n            docs.append(Document(page_content=text, metadata=metadata))", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/web_base.html"}986{"id": "b715c76ccdde-4", "text": "docs.append(Document(page_content=text, metadata=metadata))\n        return docs\n[docs]    def aload(self) -> List[Document]:\n        \"\"\"Load text from the urls in web_path async into Documents.\"\"\"\n        results = self.scrape_all(self.web_paths)\n        docs = []\n        for i in range(len(results)):\n            soup = results[i]\n            text = soup.get_text()\n            metadata = _build_metadata(soup, self.web_paths[i])\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/web_base.html"}987{"id": "2e6281f18b4c-0", "text": "Source code for langchain.document_loaders.wikipedia\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utilities.wikipedia import WikipediaAPIWrapper\n[docs]class WikipediaLoader(BaseLoader):\n    \"\"\"Loads a query result from www.wikipedia.org into a list of Documents.\n    The hard limit on the number of downloaded Documents is 300 for now.\n    Each wiki page represents one Document.\n    \"\"\"\n    def __init__(\n        self,\n        query: str,\n        lang: str = \"en\",\n        load_max_docs: Optional[int] = 100,\n        load_all_available_meta: Optional[bool] = False,\n    ):\n        self.query = query\n        self.lang = lang\n        self.load_max_docs = load_max_docs\n        self.load_all_available_meta = load_all_available_meta\n[docs]    def load(self) -> List[Document]:\n        client = WikipediaAPIWrapper(\n            lang=self.lang,\n            top_k_results=self.load_max_docs,\n            load_all_available_meta=self.load_all_available_meta,\n        )\n        docs = client.load(self.query)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/wikipedia.html"}988{"id": "160c51a8258d-0", "text": "Source code for langchain.document_loaders.onedrive\n\"\"\"Loader that loads data from OneDrive\"\"\"\nfrom __future__ import annotations\nimport logging\nimport os\nimport tempfile\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING, Dict, List, Optional, Type, Union\nfrom pydantic import BaseModel, BaseSettings, Field, FilePath, SecretStr\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.onedrive_file import OneDriveFileLoader\nif TYPE_CHECKING:\n    from O365 import Account\n    from O365.drive import Drive, Folder\nSCOPES = [\"offline_access\", \"Files.Read.All\"]\nlogger = logging.getLogger(__name__)\nclass _OneDriveSettings(BaseSettings):\n    client_id: str = Field(..., env=\"O365_CLIENT_ID\")\n    client_secret: SecretStr = Field(..., env=\"O365_CLIENT_SECRET\")\n    class Config:\n        env_prefix = \"\"\n        case_sentive = False\n        env_file = \".env\"\nclass _OneDriveTokenStorage(BaseSettings):\n    token_path: FilePath = Field(Path.home() / \".credentials\" / \"o365_token.txt\")\nclass _FileType(str, Enum):\n    DOC = \"doc\"\n    DOCX = \"docx\"\n    PDF = \"pdf\"\nclass _SupportedFileTypes(BaseModel):\n    file_types: List[_FileType]\n    def fetch_mime_types(self) -> Dict[str, str]:\n        mime_types_mapping = {}\n        for file_type in self.file_types:\n            if file_type.value == \"doc\":\n                mime_types_mapping[file_type.value] = \"application/msword\"\n            elif file_type.value == \"docx\":\n                mime_types_mapping[\n                    file_type.value", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/onedrive.html"}989{"id": "160c51a8258d-1", "text": "mime_types_mapping[\n                    file_type.value\n                ] = \"application/vnd.openxmlformats-officedocument.wordprocessingml.document\"  # noqa: E501\n            elif file_type.value == \"pdf\":\n                mime_types_mapping[file_type.value] = \"application/pdf\"\n        return mime_types_mapping\n[docs]class OneDriveLoader(BaseLoader, BaseModel):\n    settings: _OneDriveSettings = Field(default_factory=_OneDriveSettings)\n    drive_id: str = Field(...)\n    folder_path: Optional[str] = None\n    object_ids: Optional[List[str]] = None\n    auth_with_token: bool = False\n    def _auth(self) -> Type[Account]:\n        \"\"\"\n        Authenticates the OneDrive API client using the specified\n        authentication method and returns the Account object.\n        Returns:\n            Type[Account]: The authenticated Account object.\n        \"\"\"\n        try:\n            from O365 import FileSystemTokenBackend\n        except ImportError:\n            raise ImportError(\n                \"O365 package not found, please install it with `pip install o365`\"\n            )\n        if self.auth_with_token:\n            token_storage = _OneDriveTokenStorage()\n            token_path = token_storage.token_path\n            token_backend = FileSystemTokenBackend(\n                token_path=token_path.parent, token_filename=token_path.name\n            )\n            account = Account(\n                credentials=(\n                    self.settings.client_id,\n                    self.settings.client_secret.get_secret_value(),\n                ),\n                scopes=SCOPES,\n                token_backend=token_backend,\n                **{\"raise_http_errors\": False},\n            )\n        else:\n            token_backend = FileSystemTokenBackend(\n                token_path=Path.home() / \".credentials\"\n            )\n            account = Account(\n                credentials=(\n                    self.settings.client_id,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/onedrive.html"}990{"id": "160c51a8258d-2", "text": ")\n            account = Account(\n                credentials=(\n                    self.settings.client_id,\n                    self.settings.client_secret.get_secret_value(),\n                ),\n                scopes=SCOPES,\n                token_backend=token_backend,\n                **{\"raise_http_errors\": False},\n            )\n            # make the auth\n            account.authenticate()\n        return account\n    def _get_folder_from_path(self, drive: Type[Drive]) -> Union[Folder, Drive]:\n        \"\"\"\n        Returns the folder or drive object located at the\n        specified path relative to the given drive.\n        Args:\n            drive (Type[Drive]): The root drive from which the folder path is relative.\n        Returns:\n            Union[Folder, Drive]: The folder or drive object\n            located at the specified path.\n        Raises:\n            FileNotFoundError: If the path does not exist.\n        \"\"\"\n        subfolder_drive = drive\n        if self.folder_path is None:\n            return subfolder_drive\n        subfolders = [f for f in self.folder_path.split(\"/\") if f != \"\"]\n        if len(subfolders) == 0:\n            return subfolder_drive\n        items = subfolder_drive.get_items()\n        for subfolder in subfolders:\n            try:\n                subfolder_drive = list(filter(lambda x: subfolder in x.name, items))[0]\n                items = subfolder_drive.get_items()\n            except (IndexError, AttributeError):\n                raise FileNotFoundError(\"Path {} not exist.\".format(self.folder_path))\n        return subfolder_drive\n    def _load_from_folder(self, folder: Type[Folder]) -> List[Document]:\n        \"\"\"\n        Loads all supported document files from the specified folder\n        and returns a list of Document objects.\n        Args:\n            folder (Type[Folder]): The folder object to load the documents from.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/onedrive.html"}991{"id": "160c51a8258d-3", "text": "Args:\n            folder (Type[Folder]): The folder object to load the documents from.\n        Returns:\n            List[Document]: A list of Document objects representing\n            the loaded documents.\n        \"\"\"\n        docs = []\n        file_types = _SupportedFileTypes(file_types=[\"doc\", \"docx\", \"pdf\"])\n        file_mime_types = file_types.fetch_mime_types()\n        items = folder.get_items()\n        with tempfile.TemporaryDirectory() as temp_dir:\n            file_path = f\"{temp_dir}\"\n            os.makedirs(os.path.dirname(file_path), exist_ok=True)\n            for file in items:\n                if file.is_file:\n                    if file.mime_type in list(file_mime_types.values()):\n                        loader = OneDriveFileLoader(file=file)\n                        docs.extend(loader.load())\n        return docs\n    def _load_from_object_ids(self, drive: Type[Drive]) -> List[Document]:\n        \"\"\"\n        Loads all supported document files from the specified OneDrive\n        drive based on their object IDs and returns a list\n        of Document objects.\n        Args:\n            drive (Type[Drive]): The OneDrive drive object\n            to load the documents from.\n        Returns:\n            List[Document]: A list of Document objects representing\n            the loaded documents.\n        \"\"\"\n        docs = []\n        file_types = _SupportedFileTypes(file_types=[\"doc\", \"docx\", \"pdf\"])\n        file_mime_types = file_types.fetch_mime_types()\n        with tempfile.TemporaryDirectory() as temp_dir:\n            file_path = f\"{temp_dir}\"\n            os.makedirs(os.path.dirname(file_path), exist_ok=True)\n            for object_id in self.object_ids if self.object_ids else [\"\"]:\n                file = drive.get_item(object_id)\n                if not file:\n                    logging.warning(", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/onedrive.html"}992{"id": "160c51a8258d-4", "text": "file = drive.get_item(object_id)\n                if not file:\n                    logging.warning(\n                        \"There isn't a file with \"\n                        f\"object_id {object_id} in drive {drive}.\"\n                    )\n                    continue\n                if file.is_file:\n                    if file.mime_type in list(file_mime_types.values()):\n                        loader = OneDriveFileLoader(file=file)\n                        docs.extend(loader.load())\n        return docs\n[docs]    def load(self) -> List[Document]:\n        \"\"\"\n        Loads all supported document files from the specified OneDrive drive a\n        nd returns a list of Document objects.\n        Returns:\n            List[Document]: A list of Document objects\n            representing the loaded documents.\n        Raises:\n            ValueError: If the specified drive ID\n            does not correspond to a drive in the OneDrive storage.\n        \"\"\"\n        account = self._auth()\n        storage = account.storage()\n        drive = storage.get_drive(self.drive_id)\n        docs: List[Document] = []\n        if not drive:\n            raise ValueError(f\"There isn't a drive with id {self.drive_id}.\")\n        if self.folder_path:\n            folder = self._get_folder_from_path(drive=drive)\n            docs.extend(self._load_from_folder(folder=folder))\n        elif self.object_ids:\n            docs.extend(self._load_from_object_ids(drive=drive))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/onedrive.html"}993{"id": "5bd99980b71c-0", "text": "Source code for langchain.document_loaders.directory\n\"\"\"Loading logic for loading documents from a directory.\"\"\"\nimport concurrent\nimport logging\nfrom pathlib import Path\nfrom typing import Any, List, Optional, Type, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.html_bs import BSHTMLLoader\nfrom langchain.document_loaders.text import TextLoader\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\nFILE_LOADER_TYPE = Union[\n    Type[UnstructuredFileLoader], Type[TextLoader], Type[BSHTMLLoader]\n]\nlogger = logging.getLogger(__name__)\ndef _is_visible(p: Path) -> bool:\n    parts = p.parts\n    for _p in parts:\n        if _p.startswith(\".\"):\n            return False\n    return True\n[docs]class DirectoryLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from a directory.\"\"\"\n    def __init__(\n        self,\n        path: str,\n        glob: str = \"**/[!.]*\",\n        silent_errors: bool = False,\n        load_hidden: bool = False,\n        loader_cls: FILE_LOADER_TYPE = UnstructuredFileLoader,\n        loader_kwargs: Union[dict, None] = None,\n        recursive: bool = False,\n        show_progress: bool = False,\n        use_multithreading: bool = False,\n        max_concurrency: int = 4,\n    ):\n        \"\"\"Initialize with path to directory and how to glob over it.\"\"\"\n        if loader_kwargs is None:\n            loader_kwargs = {}\n        self.path = path\n        self.glob = glob\n        self.load_hidden = load_hidden\n        self.loader_cls = loader_cls\n        self.loader_kwargs = loader_kwargs\n        self.silent_errors = silent_errors", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/directory.html"}994{"id": "5bd99980b71c-1", "text": "self.loader_kwargs = loader_kwargs\n        self.silent_errors = silent_errors\n        self.recursive = recursive\n        self.show_progress = show_progress\n        self.use_multithreading = use_multithreading\n        self.max_concurrency = max_concurrency\n[docs]    def load_file(\n        self, item: Path, path: Path, docs: List[Document], pbar: Optional[Any]\n    ) -> None:\n        if item.is_file():\n            if _is_visible(item.relative_to(path)) or self.load_hidden:\n                try:\n                    sub_docs = self.loader_cls(str(item), **self.loader_kwargs).load()\n                    docs.extend(sub_docs)\n                except Exception as e:\n                    if self.silent_errors:\n                        logger.warning(e)\n                    else:\n                        raise e\n                finally:\n                    if pbar:\n                        pbar.update(1)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        p = Path(self.path)\n        if not p.exists():\n            raise FileNotFoundError(f\"Directory not found: '{self.path}'\")\n        if not p.is_dir():\n            raise ValueError(f\"Expected directory, got file: '{self.path}'\")\n        docs: List[Document] = []\n        items = list(p.rglob(self.glob) if self.recursive else p.glob(self.glob))\n        pbar = None\n        if self.show_progress:\n            try:\n                from tqdm import tqdm\n                pbar = tqdm(total=len(items))\n            except ImportError as e:\n                logger.warning(\n                    \"To log the progress of DirectoryLoader you need to install tqdm, \"\n                    \"`pip install tqdm`\"\n                )\n                if self.silent_errors:\n                    logger.warning(e)\n                else:\n                    raise e", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/directory.html"}995{"id": "5bd99980b71c-2", "text": "logger.warning(e)\n                else:\n                    raise e\n        if self.use_multithreading:\n            with concurrent.futures.ThreadPoolExecutor(\n                max_workers=self.max_concurrency\n            ) as executor:\n                executor.map(lambda i: self.load_file(i, p, docs, pbar), items)\n        else:\n            for i in items:\n                self.load_file(i, p, docs, pbar)\n        if pbar:\n            pbar.close()\n        return docs\n#\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/directory.html"}996{"id": "047adc7485e5-0", "text": "Source code for langchain.document_loaders.html_bs\n\"\"\"Loader that uses bs4 to load HTML files, enriching metadata with page title.\"\"\"\nimport logging\nfrom typing import Dict, List, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class BSHTMLLoader(BaseLoader):\n    \"\"\"Loader that uses beautiful soup to parse HTML files.\"\"\"\n    def __init__(\n        self,\n        file_path: str,\n        open_encoding: Union[str, None] = None,\n        bs_kwargs: Union[dict, None] = None,\n        get_text_separator: str = \"\",\n    ) -> None:\n        \"\"\"Initialise with path, and optionally, file encoding to use, and any kwargs\n        to pass to the BeautifulSoup object.\"\"\"\n        try:\n            import bs4  # noqa:F401\n        except ImportError:\n            raise ValueError(\n                \"beautifulsoup4 package not found, please install it with \"\n                \"`pip install beautifulsoup4`\"\n            )\n        self.file_path = file_path\n        self.open_encoding = open_encoding\n        if bs_kwargs is None:\n            bs_kwargs = {\"features\": \"lxml\"}\n        self.bs_kwargs = bs_kwargs\n        self.get_text_separator = get_text_separator\n[docs]    def load(self) -> List[Document]:\n        from bs4 import BeautifulSoup\n        \"\"\"Load HTML document into document objects.\"\"\"\n        with open(self.file_path, \"r\", encoding=self.open_encoding) as f:\n            soup = BeautifulSoup(f, **self.bs_kwargs)\n        text = soup.get_text(self.get_text_separator)\n        if soup.title:\n            title = str(soup.title.string)\n        else:\n            title = \"\"\n        metadata: Dict[str, Union[str, None]] = {", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/html_bs.html"}997{"id": "047adc7485e5-1", "text": "title = \"\"\n        metadata: Dict[str, Union[str, None]] = {\n            \"source\": self.file_path,\n            \"title\": title,\n        }\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/html_bs.html"}998{"id": "5c6438670e6a-0", "text": "Source code for langchain.document_loaders.json_loader\n\"\"\"Loader that loads data from JSON.\"\"\"\nimport json\nfrom pathlib import Path\nfrom typing import Any, Callable, Dict, List, Optional, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class JSONLoader(BaseLoader):\n    \"\"\"Loads a JSON file and references a jq schema provided to load the text into\n    documents.\n    Example:\n        [{\"text\": ...}, {\"text\": ...}, {\"text\": ...}] -> schema = .[].text\n        {\"key\": [{\"text\": ...}, {\"text\": ...}, {\"text\": ...}]} -> schema = .key[].text\n        [\"\", \"\", \"\"] -> schema = .[]\n    \"\"\"\n    def __init__(\n        self,\n        file_path: Union[str, Path],\n        jq_schema: str,\n        content_key: Optional[str] = None,\n        metadata_func: Optional[Callable[[Dict, Dict], Dict]] = None,\n        text_content: bool = True,\n    ):\n        \"\"\"Initialize the JSONLoader.\n        Args:\n            file_path (Union[str, Path]): The path to the JSON file.\n            jq_schema (str): The jq schema to use to extract the data or text from\n                the JSON.\n            content_key (str): The key to use to extract the content from the JSON if\n                the jq_schema results to a list of objects (dict).\n            metadata_func (Callable[Dict, Dict]): A function that takes in the JSON\n                object extracted by the jq_schema and the default metadata and returns\n                a dict of the updated metadata.\n            text_content (bool): Boolean flag to indicates whether the content is in\n                string format, default to True\n        \"\"\"\n        try:\n            import jq  # noqa:F401", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/json_loader.html"}999{"id": "5c6438670e6a-1", "text": "\"\"\"\n        try:\n            import jq  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"jq package not found, please install it with `pip install jq`\"\n            )\n        self.file_path = Path(file_path).resolve()\n        self._jq_schema = jq.compile(jq_schema)\n        self._content_key = content_key\n        self._metadata_func = metadata_func\n        self._text_content = text_content\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load and return documents from the JSON file.\"\"\"\n        data = self._jq_schema.input(json.loads(self.file_path.read_text()))\n        # Perform some validation\n        # This is not a perfect validation, but it should catch most cases\n        # and prevent the user from getting a cryptic error later on.\n        if self._content_key is not None:\n            self._validate_content_key(data)\n        docs = []\n        for i, sample in enumerate(data, 1):\n            metadata = dict(\n                source=str(self.file_path),\n                seq_num=i,\n            )\n            text = self._get_text(sample=sample, metadata=metadata)\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\n    def _get_text(self, sample: Any, metadata: dict) -> str:\n        \"\"\"Convert sample to string format\"\"\"\n        if self._content_key is not None:\n            content = sample.get(self._content_key)\n            if self._metadata_func is not None:\n                # We pass in the metadata dict to the metadata_func\n                # so that the user can customize the default metadata\n                # based on the content of the JSON object.\n                metadata = self._metadata_func(sample, metadata)\n        else:\n            content = sample", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/json_loader.html"}1000{"id": "5c6438670e6a-2", "text": "metadata = self._metadata_func(sample, metadata)\n        else:\n            content = sample\n        if self._text_content and not isinstance(content, str):\n            raise ValueError(\n                f\"Expected page_content is string, got {type(content)} instead. \\\n                    Set `text_content=False` if the desired input for \\\n                    `page_content` is not a string\"\n            )\n        # In case the text is None, set it to an empty string\n        elif isinstance(content, str):\n            return content\n        elif isinstance(content, dict):\n            return json.dumps(content) if content else \"\"\n        else:\n            return str(content) if content is not None else \"\"\n    def _validate_content_key(self, data: Any) -> None:\n        \"\"\"Check if content key is valid\"\"\"\n        sample = data.first()\n        if not isinstance(sample, dict):\n            raise ValueError(\n                f\"Expected the jq schema to result in a list of objects (dict), \\\n                    so sample must be a dict but got `{type(sample)}`\"\n            )\n        if sample.get(self._content_key) is None:\n            raise ValueError(\n                f\"Expected the jq schema to result in a list of objects (dict) \\\n                    with the key `{self._content_key}`\"\n            )\n        if self._metadata_func is not None:\n            sample_metadata = self._metadata_func(sample, {})\n            if not isinstance(sample_metadata, dict):\n                raise ValueError(\n                    f\"Expected the metadata_func to return a dict but got \\\n                        `{type(sample_metadata)}`\"\n                )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/json_loader.html"}1001{"id": "7ea76d01d0e6-0", "text": "Source code for langchain.document_loaders.weather\n\"\"\"Simple reader that reads weather data from OpenWeatherMap API\"\"\"\nfrom __future__ import annotations\nfrom datetime import datetime\nfrom typing import Iterator, List, Optional, Sequence\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utilities.openweathermap import OpenWeatherMapAPIWrapper\n[docs]class WeatherDataLoader(BaseLoader):\n    \"\"\"Weather Reader.\n    Reads the forecast & current weather of any location using OpenWeatherMap's free\n    API. Checkout 'https://openweathermap.org/appid' for more on how to generate a free\n    OpenWeatherMap API.\n    \"\"\"\n    def __init__(\n        self,\n        client: OpenWeatherMapAPIWrapper,\n        places: Sequence[str],\n    ) -> None:\n        \"\"\"Initialize with parameters.\"\"\"\n        super().__init__()\n        self.client = client\n        self.places = places\n[docs]    @classmethod\n    def from_params(\n        cls, places: Sequence[str], *, openweathermap_api_key: Optional[str] = None\n    ) -> WeatherDataLoader:\n        client = OpenWeatherMapAPIWrapper(openweathermap_api_key=openweathermap_api_key)\n        return cls(client, places)\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Lazily load weather data for the given locations.\"\"\"\n        for place in self.places:\n            metadata = {\"queried_at\": datetime.now()}\n            content = self.client.run(place)\n            yield Document(page_content=content, metadata=metadata)\n[docs]    def load(\n        self,\n    ) -> List[Document]:\n        \"\"\"Load weather data for the given locations.\"\"\"\n        return list(self.lazy_load())\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/weather.html"}1002{"id": "7ea76d01d0e6-1", "text": "return list(self.lazy_load())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/weather.html"}1003{"id": "171e82629da4-0", "text": "Source code for langchain.document_loaders.url\n\"\"\"Loader that uses unstructured to load HTML files.\"\"\"\nimport logging\nfrom typing import Any, List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\n[docs]class UnstructuredURLLoader(BaseLoader):\n    \"\"\"Loader that uses unstructured to load HTML files.\"\"\"\n    def __init__(\n        self,\n        urls: List[str],\n        continue_on_failure: bool = True,\n        mode: str = \"single\",\n        **unstructured_kwargs: Any,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            import unstructured  # noqa:F401\n            from unstructured.__version__ import __version__ as __unstructured_version__\n            self.__version = __unstructured_version__\n        except ImportError:\n            raise ValueError(\n                \"unstructured package not found, please install it with \"\n                \"`pip install unstructured`\"\n            )\n        self._validate_mode(mode)\n        self.mode = mode\n        headers = unstructured_kwargs.pop(\"headers\", {})\n        if len(headers.keys()) != 0:\n            warn_about_headers = False\n            if self.__is_non_html_available():\n                warn_about_headers = not self.__is_headers_available_for_non_html()\n            else:\n                warn_about_headers = not self.__is_headers_available_for_html()\n            if warn_about_headers:\n                logger.warning(\n                    \"You are using an old version of unstructured. \"\n                    \"The headers parameter is ignored\"\n                )\n        self.urls = urls\n        self.continue_on_failure = continue_on_failure\n        self.headers = headers\n        self.unstructured_kwargs = unstructured_kwargs\n    def _validate_mode(self, mode: str) -> None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url.html"}1004{"id": "171e82629da4-1", "text": "def _validate_mode(self, mode: str) -> None:\n        _valid_modes = {\"single\", \"elements\"}\n        if mode not in _valid_modes:\n            raise ValueError(\n                f\"Got {mode} for `mode`, but should be one of `{_valid_modes}`\"\n            )\n    def __is_headers_available_for_html(self) -> bool:\n        _unstructured_version = self.__version.split(\"-\")[0]\n        unstructured_version = tuple([int(x) for x in _unstructured_version.split(\".\")])\n        return unstructured_version >= (0, 5, 7)\n    def __is_headers_available_for_non_html(self) -> bool:\n        _unstructured_version = self.__version.split(\"-\")[0]\n        unstructured_version = tuple([int(x) for x in _unstructured_version.split(\".\")])\n        return unstructured_version >= (0, 5, 13)\n    def __is_non_html_available(self) -> bool:\n        _unstructured_version = self.__version.split(\"-\")[0]\n        unstructured_version = tuple([int(x) for x in _unstructured_version.split(\".\")])\n        return unstructured_version >= (0, 5, 12)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        from unstructured.partition.auto import partition\n        from unstructured.partition.html import partition_html\n        docs: List[Document] = list()\n        for url in self.urls:\n            try:\n                if self.__is_non_html_available():\n                    if self.__is_headers_available_for_non_html():\n                        elements = partition(\n                            url=url, headers=self.headers, **self.unstructured_kwargs\n                        )\n                    else:\n                        elements = partition(url=url, **self.unstructured_kwargs)\n                else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url.html"}1005{"id": "171e82629da4-2", "text": "elements = partition(url=url, **self.unstructured_kwargs)\n                else:\n                    if self.__is_headers_available_for_html():\n                        elements = partition_html(\n                            url=url, headers=self.headers, **self.unstructured_kwargs\n                        )\n                    else:\n                        elements = partition_html(url=url, **self.unstructured_kwargs)\n            except Exception as e:\n                if self.continue_on_failure:\n                    logger.error(f\"Error fetching or processing {url}, exeption: {e}\")\n                    continue\n                else:\n                    raise e\n            if self.mode == \"single\":\n                text = \"\\n\\n\".join([str(el) for el in elements])\n                metadata = {\"source\": url}\n                docs.append(Document(page_content=text, metadata=metadata))\n            elif self.mode == \"elements\":\n                for element in elements:\n                    metadata = element.metadata.to_dict()\n                    metadata[\"category\"] = element.category\n                    docs.append(Document(page_content=str(element), metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/url.html"}1006{"id": "e2a02bda4804-0", "text": "Source code for langchain.document_loaders.imsdb\n\"\"\"Loader that loads IMSDb.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class IMSDbLoader(WebBaseLoader):\n    \"\"\"Loader that loads IMSDb webpages.\"\"\"\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load webpage.\"\"\"\n        soup = self.scrape()\n        text = soup.select_one(\"td[class='scrtext']\").text\n        metadata = {\"source\": self.web_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/imsdb.html"}1007{"id": "289f7d9801c0-0", "text": "Source code for langchain.document_loaders.unstructured\n\"\"\"Loader that uses unstructured to load files.\"\"\"\nimport collections\nfrom abc import ABC, abstractmethod\nfrom typing import IO, Any, List, Sequence, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\ndef satisfies_min_unstructured_version(min_version: str) -> bool:\n    \"\"\"Checks to see if the installed unstructured version exceeds the minimum version\n    for the feature in question.\"\"\"\n    from unstructured.__version__ import __version__ as __unstructured_version__\n    min_version_tuple = tuple([int(x) for x in min_version.split(\".\")])\n    # NOTE(MthwRobinson) - enables the loader to work when you're using pre-release\n    # versions of unstructured like 0.4.17-dev1\n    _unstructured_version = __unstructured_version__.split(\"-\")[0]\n    unstructured_version_tuple = tuple(\n        [int(x) for x in _unstructured_version.split(\".\")]\n    )\n    return unstructured_version_tuple >= min_version_tuple\ndef validate_unstructured_version(min_unstructured_version: str) -> None:\n    \"\"\"Raises an error if the unstructured version does not exceed the\n    specified minimum.\"\"\"\n    if not satisfies_min_unstructured_version(min_unstructured_version):\n        raise ValueError(\n            f\"unstructured>={min_unstructured_version} is required in this loader.\"\n        )\nclass UnstructuredBaseLoader(BaseLoader, ABC):\n    \"\"\"Loader that uses unstructured to load files.\"\"\"\n    def __init__(self, mode: str = \"single\", **unstructured_kwargs: Any):\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            import unstructured  # noqa:F401\n        except ImportError:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/unstructured.html"}1008{"id": "289f7d9801c0-1", "text": "import unstructured  # noqa:F401\n        except ImportError:\n            raise ValueError(\n                \"unstructured package not found, please install it with \"\n                \"`pip install unstructured`\"\n            )\n        _valid_modes = {\"single\", \"elements\"}\n        if mode not in _valid_modes:\n            raise ValueError(\n                f\"Got {mode} for `mode`, but should be one of `{_valid_modes}`\"\n            )\n        self.mode = mode\n        if not satisfies_min_unstructured_version(\"0.5.4\"):\n            if \"strategy\" in unstructured_kwargs:\n                unstructured_kwargs.pop(\"strategy\")\n        self.unstructured_kwargs = unstructured_kwargs\n    @abstractmethod\n    def _get_elements(self) -> List:\n        \"\"\"Get elements.\"\"\"\n    @abstractmethod\n    def _get_metadata(self) -> dict:\n        \"\"\"Get metadata.\"\"\"\n    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        elements = self._get_elements()\n        if self.mode == \"elements\":\n            docs: List[Document] = list()\n            for element in elements:\n                metadata = self._get_metadata()\n                # NOTE(MthwRobinson) - the attribute check is for backward compatibility\n                # with unstructured<0.4.9. The metadata attributed was added in 0.4.9.\n                if hasattr(element, \"metadata\"):\n                    metadata.update(element.metadata.to_dict())\n                if hasattr(element, \"category\"):\n                    metadata[\"category\"] = element.category\n                docs.append(Document(page_content=str(element), metadata=metadata))\n        elif self.mode == \"single\":\n            metadata = self._get_metadata()\n            text = \"\\n\\n\".join([str(el) for el in elements])\n            docs = [Document(page_content=text, metadata=metadata)]", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/unstructured.html"}1009{"id": "289f7d9801c0-2", "text": "docs = [Document(page_content=text, metadata=metadata)]\n        else:\n            raise ValueError(f\"mode of {self.mode} not supported.\")\n        return docs\n[docs]class UnstructuredFileLoader(UnstructuredBaseLoader):\n    \"\"\"Loader that uses unstructured to load files.\"\"\"\n    def __init__(\n        self,\n        file_path: Union[str, List[str]],\n        mode: str = \"single\",\n        **unstructured_kwargs: Any,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        super().__init__(mode=mode, **unstructured_kwargs)\n    def _get_elements(self) -> List:\n        from unstructured.partition.auto import partition\n        return partition(filename=self.file_path, **self.unstructured_kwargs)\n    def _get_metadata(self) -> dict:\n        return {\"source\": self.file_path}\ndef get_elements_from_api(\n    file_path: Union[str, List[str], None] = None,\n    file: Union[IO, Sequence[IO], None] = None,\n    api_url: str = \"https://api.unstructured.io/general/v0/general\",\n    api_key: str = \"\",\n    **unstructured_kwargs: Any,\n) -> List:\n    \"\"\"Retrieves a list of elements from the Unstructured API.\"\"\"\n    if isinstance(file, collections.abc.Sequence) or isinstance(file_path, list):\n        from unstructured.partition.api import partition_multiple_via_api\n        _doc_elements = partition_multiple_via_api(\n            filenames=file_path,\n            files=file,\n            api_key=api_key,\n            api_url=api_url,\n            **unstructured_kwargs,\n        )\n        elements = []\n        for _elements in _doc_elements:\n            elements.extend(_elements)\n        return elements\n    else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/unstructured.html"}1010{"id": "289f7d9801c0-3", "text": "elements.extend(_elements)\n        return elements\n    else:\n        from unstructured.partition.api import partition_via_api\n        return partition_via_api(\n            filename=file_path,\n            file=file,\n            api_key=api_key,\n            api_url=api_url,\n            **unstructured_kwargs,\n        )\n[docs]class UnstructuredAPIFileLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses the unstructured web API to load files.\"\"\"\n    def __init__(\n        self,\n        file_path: Union[str, List[str]] = \"\",\n        mode: str = \"single\",\n        url: str = \"https://api.unstructured.io/general/v0/general\",\n        api_key: str = \"\",\n        **unstructured_kwargs: Any,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        if isinstance(file_path, str):\n            validate_unstructured_version(min_unstructured_version=\"0.6.2\")\n        else:\n            validate_unstructured_version(min_unstructured_version=\"0.6.3\")\n        self.url = url\n        self.api_key = api_key\n        super().__init__(file_path=file_path, mode=mode, **unstructured_kwargs)\n    def _get_metadata(self) -> dict:\n        return {\"source\": self.file_path}\n    def _get_elements(self) -> List:\n        return get_elements_from_api(\n            file_path=self.file_path,\n            api_key=self.api_key,\n            api_url=self.url,\n            **self.unstructured_kwargs,\n        )\n[docs]class UnstructuredFileIOLoader(UnstructuredBaseLoader):\n    \"\"\"Loader that uses unstructured to load file IO objects.\"\"\"\n    def __init__(\n        self,\n        file: Union[IO, Sequence[IO]],\n        mode: str = \"single\",", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/unstructured.html"}1011{"id": "289f7d9801c0-4", "text": "file: Union[IO, Sequence[IO]],\n        mode: str = \"single\",\n        **unstructured_kwargs: Any,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file = file\n        super().__init__(mode=mode, **unstructured_kwargs)\n    def _get_elements(self) -> List:\n        from unstructured.partition.auto import partition\n        return partition(file=self.file, **self.unstructured_kwargs)\n    def _get_metadata(self) -> dict:\n        return {}\n[docs]class UnstructuredAPIFileIOLoader(UnstructuredFileIOLoader):\n    \"\"\"Loader that uses the unstructured web API to load file IO objects.\"\"\"\n    def __init__(\n        self,\n        file: Union[IO, Sequence[IO]],\n        mode: str = \"single\",\n        url: str = \"https://api.unstructured.io/general/v0/general\",\n        api_key: str = \"\",\n        **unstructured_kwargs: Any,\n    ):\n        \"\"\"Initialize with file path.\"\"\"\n        if isinstance(file, collections.abc.Sequence):\n            validate_unstructured_version(min_unstructured_version=\"0.6.3\")\n        if file:\n            validate_unstructured_version(min_unstructured_version=\"0.6.2\")\n        self.url = url\n        self.api_key = api_key\n        super().__init__(file=file, mode=mode, **unstructured_kwargs)\n    def _get_elements(self) -> List:\n        return get_elements_from_api(\n            file=self.file,\n            api_key=self.api_key,\n            api_url=self.url,\n            **self.unstructured_kwargs,\n        )\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/unstructured.html"}1012{"id": "9c316ba69d6f-0", "text": "Source code for langchain.document_loaders.pdf\n\"\"\"Loader that loads PDF files.\"\"\"\nimport json\nimport logging\nimport os\nimport tempfile\nimport time\nfrom abc import ABC\nfrom io import StringIO\nfrom pathlib import Path\nfrom typing import Any, Iterator, List, Mapping, Optional\nfrom urllib.parse import urlparse\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.blob_loaders import Blob\nfrom langchain.document_loaders.parsers.pdf import (\n    PDFMinerParser,\n    PDFPlumberParser,\n    PyMuPDFParser,\n    PyPDFium2Parser,\n    PyPDFParser,\n)\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__file__)\n[docs]class UnstructuredPDFLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load PDF files.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.partition.pdf import partition_pdf\n        return partition_pdf(filename=self.file_path, **self.unstructured_kwargs)\nclass BasePDFLoader(BaseLoader, ABC):\n    \"\"\"Base loader class for PDF files.\n    Defaults to check for local file, but if the file is a web path, it will download it\n    to a temporary file, and use that, then clean up the temporary file after completion\n    \"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        self.web_path = None\n        if \"~\" in self.file_path:\n            self.file_path = os.path.expanduser(self.file_path)\n        # If the file is a web path, download it to a temporary file, and use that", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1013{"id": "9c316ba69d6f-1", "text": "# If the file is a web path, download it to a temporary file, and use that\n        if not os.path.isfile(self.file_path) and self._is_valid_url(self.file_path):\n            r = requests.get(self.file_path)\n            if r.status_code != 200:\n                raise ValueError(\n                    \"Check the url of your file; returned status code %s\"\n                    % r.status_code\n                )\n            self.web_path = self.file_path\n            self.temp_file = tempfile.NamedTemporaryFile()\n            self.temp_file.write(r.content)\n            self.file_path = self.temp_file.name\n        elif not os.path.isfile(self.file_path):\n            raise ValueError(\"File path %s is not a valid file or url\" % self.file_path)\n    def __del__(self) -> None:\n        if hasattr(self, \"temp_file\"):\n            self.temp_file.close()\n    @staticmethod\n    def _is_valid_url(url: str) -> bool:\n        \"\"\"Check if the url is valid.\"\"\"\n        parsed = urlparse(url)\n        return bool(parsed.netloc) and bool(parsed.scheme)\n    @property\n    def source(self) -> str:\n        return self.web_path if self.web_path is not None else self.file_path\n[docs]class OnlinePDFLoader(BasePDFLoader):\n    \"\"\"Loader that loads online PDFs.\"\"\"\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        loader = UnstructuredPDFLoader(str(self.file_path))\n        return loader.load()\n[docs]class PyPDFLoader(BasePDFLoader):\n    \"\"\"Loads a PDF with pypdf and chunks at character level.\n    Loader also stores page numbers in metadatas.\n    \"\"\"\n    def __init__(self, file_path: str) -> None:\n        \"\"\"Initialize with file path.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1014{"id": "9c316ba69d6f-2", "text": "\"\"\"Initialize with file path.\"\"\"\n        try:\n            import pypdf  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"pypdf package not found, please install it with \" \"`pip install pypdf`\"\n            )\n        self.parser = PyPDFParser()\n        super().__init__(file_path)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load given path as pages.\"\"\"\n        return list(self.lazy_load())\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Lazy load given path as pages.\"\"\"\n        blob = Blob.from_path(self.file_path)\n        yield from self.parser.parse(blob)\n[docs]class PyPDFium2Loader(BasePDFLoader):\n    \"\"\"Loads a PDF with pypdfium2 and chunks at character level.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        super().__init__(file_path)\n        self.parser = PyPDFium2Parser()\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load given path as pages.\"\"\"\n        return list(self.lazy_load())\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Lazy load given path as pages.\"\"\"\n        blob = Blob.from_path(self.file_path)\n        yield from self.parser.parse(blob)\n[docs]class PyPDFDirectoryLoader(BaseLoader):\n    \"\"\"Loads a directory with PDF files with pypdf and chunks at character level.\n    Loader also stores page numbers in metadatas.\n    \"\"\"\n    def __init__(\n        self,\n        path: str,\n        glob: str = \"**/[!.]*.pdf\",\n        silent_errors: bool = False,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1015{"id": "9c316ba69d6f-3", "text": "silent_errors: bool = False,\n        load_hidden: bool = False,\n        recursive: bool = False,\n    ):\n        self.path = path\n        self.glob = glob\n        self.load_hidden = load_hidden\n        self.recursive = recursive\n        self.silent_errors = silent_errors\n    @staticmethod\n    def _is_visible(path: Path) -> bool:\n        return not any(part.startswith(\".\") for part in path.parts)\n[docs]    def load(self) -> List[Document]:\n        p = Path(self.path)\n        docs = []\n        items = p.rglob(self.glob) if self.recursive else p.glob(self.glob)\n        for i in items:\n            if i.is_file():\n                if self._is_visible(i.relative_to(p)) or self.load_hidden:\n                    try:\n                        loader = PyPDFLoader(str(i))\n                        sub_docs = loader.load()\n                        for doc in sub_docs:\n                            doc.metadata[\"source\"] = str(i)\n                        docs.extend(sub_docs)\n                    except Exception as e:\n                        if self.silent_errors:\n                            logger.warning(e)\n                        else:\n                            raise e\n        return docs\n[docs]class PDFMinerLoader(BasePDFLoader):\n    \"\"\"Loader that uses PDFMiner to load PDF files.\"\"\"\n    def __init__(self, file_path: str) -> None:\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            from pdfminer.high_level import extract_text  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"`pdfminer` package not found, please install it with \"\n                \"`pip install pdfminer.six`\"\n            )\n        super().__init__(file_path)\n        self.parser = PDFMinerParser()\n[docs]    def load(self) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1016{"id": "9c316ba69d6f-4", "text": "[docs]    def load(self) -> List[Document]:\n        \"\"\"Eagerly load the content.\"\"\"\n        return list(self.lazy_load())\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Lazily lod documents.\"\"\"\n        blob = Blob.from_path(self.file_path)\n        yield from self.parser.parse(blob)\n[docs]class PDFMinerPDFasHTMLLoader(BasePDFLoader):\n    \"\"\"Loader that uses PDFMiner to load PDF files as HTML content.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            from pdfminer.high_level import extract_text_to_fp  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"`pdfminer` package not found, please install it with \"\n                \"`pip install pdfminer.six`\"\n            )\n        super().__init__(file_path)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        from pdfminer.high_level import extract_text_to_fp\n        from pdfminer.layout import LAParams\n        from pdfminer.utils import open_filename\n        output_string = StringIO()\n        with open_filename(self.file_path, \"rb\") as fp:\n            extract_text_to_fp(\n                fp,  # type: ignore[arg-type]\n                output_string,\n                codec=\"\",\n                laparams=LAParams(),\n                output_type=\"html\",\n            )\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=output_string.getvalue(), metadata=metadata)]\n[docs]class PyMuPDFLoader(BasePDFLoader):\n    \"\"\"Loader that uses PyMuPDF to load PDF files.\"\"\"\n    def __init__(self, file_path: str) -> None:\n        \"\"\"Initialize with file path.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1017{"id": "9c316ba69d6f-5", "text": "\"\"\"Initialize with file path.\"\"\"\n        try:\n            import fitz  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"`PyMuPDF` package not found, please install it with \"\n                \"`pip install pymupdf`\"\n            )\n        super().__init__(file_path)\n[docs]    def load(self, **kwargs: Optional[Any]) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        parser = PyMuPDFParser(text_kwargs=kwargs)\n        blob = Blob.from_path(self.file_path)\n        return parser.parse(blob)\n# MathpixPDFLoader implementation taken largely from Daniel Gross's:\n# https://gist.github.com/danielgross/3ab4104e14faccc12b49200843adab21\n[docs]class MathpixPDFLoader(BasePDFLoader):\n    def __init__(\n        self,\n        file_path: str,\n        processed_file_format: str = \"mmd\",\n        max_wait_time_seconds: int = 500,\n        should_clean_pdf: bool = False,\n        **kwargs: Any,\n    ) -> None:\n        super().__init__(file_path)\n        self.mathpix_api_key = get_from_dict_or_env(\n            kwargs, \"mathpix_api_key\", \"MATHPIX_API_KEY\"\n        )\n        self.mathpix_api_id = get_from_dict_or_env(\n            kwargs, \"mathpix_api_id\", \"MATHPIX_API_ID\"\n        )\n        self.processed_file_format = processed_file_format\n        self.max_wait_time_seconds = max_wait_time_seconds\n        self.should_clean_pdf = should_clean_pdf\n    @property\n    def headers(self) -> dict:\n        return {\"app_id\": self.mathpix_api_id, \"app_key\": self.mathpix_api_key}\n    @property", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1018{"id": "9c316ba69d6f-6", "text": "@property\n    def url(self) -> str:\n        return \"https://api.mathpix.com/v3/pdf\"\n    @property\n    def data(self) -> dict:\n        options = {\"conversion_formats\": {self.processed_file_format: True}}\n        return {\"options_json\": json.dumps(options)}\n[docs]    def send_pdf(self) -> str:\n        with open(self.file_path, \"rb\") as f:\n            files = {\"file\": f}\n            response = requests.post(\n                self.url, headers=self.headers, files=files, data=self.data\n            )\n        response_data = response.json()\n        if \"pdf_id\" in response_data:\n            pdf_id = response_data[\"pdf_id\"]\n            return pdf_id\n        else:\n            raise ValueError(\"Unable to send PDF to Mathpix.\")\n[docs]    def wait_for_processing(self, pdf_id: str) -> None:\n        url = self.url + \"/\" + pdf_id\n        for _ in range(0, self.max_wait_time_seconds, 5):\n            response = requests.get(url, headers=self.headers)\n            response_data = response.json()\n            status = response_data.get(\"status\", None)\n            if status == \"completed\":\n                return\n            elif status == \"error\":\n                raise ValueError(\"Unable to retrieve PDF from Mathpix\")\n            else:\n                print(f\"Status: {status}, waiting for processing to complete\")\n                time.sleep(5)\n        raise TimeoutError\n[docs]    def get_processed_pdf(self, pdf_id: str) -> str:\n        self.wait_for_processing(pdf_id)\n        url = f\"{self.url}/{pdf_id}.{self.processed_file_format}\"\n        response = requests.get(url, headers=self.headers)\n        return response.content.decode(\"utf-8\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1019{"id": "9c316ba69d6f-7", "text": "return response.content.decode(\"utf-8\")\n[docs]    def clean_pdf(self, contents: str) -> str:\n        contents = \"\\n\".join(\n            [line for line in contents.split(\"\\n\") if not line.startswith(\"![]\")]\n        )\n        # replace \\section{Title} with # Title\n        contents = contents.replace(\"\\\\section{\", \"# \").replace(\"}\", \"\")\n        # replace the \"\\\" slash that Mathpix adds to escape $, %, (, etc.\n        contents = (\n            contents.replace(r\"\\$\", \"$\")\n            .replace(r\"\\%\", \"%\")\n            .replace(r\"\\(\", \"(\")\n            .replace(r\"\\)\", \")\")\n        )\n        return contents\n[docs]    def load(self) -> List[Document]:\n        pdf_id = self.send_pdf()\n        contents = self.get_processed_pdf(pdf_id)\n        if self.should_clean_pdf:\n            contents = self.clean_pdf(contents)\n        metadata = {\"source\": self.source, \"file_path\": self.source}\n        return [Document(page_content=contents, metadata=metadata)]\n[docs]class PDFPlumberLoader(BasePDFLoader):\n    \"\"\"Loader that uses pdfplumber to load PDF files.\"\"\"\n    def __init__(\n        self, file_path: str, text_kwargs: Optional[Mapping[str, Any]] = None\n    ) -> None:\n        \"\"\"Initialize with file path.\"\"\"\n        try:\n            import pdfplumber  # noqa:F401\n        except ImportError:\n            raise ImportError(\n                \"pdfplumber package not found, please install it with \"\n                \"`pip install pdfplumber`\"\n            )\n        super().__init__(file_path)\n        self.text_kwargs = text_kwargs or {}\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1020{"id": "9c316ba69d6f-8", "text": "[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        parser = PDFPlumberParser(text_kwargs=self.text_kwargs)\n        blob = Blob.from_path(self.file_path)\n        return parser.parse(blob)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/pdf.html"}1021{"id": "74615fc19450-0", "text": "Source code for langchain.document_loaders.telegram\n\"\"\"Loader that loads Telegram chat json dump.\"\"\"\nfrom __future__ import annotations\nimport asyncio\nimport json\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING, Dict, List, Optional, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nif TYPE_CHECKING:\n    import pandas as pd\n    from telethon.hints import EntityLike\ndef concatenate_rows(row: dict) -> str:\n    \"\"\"Combine message information in a readable format ready to be used.\"\"\"\n    date = row[\"date\"]\n    sender = row[\"from\"]\n    text = row[\"text\"]\n    return f\"{sender} on {date}: {text}\\n\\n\"\n[docs]class TelegramChatFileLoader(BaseLoader):\n    \"\"\"Loader that loads Telegram chat json directory dump.\"\"\"\n    def __init__(self, path: str):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        p = Path(self.file_path)\n        with open(p, encoding=\"utf8\") as f:\n            d = json.load(f)\n        text = \"\".join(\n            concatenate_rows(message)\n            for message in d[\"messages\"]\n            if message[\"type\"] == \"message\" and isinstance(message[\"text\"], str)\n        )\n        metadata = {\"source\": str(p)}\n        return [Document(page_content=text, metadata=metadata)]\ndef text_to_docs(text: Union[str, List[str]]) -> List[Document]:\n    \"\"\"Converts a string or list of strings to a list of Documents with metadata.\"\"\"\n    if isinstance(text, str):\n        # Take a single string as one page", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1022{"id": "74615fc19450-1", "text": "if isinstance(text, str):\n        # Take a single string as one page\n        text = [text]\n    page_docs = [Document(page_content=page) for page in text]\n    # Add page numbers as metadata\n    for i, doc in enumerate(page_docs):\n        doc.metadata[\"page\"] = i + 1\n    # Split pages into chunks\n    doc_chunks = []\n    for doc in page_docs:\n        text_splitter = RecursiveCharacterTextSplitter(\n            chunk_size=800,\n            separators=[\"\\n\\n\", \"\\n\", \".\", \"!\", \"?\", \",\", \" \", \"\"],\n            chunk_overlap=20,\n        )\n        chunks = text_splitter.split_text(doc.page_content)\n        for i, chunk in enumerate(chunks):\n            doc = Document(\n                page_content=chunk, metadata={\"page\": doc.metadata[\"page\"], \"chunk\": i}\n            )\n            # Add sources a metadata\n            doc.metadata[\"source\"] = f\"{doc.metadata['page']}-{doc.metadata['chunk']}\"\n            doc_chunks.append(doc)\n    return doc_chunks\n[docs]class TelegramChatApiLoader(BaseLoader):\n    \"\"\"Loader that loads Telegram chat json directory dump.\"\"\"\n    def __init__(\n        self,\n        chat_entity: Optional[EntityLike] = None,\n        api_id: Optional[int] = None,\n        api_hash: Optional[str] = None,\n        username: Optional[str] = None,\n        file_path: str = \"telegram_data.json\",\n    ):\n        \"\"\"Initialize with API parameters.\"\"\"\n        self.chat_entity = chat_entity\n        self.api_id = api_id\n        self.api_hash = api_hash\n        self.username = username\n        self.file_path = file_path\n[docs]    async def fetch_data_from_telegram(self) -> None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1023{"id": "74615fc19450-2", "text": "[docs]    async def fetch_data_from_telegram(self) -> None:\n        \"\"\"Fetch data from Telegram API and save it as a JSON file.\"\"\"\n        from telethon.sync import TelegramClient\n        data = []\n        async with TelegramClient(self.username, self.api_id, self.api_hash) as client:\n            async for message in client.iter_messages(self.chat_entity):\n                is_reply = message.reply_to is not None\n                reply_to_id = message.reply_to.reply_to_msg_id if is_reply else None\n                data.append(\n                    {\n                        \"sender_id\": message.sender_id,\n                        \"text\": message.text,\n                        \"date\": message.date.isoformat(),\n                        \"message.id\": message.id,\n                        \"is_reply\": is_reply,\n                        \"reply_to_id\": reply_to_id,\n                    }\n                )\n        with open(self.file_path, \"w\", encoding=\"utf-8\") as f:\n            json.dump(data, f, ensure_ascii=False, indent=4)\n    def _get_message_threads(self, data: pd.DataFrame) -> dict:\n        \"\"\"Create a dictionary of message threads from the given data.\n        Args:\n            data (pd.DataFrame): A DataFrame containing the conversation \\\n                data with columns:\n                - message.sender_id\n                - text\n                - date\n                - message.id\n                - is_reply\n                - reply_to_id\n        Returns:\n            dict: A dictionary where the key is the parent message ID and \\\n                the value is a list of message IDs in ascending order.\n        \"\"\"\n        def find_replies(parent_id: int, reply_data: pd.DataFrame) -> List[int]:\n            \"\"\"\n            Recursively find all replies to a given parent message ID.\n            Args:\n                parent_id (int): The parent message ID.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1024{"id": "74615fc19450-3", "text": "Args:\n                parent_id (int): The parent message ID.\n                reply_data (pd.DataFrame): A DataFrame containing reply messages.\n            Returns:\n                list: A list of message IDs that are replies to the parent message ID.\n            \"\"\"\n            # Find direct replies to the parent message ID\n            direct_replies = reply_data[reply_data[\"reply_to_id\"] == parent_id][\n                \"message.id\"\n            ].tolist()\n            # Recursively find replies to the direct replies\n            all_replies = []\n            for reply_id in direct_replies:\n                all_replies += [reply_id] + find_replies(reply_id, reply_data)\n            return all_replies\n        # Filter out parent messages\n        parent_messages = data[~data[\"is_reply\"]]\n        # Filter out reply messages and drop rows with NaN in 'reply_to_id'\n        reply_messages = data[data[\"is_reply\"]].dropna(subset=[\"reply_to_id\"])\n        # Convert 'reply_to_id' to integer\n        reply_messages[\"reply_to_id\"] = reply_messages[\"reply_to_id\"].astype(int)\n        # Create a dictionary of message threads with parent message IDs as keys and \\\n        # lists of reply message IDs as values\n        message_threads = {\n            parent_id: [parent_id] + find_replies(parent_id, reply_messages)\n            for parent_id in parent_messages[\"message.id\"]\n        }\n        return message_threads\n    def _combine_message_texts(\n        self, message_threads: Dict[int, List[int]], data: pd.DataFrame\n    ) -> str:\n        \"\"\"\n        Combine the message texts for each parent message ID based \\\n            on the list of message threads.\n        Args:\n            message_threads (dict): A dictionary where the key is the parent message \\", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1025{"id": "74615fc19450-4", "text": "Args:\n            message_threads (dict): A dictionary where the key is the parent message \\\n                ID and the value is a list of message IDs in ascending order.\n            data (pd.DataFrame): A DataFrame containing the conversation data:\n                - message.sender_id\n                - text\n                - date\n                - message.id\n                - is_reply\n                - reply_to_id\n        Returns:\n            str: A combined string of message texts sorted by date.\n        \"\"\"\n        combined_text = \"\"\n        # Iterate through sorted parent message IDs\n        for parent_id, message_ids in message_threads.items():\n            # Get the message texts for the message IDs and sort them by date\n            message_texts = (\n                data[data[\"message.id\"].isin(message_ids)]\n                .sort_values(by=\"date\")[\"text\"]\n                .tolist()\n            )\n            message_texts = [str(elem) for elem in message_texts]\n            # Combine the message texts\n            combined_text += \" \".join(message_texts) + \".\\n\"\n        return combined_text.strip()\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        if self.chat_entity is not None:\n            try:\n                import nest_asyncio\n                nest_asyncio.apply()\n                asyncio.run(self.fetch_data_from_telegram())\n            except ImportError:\n                raise ImportError(\n                    \"\"\"`nest_asyncio` package not found.\n                    please install with `pip install nest_asyncio`\n                    \"\"\"\n                )\n        p = Path(self.file_path)\n        with open(p, encoding=\"utf8\") as f:\n            d = json.load(f)\n        try:\n            import pandas as pd\n        except ImportError:\n            raise ImportError(\n                \"\"\"`pandas` package not found. \n                please install with `pip install pandas`", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1026{"id": "74615fc19450-5", "text": "please install with `pip install pandas`\n                \"\"\"\n            )\n        normalized_messages = pd.json_normalize(d)\n        df = pd.DataFrame(normalized_messages)\n        message_threads = self._get_message_threads(df)\n        combined_texts = self._combine_message_texts(message_threads, df)\n        return text_to_docs(combined_texts)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html"}1027{"id": "1f6418546062-0", "text": "Source code for langchain.document_loaders.obsidian\n\"\"\"Loader that loads Obsidian directory dump.\"\"\"\nimport re\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class ObsidianLoader(BaseLoader):\n    \"\"\"Loader that loads Obsidian files from disk.\"\"\"\n    FRONT_MATTER_REGEX = re.compile(r\"^---\\n(.*?)\\n---\\n\", re.MULTILINE | re.DOTALL)\n    def __init__(\n        self, path: str, encoding: str = \"UTF-8\", collect_metadata: bool = True\n    ):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n        self.encoding = encoding\n        self.collect_metadata = collect_metadata\n    def _parse_front_matter(self, content: str) -> dict:\n        \"\"\"Parse front matter metadata from the content and return it as a dict.\"\"\"\n        if not self.collect_metadata:\n            return {}\n        match = self.FRONT_MATTER_REGEX.search(content)\n        front_matter = {}\n        if match:\n            lines = match.group(1).split(\"\\n\")\n            for line in lines:\n                if \":\" in line:\n                    key, value = line.split(\":\", 1)\n                    front_matter[key.strip()] = value.strip()\n                else:\n                    # Skip lines without a colon\n                    continue\n        return front_matter\n    def _remove_front_matter(self, content: str) -> str:\n        \"\"\"Remove front matter metadata from the given content.\"\"\"\n        if not self.collect_metadata:\n            return content\n        return self.FRONT_MATTER_REGEX.sub(\"\", content)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        ps = list(Path(self.file_path).glob(\"**/*.md\"))", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/obsidian.html"}1028{"id": "1f6418546062-1", "text": "\"\"\"Load documents.\"\"\"\n        ps = list(Path(self.file_path).glob(\"**/*.md\"))\n        docs = []\n        for p in ps:\n            with open(p, encoding=self.encoding) as f:\n                text = f.read()\n            front_matter = self._parse_front_matter(text)\n            text = self._remove_front_matter(text)\n            metadata = {\n                \"source\": str(p.name),\n                \"path\": str(p),\n                \"created\": p.stat().st_ctime,\n                \"last_modified\": p.stat().st_mtime,\n                \"last_accessed\": p.stat().st_atime,\n                **front_matter,\n            }\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/obsidian.html"}1029{"id": "c4ef427dac41-0", "text": "Source code for langchain.document_loaders.youtube\n\"\"\"Loader that loads YouTube transcript.\"\"\"\nfrom __future__ import annotations\nimport logging\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional\nfrom urllib.parse import parse_qs, urlparse\nfrom pydantic import root_validator\nfrom pydantic.dataclasses import dataclass\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nlogger = logging.getLogger(__name__)\nSCOPES = [\"https://www.googleapis.com/auth/youtube.readonly\"]\n[docs]@dataclass\nclass GoogleApiClient:\n    \"\"\"A Generic Google Api Client.\n    To use, you should have the ``google_auth_oauthlib,youtube_transcript_api,google``\n    python package installed.\n    As the google api expects credentials you need to set up a google account and\n    register your Service. \"https://developers.google.com/docs/api/quickstart/python\"\n    Example:\n        .. code-block:: python\n            from langchain.document_loaders import GoogleApiClient\n            google_api_client = GoogleApiClient(\n                service_account_path=Path(\"path_to_your_sec_file.json\")\n            )\n    \"\"\"\n    credentials_path: Path = Path.home() / \".credentials\" / \"credentials.json\"\n    service_account_path: Path = Path.home() / \".credentials\" / \"credentials.json\"\n    token_path: Path = Path.home() / \".credentials\" / \"token.json\"\n    def __post_init__(self) -> None:\n        self.creds = self._load_credentials()\n[docs]    @root_validator\n    def validate_channel_or_videoIds_is_set(\n        cls, values: Dict[str, Any]\n    ) -> Dict[str, Any]:\n        \"\"\"Validate that either folder_id or document_ids is set, but not both.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1030{"id": "c4ef427dac41-1", "text": "\"\"\"Validate that either folder_id or document_ids is set, but not both.\"\"\"\n        if not values.get(\"credentials_path\") and not values.get(\n            \"service_account_path\"\n        ):\n            raise ValueError(\"Must specify either channel_name or video_ids\")\n        return values\n    def _load_credentials(self) -> Any:\n        \"\"\"Load credentials.\"\"\"\n        # Adapted from https://developers.google.com/drive/api/v3/quickstart/python\n        try:\n            from google.auth.transport.requests import Request\n            from google.oauth2 import service_account\n            from google.oauth2.credentials import Credentials\n            from google_auth_oauthlib.flow import InstalledAppFlow\n            from youtube_transcript_api import YouTubeTranscriptApi  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"You must run\"\n                \"`pip install --upgrade \"\n                \"google-api-python-client google-auth-httplib2 \"\n                \"google-auth-oauthlib \"\n                \"youtube-transcript-api` \"\n                \"to use the Google Drive loader\"\n            )\n        creds = None\n        if self.service_account_path.exists():\n            return service_account.Credentials.from_service_account_file(\n                str(self.service_account_path)\n            )\n        if self.token_path.exists():\n            creds = Credentials.from_authorized_user_file(str(self.token_path), SCOPES)\n        if not creds or not creds.valid:\n            if creds and creds.expired and creds.refresh_token:\n                creds.refresh(Request())\n            else:\n                flow = InstalledAppFlow.from_client_secrets_file(\n                    str(self.credentials_path), SCOPES\n                )\n                creds = flow.run_local_server(port=0)\n            with open(self.token_path, \"w\") as token:\n                token.write(creds.to_json())\n        return creds", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1031{"id": "c4ef427dac41-2", "text": "token.write(creds.to_json())\n        return creds\nALLOWED_SCHEMAS = {\"http\", \"https\"}\nALLOWED_NETLOCK = {\n    \"youtu.be\",\n    \"m.youtube.com\",\n    \"youtube.com\",\n    \"www.youtube.com\",\n    \"www.youtube-nocookie.com\",\n    \"vid.plus\",\n}\ndef _parse_video_id(url: str) -> Optional[str]:\n    \"\"\"Parse a youtube url and return the video id if valid, otherwise None.\"\"\"\n    parsed_url = urlparse(url)\n    if parsed_url.scheme not in ALLOWED_SCHEMAS:\n        return None\n    if parsed_url.netloc not in ALLOWED_NETLOCK:\n        return None\n    path = parsed_url.path\n    if path.endswith(\"/watch\"):\n        query = parsed_url.query\n        parsed_query = parse_qs(query)\n        if \"v\" in parsed_query:\n            ids = parsed_query[\"v\"]\n            video_id = ids if isinstance(ids, str) else ids[0]\n        else:\n            return None\n    else:\n        path = parsed_url.path.lstrip(\"/\")\n        video_id = path.split(\"/\")[-1]\n    if len(video_id) != 11:  # Video IDs are 11 characters long\n        return None\n    return video_id\n[docs]class YoutubeLoader(BaseLoader):\n    \"\"\"Loader that loads Youtube transcripts.\"\"\"\n    def __init__(\n        self,\n        video_id: str,\n        add_video_info: bool = False,\n        language: str = \"en\",\n        continue_on_failure: bool = False,\n    ):\n        \"\"\"Initialize with YouTube video ID.\"\"\"\n        self.video_id = video_id\n        self.add_video_info = add_video_info\n        self.language = language\n        self.continue_on_failure = continue_on_failure\n[docs]    @staticmethod", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1032{"id": "c4ef427dac41-3", "text": "self.continue_on_failure = continue_on_failure\n[docs]    @staticmethod\n    def extract_video_id(youtube_url: str) -> str:\n        \"\"\"Extract video id from common YT urls.\"\"\"\n        video_id = _parse_video_id(youtube_url)\n        if not video_id:\n            raise ValueError(\n                f\"Could not determine the video ID for the URL {youtube_url}\"\n            )\n        return video_id\n[docs]    @classmethod\n    def from_youtube_url(cls, youtube_url: str, **kwargs: Any) -> YoutubeLoader:\n        \"\"\"Given youtube URL, load video.\"\"\"\n        video_id = cls.extract_video_id(youtube_url)\n        return cls(video_id, **kwargs)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            from youtube_transcript_api import (\n                NoTranscriptFound,\n                TranscriptsDisabled,\n                YouTubeTranscriptApi,\n            )\n        except ImportError:\n            raise ImportError(\n                \"Could not import youtube_transcript_api python package. \"\n                \"Please install it with `pip install youtube-transcript-api`.\"\n            )\n        metadata = {\"source\": self.video_id}\n        if self.add_video_info:\n            # Get more video meta info\n            # Such as title, description, thumbnail url, publish_date\n            video_info = self._get_video_info()\n            metadata.update(video_info)\n        try:\n            transcript_list = YouTubeTranscriptApi.list_transcripts(self.video_id)\n        except TranscriptsDisabled:\n            return []\n        try:\n            transcript = transcript_list.find_transcript([self.language])\n        except NoTranscriptFound:\n            en_transcript = transcript_list.find_transcript([\"en\"])\n            transcript = en_transcript.translate(self.language)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1033{"id": "c4ef427dac41-4", "text": "transcript = en_transcript.translate(self.language)\n        transcript_pieces = transcript.fetch()\n        transcript = \" \".join([t[\"text\"].strip(\" \") for t in transcript_pieces])\n        return [Document(page_content=transcript, metadata=metadata)]\n    def _get_video_info(self) -> dict:\n        \"\"\"Get important video information.\n        Components are:\n            - title\n            - description\n            - thumbnail url,\n            - publish_date\n            - channel_author\n            - and more.\n        \"\"\"\n        try:\n            from pytube import YouTube\n        except ImportError:\n            raise ImportError(\n                \"Could not import pytube python package. \"\n                \"Please install it with `pip install pytube`.\"\n            )\n        yt = YouTube(f\"https://www.youtube.com/watch?v={self.video_id}\")\n        video_info = {\n            \"title\": yt.title,\n            \"description\": yt.description,\n            \"view_count\": yt.views,\n            \"thumbnail_url\": yt.thumbnail_url,\n            \"publish_date\": yt.publish_date,\n            \"length\": yt.length,\n            \"author\": yt.author,\n        }\n        return video_info\n[docs]@dataclass\nclass GoogleApiYoutubeLoader(BaseLoader):\n    \"\"\"Loader that loads all Videos from a Channel\n    To use, you should have the ``googleapiclient,youtube_transcript_api``\n    python package installed.\n    As the service needs a google_api_client, you first have to initialize\n    the GoogleApiClient.\n    Additionally you have to either provide a channel name or a list of videoids\n    \"https://developers.google.com/docs/api/quickstart/python\"\n    Example:\n        .. code-block:: python\n            from langchain.document_loaders import GoogleApiClient\n            from langchain.document_loaders import GoogleApiYoutubeLoader", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1034{"id": "c4ef427dac41-5", "text": "from langchain.document_loaders import GoogleApiYoutubeLoader\n            google_api_client = GoogleApiClient(\n                service_account_path=Path(\"path_to_your_sec_file.json\")\n            )\n            loader = GoogleApiYoutubeLoader(\n                google_api_client=google_api_client,\n                channel_name = \"CodeAesthetic\"\n            )\n            load.load()\n    \"\"\"\n    google_api_client: GoogleApiClient\n    channel_name: Optional[str] = None\n    video_ids: Optional[List[str]] = None\n    add_video_info: bool = True\n    captions_language: str = \"en\"\n    continue_on_failure: bool = False\n    def __post_init__(self) -> None:\n        self.youtube_client = self._build_youtube_client(self.google_api_client.creds)\n    def _build_youtube_client(self, creds: Any) -> Any:\n        try:\n            from googleapiclient.discovery import build\n            from youtube_transcript_api import YouTubeTranscriptApi  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"You must run\"\n                \"`pip install --upgrade \"\n                \"google-api-python-client google-auth-httplib2 \"\n                \"google-auth-oauthlib \"\n                \"youtube-transcript-api` \"\n                \"to use the Google Drive loader\"\n            )\n        return build(\"youtube\", \"v3\", credentials=creds)\n[docs]    @root_validator\n    def validate_channel_or_videoIds_is_set(\n        cls, values: Dict[str, Any]\n    ) -> Dict[str, Any]:\n        \"\"\"Validate that either folder_id or document_ids is set, but not both.\"\"\"\n        if not values.get(\"channel_name\") and not values.get(\"video_ids\"):\n            raise ValueError(\"Must specify either channel_name or video_ids\")\n        return values", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1035{"id": "c4ef427dac41-6", "text": "raise ValueError(\"Must specify either channel_name or video_ids\")\n        return values\n    def _get_transcripe_for_video_id(self, video_id: str) -> str:\n        from youtube_transcript_api import NoTranscriptFound, YouTubeTranscriptApi\n        transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)\n        try:\n            transcript = transcript_list.find_transcript([self.captions_language])\n        except NoTranscriptFound:\n            for available_transcript in transcript_list:\n                transcript = available_transcript.translate(self.captions_language)\n                continue\n        transcript_pieces = transcript.fetch()\n        return \" \".join([t[\"text\"].strip(\" \") for t in transcript_pieces])\n    def _get_document_for_video_id(self, video_id: str, **kwargs: Any) -> Document:\n        captions = self._get_transcripe_for_video_id(video_id)\n        video_response = (\n            self.youtube_client.videos()\n            .list(\n                part=\"id,snippet\",\n                id=video_id,\n            )\n            .execute()\n        )\n        return Document(\n            page_content=captions,\n            metadata=video_response.get(\"items\")[0],\n        )\n    def _get_channel_id(self, channel_name: str) -> str:\n        request = self.youtube_client.search().list(\n            part=\"id\",\n            q=channel_name,\n            type=\"channel\",\n            maxResults=1,  # we only need one result since channel names are unique\n        )\n        response = request.execute()\n        channel_id = response[\"items\"][0][\"id\"][\"channelId\"]\n        return channel_id\n    def _get_document_for_channel(self, channel: str, **kwargs: Any) -> List[Document]:\n        try:\n            from youtube_transcript_api import (", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1036{"id": "c4ef427dac41-7", "text": "try:\n            from youtube_transcript_api import (\n                NoTranscriptFound,\n                TranscriptsDisabled,\n            )\n        except ImportError:\n            raise ImportError(\n                \"You must run\"\n                \"`pip install --upgrade \"\n                \"youtube-transcript-api` \"\n                \"to use the youtube loader\"\n            )\n        channel_id = self._get_channel_id(channel)\n        request = self.youtube_client.search().list(\n            part=\"id,snippet\",\n            channelId=channel_id,\n            maxResults=50,  # adjust this value to retrieve more or fewer videos\n        )\n        video_ids = []\n        while request is not None:\n            response = request.execute()\n            # Add each video ID to the list\n            for item in response[\"items\"]:\n                if not item[\"id\"].get(\"videoId\"):\n                    continue\n                meta_data = {\"videoId\": item[\"id\"][\"videoId\"]}\n                if self.add_video_info:\n                    item[\"snippet\"].pop(\"thumbnails\")\n                    meta_data.update(item[\"snippet\"])\n                try:\n                    page_content = self._get_transcripe_for_video_id(\n                        item[\"id\"][\"videoId\"]\n                    )\n                    video_ids.append(\n                        Document(\n                            page_content=page_content,\n                            metadata=meta_data,\n                        )\n                    )\n                except (TranscriptsDisabled, NoTranscriptFound) as e:\n                    if self.continue_on_failure:\n                        logger.error(\n                            \"Error fetching transscript \"\n                            + f\" {item['id']['videoId']}, exception: {e}\"\n                        )\n                    else:\n                        raise e\n                    pass\n            request = self.youtube_client.search().list_next(request, response)\n        return video_ids\n[docs]    def load(self) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1037{"id": "c4ef427dac41-8", "text": "return video_ids\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        document_list = []\n        if self.channel_name:\n            document_list.extend(self._get_document_for_channel(self.channel_name))\n        elif self.video_ids:\n            document_list.extend(\n                [\n                    self._get_document_for_video_id(video_id)\n                    for video_id in self.video_ids\n                ]\n            )\n        else:\n            raise ValueError(\"Must specify either channel_name or video_ids\")\n        return document_list\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/youtube.html"}1038{"id": "fff8e16e66a9-0", "text": "Source code for langchain.document_loaders.notebook\n\"\"\"Loader that loads .ipynb notebook files.\"\"\"\nimport json\nfrom pathlib import Path\nfrom typing import Any, List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\ndef concatenate_cells(\n    cell: dict, include_outputs: bool, max_output_length: int, traceback: bool\n) -> str:\n    \"\"\"Combine cells information in a readable format ready to be used.\"\"\"\n    cell_type = cell[\"cell_type\"]\n    source = cell[\"source\"]\n    output = cell[\"outputs\"]\n    if include_outputs and cell_type == \"code\" and output:\n        if \"ename\" in output[0].keys():\n            error_name = output[0][\"ename\"]\n            error_value = output[0][\"evalue\"]\n            if traceback:\n                traceback = output[0][\"traceback\"]\n                return (\n                    f\"'{cell_type}' cell: '{source}'\\n, gives error '{error_name}',\"\n                    f\" with description '{error_value}'\\n\"\n                    f\"and traceback '{traceback}'\\n\\n\"\n                )\n            else:\n                return (\n                    f\"'{cell_type}' cell: '{source}'\\n, gives error '{error_name}',\"\n                    f\"with description '{error_value}'\\n\\n\"\n                )\n        elif output[0][\"output_type\"] == \"stream\":\n            output = output[0][\"text\"]\n            min_output = min(max_output_length, len(output))\n            return (\n                f\"'{cell_type}' cell: '{source}'\\n with \"\n                f\"output: '{output[:min_output]}'\\n\\n\"\n            )\n    else:\n        return f\"'{cell_type}' cell: '{source}'\\n\\n\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notebook.html"}1039{"id": "fff8e16e66a9-1", "text": "return f\"'{cell_type}' cell: '{source}'\\n\\n\"\n    return \"\"\ndef remove_newlines(x: Any) -> Any:\n    \"\"\"Remove recursively newlines, no matter the data structure they are stored in.\"\"\"\n    import pandas as pd\n    if isinstance(x, str):\n        return x.replace(\"\\n\", \"\")\n    elif isinstance(x, list):\n        return [remove_newlines(elem) for elem in x]\n    elif isinstance(x, pd.DataFrame):\n        return x.applymap(remove_newlines)\n    else:\n        return x\n[docs]class NotebookLoader(BaseLoader):\n    \"\"\"Loader that loads .ipynb notebook files.\"\"\"\n    def __init__(\n        self,\n        path: str,\n        include_outputs: bool = False,\n        max_output_length: int = 10,\n        remove_newline: bool = False,\n        traceback: bool = False,\n    ):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n        self.include_outputs = include_outputs\n        self.max_output_length = max_output_length\n        self.remove_newline = remove_newline\n        self.traceback = traceback\n[docs]    def load(\n        self,\n    ) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            import pandas as pd\n        except ImportError:\n            raise ImportError(\n                \"pandas is needed for Notebook Loader, \"\n                \"please install with `pip install pandas`\"\n            )\n        p = Path(self.file_path)\n        with open(p, encoding=\"utf8\") as f:\n            d = json.load(f)\n        data = pd.json_normalize(d[\"cells\"])\n        filtered_data = data[[\"cell_type\", \"source\", \"outputs\"]]\n        if self.remove_newline:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notebook.html"}1040{"id": "fff8e16e66a9-2", "text": "if self.remove_newline:\n            filtered_data = filtered_data.applymap(remove_newlines)\n        text = filtered_data.apply(\n            lambda x: concatenate_cells(\n                x, self.include_outputs, self.max_output_length, self.traceback\n            ),\n            axis=1,\n        ).str.cat(sep=\" \")\n        metadata = {\"source\": str(p)}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notebook.html"}1041{"id": "e18eb6713e9f-0", "text": "Source code for langchain.document_loaders.toml\nimport json\nfrom pathlib import Path\nfrom typing import Iterator, List, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class TomlLoader(BaseLoader):\n    \"\"\"\n    A TOML document loader that inherits from the BaseLoader class.\n    This class can be initialized with either a single source file or a source\n    directory containing TOML files.\n    \"\"\"\n    def __init__(self, source: Union[str, Path]):\n        \"\"\"Initialize the TomlLoader with a source file or directory.\"\"\"\n        self.source = Path(source)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load and return all documents.\"\"\"\n        return list(self.lazy_load())\n[docs]    def lazy_load(self) -> Iterator[Document]:\n        \"\"\"Lazily load the TOML documents from the source file or directory.\"\"\"\n        import tomli\n        if self.source.is_file() and self.source.suffix == \".toml\":\n            files = [self.source]\n        elif self.source.is_dir():\n            files = list(self.source.glob(\"**/*.toml\"))\n        else:\n            raise ValueError(\"Invalid source path or file type\")\n        for file_path in files:\n            with file_path.open(\"r\", encoding=\"utf-8\") as file:\n                content = file.read()\n                try:\n                    data = tomli.loads(content)\n                    doc = Document(\n                        page_content=json.dumps(data),\n                        metadata={\"source\": str(file_path)},\n                    )\n                    yield doc\n                except tomli.TOMLDecodeError as e:\n                    print(f\"Error parsing TOML file {file_path}: {e}\")\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/toml.html"}1042{"id": "e18eb6713e9f-1", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/toml.html"}1043{"id": "3bb3eb6dfd2c-0", "text": "Source code for langchain.document_loaders.blackboard\n\"\"\"Loader that loads all documents from a blackboard course.\"\"\"\nimport contextlib\nimport re\nfrom pathlib import Path\nfrom typing import Any, List, Optional, Tuple\nfrom urllib.parse import unquote\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.directory import DirectoryLoader\nfrom langchain.document_loaders.pdf import PyPDFLoader\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class BlackboardLoader(WebBaseLoader):\n    \"\"\"Loader that loads all documents from a Blackboard course.\n    This loader is not compatible with all Blackboard courses. It is only\n    compatible with courses that use the new Blackboard interface.\n    To use this loader, you must have the BbRouter cookie. You can get this\n    cookie by logging into the course and then copying the value of the\n    BbRouter cookie from the browser's developer tools.\n    Example:\n        .. code-block:: python\n            from langchain.document_loaders import BlackboardLoader\n            loader = BlackboardLoader(\n                blackboard_course_url=\"https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1\",\n                bbrouter=\"expires:12345...\",\n            )\n            documents = loader.load()\n    \"\"\"\n    base_url: str\n    folder_path: str\n    load_all_recursively: bool\n    def __init__(\n        self,\n        blackboard_course_url: str,\n        bbrouter: str,\n        load_all_recursively: bool = True,\n        basic_auth: Optional[Tuple[str, str]] = None,\n        cookies: Optional[dict] = None,\n    ):\n        \"\"\"Initialize with blackboard course url.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1044{"id": "3bb3eb6dfd2c-1", "text": "):\n        \"\"\"Initialize with blackboard course url.\n        The BbRouter cookie is required for most blackboard courses.\n        Args:\n            blackboard_course_url: Blackboard course url.\n            bbrouter: BbRouter cookie.\n            load_all_recursively: If True, load all documents recursively.\n            basic_auth: Basic auth credentials.\n            cookies: Cookies.\n        Raises:\n            ValueError: If blackboard course url is invalid.\n        \"\"\"\n        super().__init__(blackboard_course_url)\n        # Get base url\n        try:\n            self.base_url = blackboard_course_url.split(\"/webapps/blackboard\")[0]\n        except IndexError:\n            raise ValueError(\n                \"Invalid blackboard course url. \"\n                \"Please provide a url that starts with \"\n                \"https://<blackboard_url>/webapps/blackboard\"\n            )\n        if basic_auth is not None:\n            self.session.auth = basic_auth\n        # Combine cookies\n        if cookies is None:\n            cookies = {}\n        cookies.update({\"BbRouter\": bbrouter})\n        self.session.cookies.update(cookies)\n        self.load_all_recursively = load_all_recursively\n        self.check_bs4()\n[docs]    def check_bs4(self) -> None:\n        \"\"\"Check if BeautifulSoup4 is installed.\n        Raises:\n            ImportError: If BeautifulSoup4 is not installed.\n        \"\"\"\n        try:\n            import bs4  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"BeautifulSoup4 is required for BlackboardLoader. \"\n                \"Please install it with `pip install beautifulsoup4`.\"\n            )\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load data into document objects.\n        Returns:\n            List of documents.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1045{"id": "3bb3eb6dfd2c-2", "text": "\"\"\"Load data into document objects.\n        Returns:\n            List of documents.\n        \"\"\"\n        if self.load_all_recursively:\n            soup_info = self.scrape()\n            self.folder_path = self._get_folder_path(soup_info)\n            relative_paths = self._get_paths(soup_info)\n            documents = []\n            for path in relative_paths:\n                url = self.base_url + path\n                print(f\"Fetching documents from {url}\")\n                soup_info = self._scrape(url)\n                with contextlib.suppress(ValueError):\n                    documents.extend(self._get_documents(soup_info))\n            return documents\n        else:\n            print(f\"Fetching documents from {self.web_path}\")\n            soup_info = self.scrape()\n            self.folder_path = self._get_folder_path(soup_info)\n            return self._get_documents(soup_info)\n    def _get_folder_path(self, soup: Any) -> str:\n        \"\"\"Get the folder path to save the documents in.\n        Args:\n            soup: BeautifulSoup4 soup object.\n        Returns:\n            Folder path.\n        \"\"\"\n        # Get the course name\n        course_name = soup.find(\"span\", {\"id\": \"crumb_1\"})\n        if course_name is None:\n            raise ValueError(\"No course name found.\")\n        course_name = course_name.text.strip()\n        # Prepare the folder path\n        course_name_clean = (\n            unquote(course_name)\n            .replace(\" \", \"_\")\n            .replace(\"/\", \"_\")\n            .replace(\":\", \"_\")\n            .replace(\",\", \"_\")\n            .replace(\"?\", \"_\")\n            .replace(\"'\", \"_\")\n            .replace(\"!\", \"_\")\n            .replace('\"', \"_\")\n        )\n        # Get the folder path\n        folder_path = Path(\".\") / course_name_clean", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1046{"id": "3bb3eb6dfd2c-3", "text": ")\n        # Get the folder path\n        folder_path = Path(\".\") / course_name_clean\n        return str(folder_path)\n    def _get_documents(self, soup: Any) -> List[Document]:\n        \"\"\"Fetch content from page and return Documents.\n        Args:\n            soup: BeautifulSoup4 soup object.\n        Returns:\n            List of documents.\n        \"\"\"\n        attachments = self._get_attachments(soup)\n        self._download_attachments(attachments)\n        documents = self._load_documents()\n        return documents\n    def _get_attachments(self, soup: Any) -> List[str]:\n        \"\"\"Get all attachments from a page.\n        Args:\n            soup: BeautifulSoup4 soup object.\n        Returns:\n            List of attachments.\n        \"\"\"\n        from bs4 import BeautifulSoup, Tag\n        # Get content list\n        content_list = soup.find(\"ul\", {\"class\": \"contentList\"})\n        if content_list is None:\n            raise ValueError(\"No content list found.\")\n        content_list: BeautifulSoup  # type: ignore\n        # Get all attachments\n        attachments = []\n        for attachment in content_list.find_all(\"ul\", {\"class\": \"attachments\"}):\n            attachment: Tag  # type: ignore\n            for link in attachment.find_all(\"a\"):\n                link: Tag  # type: ignore\n                href = link.get(\"href\")\n                # Only add if href is not None and does not start with #\n                if href is not None and not href.startswith(\"#\"):\n                    attachments.append(href)\n        return attachments\n    def _download_attachments(self, attachments: List[str]) -> None:\n        \"\"\"Download all attachments.\n        Args:\n            attachments: List of attachments.\n        \"\"\"\n        # Make sure the folder exists\n        Path(self.folder_path).mkdir(parents=True, exist_ok=True)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1047{"id": "3bb3eb6dfd2c-4", "text": "Path(self.folder_path).mkdir(parents=True, exist_ok=True)\n        # Download all attachments\n        for attachment in attachments:\n            self.download(attachment)\n    def _load_documents(self) -> List[Document]:\n        \"\"\"Load all documents in the folder.\n        Returns:\n            List of documents.\n        \"\"\"\n        # Create the document loader\n        loader = DirectoryLoader(\n            path=self.folder_path, glob=\"*.pdf\", loader_cls=PyPDFLoader  # type: ignore\n        )\n        # Load the documents\n        documents = loader.load()\n        # Return all documents\n        return documents\n    def _get_paths(self, soup: Any) -> List[str]:\n        \"\"\"Get all relative paths in the navbar.\"\"\"\n        relative_paths = []\n        course_menu = soup.find(\"ul\", {\"class\": \"courseMenu\"})\n        if course_menu is None:\n            raise ValueError(\"No course menu found.\")\n        for link in course_menu.find_all(\"a\"):\n            href = link.get(\"href\")\n            if href is not None and href.startswith(\"/\"):\n                relative_paths.append(href)\n        return relative_paths\n[docs]    def download(self, path: str) -> None:\n        \"\"\"Download a file from a url.\n        Args:\n            path: Path to the file.\n        \"\"\"\n        # Get the file content\n        response = self.session.get(self.base_url + path, allow_redirects=True)\n        # Get the filename\n        filename = self.parse_filename(response.url)\n        # Write the file to disk\n        with open(Path(self.folder_path) / filename, \"wb\") as f:\n            f.write(response.content)\n[docs]    def parse_filename(self, url: str) -> str:\n        \"\"\"Parse the filename from a url.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1048{"id": "3bb3eb6dfd2c-5", "text": "\"\"\"Parse the filename from a url.\n        Args:\n            url: Url to parse the filename from.\n        Returns:\n            The filename.\n        \"\"\"\n        if (url_path := Path(url)) and url_path.suffix == \".pdf\":\n            return url_path.name\n        else:\n            return self._parse_filename_from_url(url)\n    def _parse_filename_from_url(self, url: str) -> str:\n        \"\"\"Parse the filename from a url.\n        Args:\n            url: Url to parse the filename from.\n        Returns:\n            The filename.\n        Raises:\n            ValueError: If the filename could not be parsed.\n        \"\"\"\n        filename_matches = re.search(r\"filename%2A%3DUTF-8%27%27(.+)\", url)\n        if filename_matches:\n            filename = filename_matches.group(1)\n        else:\n            raise ValueError(f\"Could not parse filename from {url}\")\n        if \".pdf\" not in filename:\n            raise ValueError(f\"Incorrect file type: {filename}\")\n        filename = filename.split(\".pdf\")[0] + \".pdf\"\n        filename = unquote(filename)\n        filename = filename.replace(\"%20\", \" \")\n        return filename\nif __name__ == \"__main__\":\n    loader = BlackboardLoader(\n        \"https://<YOUR BLACKBOARD URL\"\n        \" HERE>/webapps/blackboard/content/listContent.jsp?course_id=_<YOUR COURSE ID\"\n        \" HERE>_1&content_id=_<YOUR CONTENT ID HERE>_1&mode=reset\",\n        \"<YOUR BBROUTER COOKIE HERE>\",\n        load_all_recursively=True,\n    )\n    documents = loader.load()\n    print(f\"Loaded {len(documents)} pages of PDFs from {loader.web_path}\")\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1049{"id": "3bb3eb6dfd2c-6", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blackboard.html"}1050{"id": "9634634a0b5c-0", "text": "Source code for langchain.document_loaders.docugami\n\"\"\"Loader that loads processed documents from Docugami.\"\"\"\nimport io\nimport logging\nimport os\nimport re\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Mapping, Optional, Sequence, Union\nimport requests\nfrom pydantic import BaseModel, root_validator\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nTD_NAME = \"{http://www.w3.org/1999/xhtml}td\"\nTABLE_NAME = \"{http://www.w3.org/1999/xhtml}table\"\nXPATH_KEY = \"xpath\"\nDOCUMENT_ID_KEY = \"id\"\nDOCUMENT_NAME_KEY = \"name\"\nSTRUCTURE_KEY = \"structure\"\nTAG_KEY = \"tag\"\nPROJECTS_KEY = \"projects\"\nDEFAULT_API_ENDPOINT = \"https://api.docugami.com/v1preview1\"\nlogger = logging.getLogger(__name__)\n[docs]class DocugamiLoader(BaseLoader, BaseModel):\n    \"\"\"Loader that loads processed docs from Docugami.\n    To use, you should have the ``lxml`` python package installed.\n    \"\"\"\n    api: str = DEFAULT_API_ENDPOINT\n    access_token: Optional[str] = os.environ.get(\"DOCUGAMI_API_KEY\")\n    docset_id: Optional[str]\n    document_ids: Optional[Sequence[str]]\n    file_paths: Optional[Sequence[Union[Path, str]]]\n    min_chunk_size: int = 32  # appended to the next chunk to avoid over-chunking\n    @root_validator\n    def validate_local_or_remote(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Validate that either local file paths are given, or remote API docset ID.\"\"\"\n        if values.get(\"file_paths\") and values.get(\"docset_id\"):", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1051{"id": "9634634a0b5c-1", "text": "if values.get(\"file_paths\") and values.get(\"docset_id\"):\n            raise ValueError(\"Cannot specify both file_paths and remote API docset_id\")\n        if not values.get(\"file_paths\") and not values.get(\"docset_id\"):\n            raise ValueError(\"Must specify either file_paths or remote API docset_id\")\n        if values.get(\"docset_id\") and not values.get(\"access_token\"):\n            raise ValueError(\"Must specify access token if using remote API docset_id\")\n        return values\n    def _parse_dgml(\n        self, document: Mapping, content: bytes, doc_metadata: Optional[Mapping] = None\n    ) -> List[Document]:\n        \"\"\"Parse a single DGML document into a list of Documents.\"\"\"\n        try:\n            from lxml import etree\n        except ImportError:\n            raise ImportError(\n                \"Could not import lxml python package. \"\n                \"Please install it with `pip install lxml`.\"\n            )\n        # helpers\n        def _xpath_qname_for_chunk(chunk: Any) -> str:\n            \"\"\"Get the xpath qname for a chunk.\"\"\"\n            qname = f\"{chunk.prefix}:{chunk.tag.split('}')[-1]}\"\n            parent = chunk.getparent()\n            if parent is not None:\n                doppelgangers = [x for x in parent if x.tag == chunk.tag]\n                if len(doppelgangers) > 1:\n                    idx_of_self = doppelgangers.index(chunk)\n                    qname = f\"{qname}[{idx_of_self + 1}]\"\n            return qname\n        def _xpath_for_chunk(chunk: Any) -> str:\n            \"\"\"Get the xpath for a chunk.\"\"\"\n            ancestor_chain = chunk.xpath(\"ancestor-or-self::*\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1052{"id": "9634634a0b5c-2", "text": "ancestor_chain = chunk.xpath(\"ancestor-or-self::*\")\n            return \"/\" + \"/\".join(_xpath_qname_for_chunk(x) for x in ancestor_chain)\n        def _structure_value(node: Any) -> str:\n            \"\"\"Get the structure value for a node.\"\"\"\n            structure = (\n                \"table\"\n                if node.tag == TABLE_NAME\n                else node.attrib[\"structure\"]\n                if \"structure\" in node.attrib\n                else None\n            )\n            return structure\n        def _is_structural(node: Any) -> bool:\n            \"\"\"Check if a node is structural.\"\"\"\n            return _structure_value(node) is not None\n        def _is_heading(node: Any) -> bool:\n            \"\"\"Check if a node is a heading.\"\"\"\n            structure = _structure_value(node)\n            return structure is not None and structure.lower().startswith(\"h\")\n        def _get_text(node: Any) -> str:\n            \"\"\"Get the text of a node.\"\"\"\n            return \" \".join(node.itertext()).strip()\n        def _has_structural_descendant(node: Any) -> bool:\n            \"\"\"Check if a node has a structural descendant.\"\"\"\n            for child in node:\n                if _is_structural(child) or _has_structural_descendant(child):\n                    return True\n            return False\n        def _leaf_structural_nodes(node: Any) -> List:\n            \"\"\"Get the leaf structural nodes of a node.\"\"\"\n            if _is_structural(node) and not _has_structural_descendant(node):\n                return [node]\n            else:\n                leaf_nodes = []\n                for child in node:\n                    leaf_nodes.extend(_leaf_structural_nodes(child))\n                return leaf_nodes\n        def _create_doc(node: Any, text: str) -> Document:\n            \"\"\"Create a Document from a node and text.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1053{"id": "9634634a0b5c-3", "text": "\"\"\"Create a Document from a node and text.\"\"\"\n            metadata = {\n                XPATH_KEY: _xpath_for_chunk(node),\n                DOCUMENT_ID_KEY: document[\"id\"],\n                DOCUMENT_NAME_KEY: document[\"name\"],\n                STRUCTURE_KEY: node.attrib.get(\"structure\", \"\"),\n                TAG_KEY: re.sub(r\"\\{.*\\}\", \"\", node.tag),\n            }\n            if doc_metadata:\n                metadata.update(doc_metadata)\n            return Document(\n                page_content=text,\n                metadata=metadata,\n            )\n        # parse the tree and return chunks\n        tree = etree.parse(io.BytesIO(content))\n        root = tree.getroot()\n        chunks: List[Document] = []\n        prev_small_chunk_text = None\n        for node in _leaf_structural_nodes(root):\n            text = _get_text(node)\n            if prev_small_chunk_text:\n                text = prev_small_chunk_text + \" \" + text\n                prev_small_chunk_text = None\n            if _is_heading(node) or len(text) < self.min_chunk_size:\n                # Save headings or other small chunks to be appended to the next chunk\n                prev_small_chunk_text = text\n            else:\n                chunks.append(_create_doc(node, text))\n        if prev_small_chunk_text and len(chunks) > 0:\n            # small chunk at the end left over, just append to last chunk\n            chunks[-1].page_content += \" \" + prev_small_chunk_text\n        return chunks\n    def _document_details_for_docset_id(self, docset_id: str) -> List[Dict]:\n        \"\"\"Gets all document details for the given docset ID\"\"\"\n        url = f\"{self.api}/docsets/{docset_id}/documents\"\n        all_documents = []\n        while url:\n            response = requests.get(\n                url,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1054{"id": "9634634a0b5c-4", "text": "while url:\n            response = requests.get(\n                url,\n                headers={\"Authorization\": f\"Bearer {self.access_token}\"},\n            )\n            if response.ok:\n                data = response.json()\n                all_documents.extend(data[\"documents\"])\n                url = data.get(\"next\", None)\n            else:\n                raise Exception(\n                    f\"Failed to download {url} (status: {response.status_code})\"\n                )\n        return all_documents\n    def _project_details_for_docset_id(self, docset_id: str) -> List[Dict]:\n        \"\"\"Gets all project details for the given docset ID\"\"\"\n        url = f\"{self.api}/projects?docset.id={docset_id}\"\n        all_projects = []\n        while url:\n            response = requests.request(\n                \"GET\",\n                url,\n                headers={\"Authorization\": f\"Bearer {self.access_token}\"},\n                data={},\n            )\n            if response.ok:\n                data = response.json()\n                all_projects.extend(data[\"projects\"])\n                url = data.get(\"next\", None)\n            else:\n                raise Exception(\n                    f\"Failed to download {url} (status: {response.status_code})\"\n                )\n        return all_projects\n    def _metadata_for_project(self, project: Dict) -> Dict:\n        \"\"\"Gets project metadata for all files\"\"\"\n        project_id = project.get(\"id\")\n        url = f\"{self.api}/projects/{project_id}/artifacts/latest\"\n        all_artifacts = []\n        while url:\n            response = requests.request(\n                \"GET\",\n                url,\n                headers={\"Authorization\": f\"Bearer {self.access_token}\"},\n                data={},\n            )\n            if response.ok:\n                data = response.json()", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1055{"id": "9634634a0b5c-5", "text": "data={},\n            )\n            if response.ok:\n                data = response.json()\n                all_artifacts.extend(data[\"artifacts\"])\n                url = data.get(\"next\", None)\n            else:\n                raise Exception(\n                    f\"Failed to download {url} (status: {response.status_code})\"\n                )\n        per_file_metadata = {}\n        for artifact in all_artifacts:\n            artifact_name = artifact.get(\"name\")\n            artifact_url = artifact.get(\"url\")\n            artifact_doc = artifact.get(\"document\")\n            if artifact_name == f\"{project_id}.xml\" and artifact_url and artifact_doc:\n                doc_id = artifact_doc[\"id\"]\n                metadata: Dict = {}\n                # the evaluated XML for each document is named after the project\n                response = requests.request(\n                    \"GET\",\n                    f\"{artifact_url}/content\",\n                    headers={\"Authorization\": f\"Bearer {self.access_token}\"},\n                    data={},\n                )\n                if response.ok:\n                    try:\n                        from lxml import etree\n                    except ImportError:\n                        raise ImportError(\n                            \"Could not import lxml python package. \"\n                            \"Please install it with `pip install lxml`.\"\n                        )\n                    artifact_tree = etree.parse(io.BytesIO(response.content))\n                    artifact_root = artifact_tree.getroot()\n                    ns = artifact_root.nsmap\n                    entries = artifact_root.xpath(\"//wp:Entry\", namespaces=ns)\n                    for entry in entries:\n                        heading = entry.xpath(\"./wp:Heading\", namespaces=ns)[0].text\n                        value = \" \".join(\n                            entry.xpath(\"./wp:Value\", namespaces=ns)[0].itertext()\n                        ).strip()\n                        metadata[heading] = value\n                    per_file_metadata[doc_id] = metadata\n                else:\n                    raise Exception(", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1056{"id": "9634634a0b5c-6", "text": "per_file_metadata[doc_id] = metadata\n                else:\n                    raise Exception(\n                        f\"Failed to download {artifact_url}/content \"\n                        + \"(status: {response.status_code})\"\n                    )\n        return per_file_metadata\n    def _load_chunks_for_document(\n        self, docset_id: str, document: Dict, doc_metadata: Optional[Dict] = None\n    ) -> List[Document]:\n        \"\"\"Load chunks for a document.\"\"\"\n        document_id = document[\"id\"]\n        url = f\"{self.api}/docsets/{docset_id}/documents/{document_id}/dgml\"\n        response = requests.request(\n            \"GET\",\n            url,\n            headers={\"Authorization\": f\"Bearer {self.access_token}\"},\n            data={},\n        )\n        if response.ok:\n            return self._parse_dgml(document, response.content, doc_metadata)\n        else:\n            raise Exception(\n                f\"Failed to download {url} (status: {response.status_code})\"\n            )\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        chunks: List[Document] = []\n        if self.access_token and self.docset_id:\n            # remote mode\n            _document_details = self._document_details_for_docset_id(self.docset_id)\n            if self.document_ids:\n                _document_details = [\n                    d for d in _document_details if d[\"id\"] in self.document_ids\n                ]\n            _project_details = self._project_details_for_docset_id(self.docset_id)\n            combined_project_metadata = {}\n            if _project_details:\n                # if there are any projects for this docset, load project metadata\n                for project in _project_details:\n                    metadata = self._metadata_for_project(project)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1057{"id": "9634634a0b5c-7", "text": "for project in _project_details:\n                    metadata = self._metadata_for_project(project)\n                    combined_project_metadata.update(metadata)\n            for doc in _document_details:\n                doc_metadata = combined_project_metadata.get(doc[\"id\"])\n                chunks += self._load_chunks_for_document(\n                    self.docset_id, doc, doc_metadata\n                )\n        elif self.file_paths:\n            # local mode (for integration testing, or pre-downloaded XML)\n            for path in self.file_paths:\n                path = Path(path)\n                with open(path, \"rb\") as file:\n                    chunks += self._parse_dgml(\n                        {\n                            DOCUMENT_ID_KEY: path.name,\n                            DOCUMENT_NAME_KEY: path.name,\n                        },\n                        file.read(),\n                    )\n        return chunks\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/docugami.html"}1058{"id": "aef879bf0a65-0", "text": "Source code for langchain.document_loaders.notiondb\n\"\"\"Notion DB loader for langchain\"\"\"\nfrom typing import Any, Dict, List, Optional\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nNOTION_BASE_URL = \"https://api.notion.com/v1\"\nDATABASE_URL = NOTION_BASE_URL + \"/databases/{database_id}/query\"\nPAGE_URL = NOTION_BASE_URL + \"/pages/{page_id}\"\nBLOCK_URL = NOTION_BASE_URL + \"/blocks/{block_id}/children\"\n[docs]class NotionDBLoader(BaseLoader):\n    \"\"\"Notion DB Loader.\n    Reads content from pages within a Noton Database.\n    Args:\n        integration_token (str): Notion integration token.\n        database_id (str): Notion database id.\n        request_timeout_sec (int): Timeout for Notion requests in seconds.\n    \"\"\"\n    def __init__(\n        self,\n        integration_token: str,\n        database_id: str,\n        request_timeout_sec: Optional[int] = 10,\n    ) -> None:\n        \"\"\"Initialize with parameters.\"\"\"\n        if not integration_token:\n            raise ValueError(\"integration_token must be provided\")\n        if not database_id:\n            raise ValueError(\"database_id must be provided\")\n        self.token = integration_token\n        self.database_id = database_id\n        self.headers = {\n            \"Authorization\": \"Bearer \" + self.token,\n            \"Content-Type\": \"application/json\",\n            \"Notion-Version\": \"2022-06-28\",\n        }\n        self.request_timeout_sec = request_timeout_sec\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents from the Notion database.\n        Returns:\n            List[Document]: List of documents.\n        \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notiondb.html"}1059{"id": "aef879bf0a65-1", "text": "Returns:\n            List[Document]: List of documents.\n        \"\"\"\n        page_ids = self._retrieve_page_ids()\n        return list(self.load_page(page_id) for page_id in page_ids)\n    def _retrieve_page_ids(\n        self, query_dict: Dict[str, Any] = {\"page_size\": 100}\n    ) -> List[str]:\n        \"\"\"Get all the pages from a Notion database.\"\"\"\n        pages: List[Dict[str, Any]] = []\n        while True:\n            data = self._request(\n                DATABASE_URL.format(database_id=self.database_id),\n                method=\"POST\",\n                query_dict=query_dict,\n            )\n            pages.extend(data.get(\"results\"))\n            if not data.get(\"has_more\"):\n                break\n            query_dict[\"start_cursor\"] = data.get(\"next_cursor\")\n        page_ids = [page[\"id\"] for page in pages]\n        return page_ids\n[docs]    def load_page(self, page_id: str) -> Document:\n        \"\"\"Read a page.\"\"\"\n        data = self._request(PAGE_URL.format(page_id=page_id))\n        # load properties as metadata\n        metadata: Dict[str, Any] = {}\n        for prop_name, prop_data in data[\"properties\"].items():\n            prop_type = prop_data[\"type\"]\n            if prop_type == \"rich_text\":\n                value = (\n                    prop_data[\"rich_text\"][0][\"plain_text\"]\n                    if prop_data[\"rich_text\"]\n                    else None\n                )\n            elif prop_type == \"title\":\n                value = (\n                    prop_data[\"title\"][0][\"plain_text\"] if prop_data[\"title\"] else None\n                )\n            elif prop_type == \"multi_select\":\n                value = (\n                    [item[\"name\"] for item in prop_data[\"multi_select\"]]", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notiondb.html"}1060{"id": "aef879bf0a65-2", "text": "value = (\n                    [item[\"name\"] for item in prop_data[\"multi_select\"]]\n                    if prop_data[\"multi_select\"]\n                    else []\n                )\n            elif prop_type == \"url\":\n                value = prop_data[\"url\"]\n            else:\n                value = None\n            metadata[prop_name.lower()] = value\n        metadata[\"id\"] = page_id\n        return Document(page_content=self._load_blocks(page_id), metadata=metadata)\n    def _load_blocks(self, block_id: str, num_tabs: int = 0) -> str:\n        \"\"\"Read a block and its children.\"\"\"\n        result_lines_arr: List[str] = []\n        cur_block_id: str = block_id\n        while cur_block_id:\n            data = self._request(BLOCK_URL.format(block_id=cur_block_id))\n            for result in data[\"results\"]:\n                result_obj = result[result[\"type\"]]\n                if \"rich_text\" not in result_obj:\n                    continue\n                cur_result_text_arr: List[str] = []\n                for rich_text in result_obj[\"rich_text\"]:\n                    if \"text\" in rich_text:\n                        cur_result_text_arr.append(\n                            \"\\t\" * num_tabs + rich_text[\"text\"][\"content\"]\n                        )\n                if result[\"has_children\"]:\n                    children_text = self._load_blocks(\n                        result[\"id\"], num_tabs=num_tabs + 1\n                    )\n                    cur_result_text_arr.append(children_text)\n                result_lines_arr.append(\"\\n\".join(cur_result_text_arr))\n            cur_block_id = data.get(\"next_cursor\")\n        return \"\\n\".join(result_lines_arr)\n    def _request(\n        self, url: str, method: str = \"GET\", query_dict: Dict[str, Any] = {}\n    ) -> Any:\n        res = requests.request(", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notiondb.html"}1061{"id": "aef879bf0a65-3", "text": ") -> Any:\n        res = requests.request(\n            method,\n            url,\n            headers=self.headers,\n            json=query_dict,\n            timeout=self.request_timeout_sec,\n        )\n        res.raise_for_status()\n        return res.json()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/notiondb.html"}1062{"id": "78e5c74bc618-0", "text": "Source code for langchain.document_loaders.psychic\n\"\"\"Loader that loads documents from Psychic.dev.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class PsychicLoader(BaseLoader):\n    \"\"\"Loader that loads documents from Psychic.dev.\"\"\"\n    def __init__(self, api_key: str, connector_id: str, connection_id: str):\n        \"\"\"Initialize with API key, connector id, and connection id.\"\"\"\n        try:\n            from psychicapi import ConnectorId, Psychic  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"`psychicapi` package not found, please run `pip install psychicapi`\"\n            )\n        self.psychic = Psychic(secret_key=api_key)\n        self.connector_id = ConnectorId(connector_id)\n        self.connection_id = connection_id\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        psychic_docs = self.psychic.get_documents(self.connector_id, self.connection_id)\n        return [\n            Document(\n                page_content=doc[\"content\"],\n                metadata={\"title\": doc[\"title\"], \"source\": doc[\"uri\"]},\n            )\n            for doc in psychic_docs\n        ]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/psychic.html"}1063{"id": "64c0a794c371-0", "text": "Source code for langchain.document_loaders.hugging_face_dataset\n\"\"\"Loader that loads HuggingFace datasets.\"\"\"\nfrom typing import Iterator, List, Mapping, Optional, Sequence, Union\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class HuggingFaceDatasetLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from the Hugging Face Hub.\"\"\"\n    def __init__(\n        self,\n        path: str,\n        page_content_column: str = \"text\",\n        name: Optional[str] = None,\n        data_dir: Optional[str] = None,\n        data_files: Optional[\n            Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]\n        ] = None,\n        cache_dir: Optional[str] = None,\n        keep_in_memory: Optional[bool] = None,\n        save_infos: bool = False,\n        use_auth_token: Optional[Union[bool, str]] = None,\n        num_proc: Optional[int] = None,\n    ):\n        \"\"\"Initialize the HuggingFaceDatasetLoader.\n        Args:\n            path: Path or name of the dataset.\n            page_content_column: Page content column name.\n            name: Name of the dataset configuration.\n            data_dir: Data directory of the dataset configuration.\n            data_files: Path(s) to source data file(s).\n            cache_dir: Directory to read/write data.\n            keep_in_memory: Whether to copy the dataset in-memory.\n            save_infos: Save the dataset information (checksums/size/splits/...).\n            use_auth_token: Bearer token for remote files on the Datasets Hub.\n            num_proc: Number of processes.\n        \"\"\"\n        self.path = path\n        self.page_content_column = page_content_column\n        self.name = name", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/hugging_face_dataset.html"}1064{"id": "64c0a794c371-1", "text": "self.page_content_column = page_content_column\n        self.name = name\n        self.data_dir = data_dir\n        self.data_files = data_files\n        self.cache_dir = cache_dir\n        self.keep_in_memory = keep_in_memory\n        self.save_infos = save_infos\n        self.use_auth_token = use_auth_token\n        self.num_proc = num_proc\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Load documents lazily.\"\"\"\n        try:\n            from datasets import load_dataset\n        except ImportError:\n            raise ImportError(\n                \"Could not import datasets python package. \"\n                \"Please install it with `pip install datasets`.\"\n            )\n        dataset = load_dataset(\n            path=self.path,\n            name=self.name,\n            data_dir=self.data_dir,\n            data_files=self.data_files,\n            cache_dir=self.cache_dir,\n            keep_in_memory=self.keep_in_memory,\n            save_infos=self.save_infos,\n            use_auth_token=self.use_auth_token,\n            num_proc=self.num_proc,\n        )\n        yield from (\n            Document(\n                page_content=row.pop(self.page_content_column),\n                metadata=row,\n            )\n            for key in dataset.keys()\n            for row in dataset[key]\n        )\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        return list(self.lazy_load())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/hugging_face_dataset.html"}1065{"id": "7dd689f6b514-0", "text": "Source code for langchain.document_loaders.tomarkdown\n\"\"\"Loader that loads HTML to markdown using 2markdown.\"\"\"\nfrom __future__ import annotations\nfrom typing import Iterator, List\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class ToMarkdownLoader(BaseLoader):\n    \"\"\"Loader that loads HTML to markdown using 2markdown.\"\"\"\n    def __init__(self, url: str, api_key: str):\n        \"\"\"Initialize with url and api key.\"\"\"\n        self.url = url\n        self.api_key = api_key\n[docs]    def lazy_load(\n        self,\n    ) -> Iterator[Document]:\n        \"\"\"Lazily load the file.\"\"\"\n        response = requests.post(\n            \"https://2markdown.com/api/2md\",\n            headers={\"X-Api-Key\": self.api_key},\n            json={\"url\": self.url},\n        )\n        text = response.json()[\"article\"]\n        metadata = {\"source\": self.url}\n        yield Document(page_content=text, metadata=metadata)\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load file.\"\"\"\n        return list(self.lazy_load())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/tomarkdown.html"}1066{"id": "406cee6f29ab-0", "text": "Source code for langchain.document_loaders.joplin\nimport json\nimport urllib\nfrom datetime import datetime\nfrom typing import Iterator, List, Optional\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.schema import Document\nfrom langchain.utils import get_from_env\nLINK_NOTE_TEMPLATE = \"joplin://x-callback-url/openNote?id={id}\"\n[docs]class JoplinLoader(BaseLoader):\n    \"\"\"\n    Loader that fetches notes from Joplin.\n    In order to use this loader, you need to have Joplin running with the\n    Web Clipper enabled (look for \"Web Clipper\" in the app settings).\n    To get the access token, you need to go to the Web Clipper options and\n    under \"Advanced Options\" you will find the access token.\n    You can find more information about the Web Clipper service here:\n    https://joplinapp.org/clipper/\n    \"\"\"\n    def __init__(\n        self,\n        access_token: Optional[str] = None,\n        port: int = 41184,\n        host: str = \"localhost\",\n    ) -> None:\n        access_token = access_token or get_from_env(\n            \"access_token\", \"JOPLIN_ACCESS_TOKEN\"\n        )\n        base_url = f\"http://{host}:{port}\"\n        self._get_note_url = (\n            f\"{base_url}/notes?token={access_token}\"\n            f\"&fields=id,parent_id,title,body,created_time,updated_time&page={{page}}\"\n        )\n        self._get_folder_url = (\n            f\"{base_url}/folders/{{id}}?token={access_token}&fields=title\"\n        )\n        self._get_tag_url = (", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/joplin.html"}1067{"id": "406cee6f29ab-1", "text": ")\n        self._get_tag_url = (\n            f\"{base_url}/notes/{{id}}/tags?token={access_token}&fields=title\"\n        )\n    def _get_notes(self) -> Iterator[Document]:\n        has_more = True\n        page = 1\n        while has_more:\n            req_note = urllib.request.Request(self._get_note_url.format(page=page))\n            with urllib.request.urlopen(req_note) as response:\n                json_data = json.loads(response.read().decode())\n                for note in json_data[\"items\"]:\n                    metadata = {\n                        \"source\": LINK_NOTE_TEMPLATE.format(id=note[\"id\"]),\n                        \"folder\": self._get_folder(note[\"parent_id\"]),\n                        \"tags\": self._get_tags(note[\"id\"]),\n                        \"title\": note[\"title\"],\n                        \"created_time\": self._convert_date(note[\"created_time\"]),\n                        \"updated_time\": self._convert_date(note[\"updated_time\"]),\n                    }\n                    yield Document(page_content=note[\"body\"], metadata=metadata)\n                has_more = json_data[\"has_more\"]\n                page += 1\n    def _get_folder(self, folder_id: str) -> str:\n        req_folder = urllib.request.Request(self._get_folder_url.format(id=folder_id))\n        with urllib.request.urlopen(req_folder) as response:\n            json_data = json.loads(response.read().decode())\n            return json_data[\"title\"]\n    def _get_tags(self, note_id: str) -> List[str]:\n        req_tag = urllib.request.Request(self._get_tag_url.format(id=note_id))\n        with urllib.request.urlopen(req_tag) as response:\n            json_data = json.loads(response.read().decode())\n            return [tag[\"title\"] for tag in json_data[\"items\"]]\n    def _convert_date(self, date: int) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/joplin.html"}1068{"id": "406cee6f29ab-2", "text": "def _convert_date(self, date: int) -> str:\n        return datetime.fromtimestamp(date / 1000).strftime(\"%Y-%m-%d %H:%M:%S\")\n[docs]    def lazy_load(self) -> Iterator[Document]:\n        yield from self._get_notes()\n[docs]    def load(self) -> List[Document]:\n        return list(self.lazy_load())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/joplin.html"}1069{"id": "640e114303c2-0", "text": "Source code for langchain.document_loaders.bibtex\nimport logging\nimport re\nfrom pathlib import Path\nfrom typing import Any, Iterator, List, Mapping, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utilities.bibtex import BibtexparserWrapper\nlogger = logging.getLogger(__name__)\n[docs]class BibtexLoader(BaseLoader):\n    \"\"\"Loads a bibtex file into a list of Documents.\n    Each document represents one entry from the bibtex file.\n    If a PDF file is present in the `file` bibtex field, the original PDF\n    is loaded into the document text. If no such file entry is present,\n    the `abstract` field is used instead.\n    \"\"\"\n    def __init__(\n        self,\n        file_path: str,\n        *,\n        parser: Optional[BibtexparserWrapper] = None,\n        max_docs: Optional[int] = None,\n        max_content_chars: Optional[int] = 4_000,\n        load_extra_metadata: bool = False,\n        file_pattern: str = r\"[^:]+\\.pdf\",\n    ):\n        \"\"\"Initialize the BibtexLoader.\n        Args:\n            file_path: Path to the bibtex file.\n            max_docs: Max number of associated documents to load. Use -1 means\n                           no limit.\n        \"\"\"\n        self.file_path = file_path\n        self.parser = parser or BibtexparserWrapper()\n        self.max_docs = max_docs\n        self.max_content_chars = max_content_chars\n        self.load_extra_metadata = load_extra_metadata\n        self.file_regex = re.compile(file_pattern)\n    def _load_entry(self, entry: Mapping[str, Any]) -> Optional[Document]:\n        import fitz", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bibtex.html"}1070{"id": "640e114303c2-1", "text": "import fitz\n        parent_dir = Path(self.file_path).parent\n        # regex is useful for Zotero flavor bibtex files\n        file_names = self.file_regex.findall(entry.get(\"file\", \"\"))\n        if not file_names:\n            return None\n        texts: List[str] = []\n        for file_name in file_names:\n            try:\n                with fitz.open(parent_dir / file_name) as f:\n                    texts.extend(page.get_text() for page in f)\n            except FileNotFoundError as e:\n                logger.debug(e)\n        content = \"\\n\".join(texts) or entry.get(\"abstract\", \"\")\n        if self.max_content_chars:\n            content = content[: self.max_content_chars]\n        metadata = self.parser.get_metadata(entry, load_extra=self.load_extra_metadata)\n        return Document(\n            page_content=content,\n            metadata=metadata,\n        )\n[docs]    def lazy_load(self) -> Iterator[Document]:\n        \"\"\"Load bibtex file using bibtexparser and get the article texts plus the\n        article metadata.\n        See https://bibtexparser.readthedocs.io/en/master/\n        Returns:\n            a list of documents with the document.page_content in text format\n        \"\"\"\n        try:\n            import fitz  # noqa: F401\n        except ImportError:\n            raise ImportError(\n                \"PyMuPDF package not found, please install it with \"\n                \"`pip install pymupdf`\"\n            )\n        entries = self.parser.load_bibtex_entries(self.file_path)\n        if self.max_docs:\n            entries = entries[: self.max_docs]\n        for entry in entries:\n            doc = self._load_entry(entry)\n            if doc:\n                yield doc\n[docs]    def load(self) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bibtex.html"}1071{"id": "640e114303c2-2", "text": "yield doc\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load bibtex file documents from the given bibtex file path.\n        See https://bibtexparser.readthedocs.io/en/master/\n        Args:\n            file_path: the path to the bibtex file\n        Returns:\n            a list of documents with the document.page_content in text format\n        \"\"\"\n        return list(self.lazy_load())\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bibtex.html"}1072{"id": "52045bf1e85a-0", "text": "Source code for langchain.document_loaders.image_captions\n\"\"\"\nLoader that loads image captions\nBy default, the loader utilizes the pre-trained BLIP image captioning model.\nhttps://huggingface.co/Salesforce/blip-image-captioning-base\n\"\"\"\nfrom typing import Any, List, Tuple, Union\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class ImageCaptionLoader(BaseLoader):\n    \"\"\"Loader that loads the captions of an image\"\"\"\n    def __init__(\n        self,\n        path_images: Union[str, List[str]],\n        blip_processor: str = \"Salesforce/blip-image-captioning-base\",\n        blip_model: str = \"Salesforce/blip-image-captioning-base\",\n    ):\n        \"\"\"\n        Initialize with a list of image paths\n        \"\"\"\n        if isinstance(path_images, str):\n            self.image_paths = [path_images]\n        else:\n            self.image_paths = path_images\n        self.blip_processor = blip_processor\n        self.blip_model = blip_model\n[docs]    def load(self) -> List[Document]:\n        \"\"\"\n        Load from a list of image files\n        \"\"\"\n        try:\n            from transformers import BlipForConditionalGeneration, BlipProcessor\n        except ImportError:\n            raise ImportError(\n                \"`transformers` package not found, please install with \"\n                \"`pip install transformers`.\"\n            )\n        processor = BlipProcessor.from_pretrained(self.blip_processor)\n        model = BlipForConditionalGeneration.from_pretrained(self.blip_model)\n        results = []\n        for path_image in self.image_paths:\n            caption, metadata = self._get_captions_and_metadata(\n                model=model, processor=processor, path_image=path_image\n            )", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/image_captions.html"}1073{"id": "52045bf1e85a-1", "text": "model=model, processor=processor, path_image=path_image\n            )\n            doc = Document(page_content=caption, metadata=metadata)\n            results.append(doc)\n        return results\n    def _get_captions_and_metadata(\n        self, model: Any, processor: Any, path_image: str\n    ) -> Tuple[str, dict]:\n        \"\"\"\n        Helper function for getting the captions and metadata of an image\n        \"\"\"\n        try:\n            from PIL import Image\n        except ImportError:\n            raise ImportError(\n                \"`PIL` package not found, please install with `pip install pillow`\"\n            )\n        try:\n            if path_image.startswith(\"http://\") or path_image.startswith(\"https://\"):\n                image = Image.open(requests.get(path_image, stream=True).raw).convert(\n                    \"RGB\"\n                )\n            else:\n                image = Image.open(path_image).convert(\"RGB\")\n        except Exception:\n            raise ValueError(f\"Could not get image data for {path_image}\")\n        inputs = processor(image, \"an image of\", return_tensors=\"pt\")\n        output = model.generate(**inputs)\n        caption: str = processor.decode(output[0])\n        metadata: dict = {\"image_path\": path_image}\n        return caption, metadata\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/image_captions.html"}1074{"id": "d8b568746a06-0", "text": "Source code for langchain.document_loaders.csv_loader\nimport csv\nfrom typing import Dict, List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class CSVLoader(BaseLoader):\n    \"\"\"Loads a CSV file into a list of documents.\n    Each document represents one row of the CSV file. Every row is converted into a\n    key/value pair and outputted to a new line in the document's page_content.\n    The source for each document loaded from csv is set to the value of the\n    `file_path` argument for all doucments by default.\n    You can override this by setting the `source_column` argument to the\n    name of a column in the CSV file.\n    The source of each document will then be set to the value of the column\n    with the name specified in `source_column`.\n    Output Example:\n        .. code-block:: txt\n            column1: value1\n            column2: value2\n            column3: value3\n    \"\"\"\n    def __init__(\n        self,\n        file_path: str,\n        source_column: Optional[str] = None,\n        csv_args: Optional[Dict] = None,\n        encoding: Optional[str] = None,\n    ):\n        self.file_path = file_path\n        self.source_column = source_column\n        self.encoding = encoding\n        self.csv_args = csv_args or {}\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load data into document objects.\"\"\"\n        docs = []\n        with open(self.file_path, newline=\"\", encoding=self.encoding) as csvfile:\n            csv_reader = csv.DictReader(csvfile, **self.csv_args)  # type: ignore\n            for i, row in enumerate(csv_reader):", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/csv_loader.html"}1075{"id": "d8b568746a06-1", "text": "for i, row in enumerate(csv_reader):\n                content = \"\\n\".join(f\"{k.strip()}: {v.strip()}\" for k, v in row.items())\n                try:\n                    source = (\n                        row[self.source_column]\n                        if self.source_column is not None\n                        else self.file_path\n                    )\n                except KeyError:\n                    raise ValueError(\n                        f\"Source column '{self.source_column}' not found in CSV file.\"\n                    )\n                metadata = {\"source\": source, \"row\": i}\n                doc = Document(page_content=content, metadata=metadata)\n                docs.append(doc)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/csv_loader.html"}1076{"id": "575837a06c96-0", "text": "Source code for langchain.document_loaders.s3_file\n\"\"\"Loading logic for loading documents from an s3 file.\"\"\"\nimport os\nimport tempfile\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class S3FileLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from s3.\"\"\"\n    def __init__(self, bucket: str, key: str):\n        \"\"\"Initialize with bucket and key name.\"\"\"\n        self.bucket = bucket\n        self.key = key\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            import boto3\n        except ImportError:\n            raise ImportError(\n                \"Could not import `boto3` python package. \"\n                \"Please install it with `pip install boto3`.\"\n            )\n        s3 = boto3.client(\"s3\")\n        with tempfile.TemporaryDirectory() as temp_dir:\n            file_path = f\"{temp_dir}/{self.key}\"\n            os.makedirs(os.path.dirname(file_path), exist_ok=True)\n            s3.download_file(self.bucket, self.key, file_path)\n            loader = UnstructuredFileLoader(file_path)\n            return loader.load()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/s3_file.html"}1077{"id": "fde2b0a5d1b9-0", "text": "Source code for langchain.document_loaders.googledrive\n\"\"\"Loader that loads data from Google Drive.\"\"\"\n# Prerequisites:\n# 1. Create a Google Cloud project\n# 2. Enable the Google Drive API:\n#   https://console.cloud.google.com/flows/enableapi?apiid=drive.googleapis.com\n# 3. Authorize credentials for desktop app:\n#   https://developers.google.com/drive/api/quickstart/python#authorize_credentials_for_a_desktop_application # noqa: E501\n# 4. For service accounts visit\n#   https://cloud.google.com/iam/docs/service-accounts-create\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Sequence, Union\nfrom pydantic import BaseModel, root_validator, validator\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nSCOPES = [\"https://www.googleapis.com/auth/drive.readonly\"]\n[docs]class GoogleDriveLoader(BaseLoader, BaseModel):\n    \"\"\"Loader that loads Google Docs from Google Drive.\"\"\"\n    service_account_key: Path = Path.home() / \".credentials\" / \"keys.json\"\n    credentials_path: Path = Path.home() / \".credentials\" / \"credentials.json\"\n    token_path: Path = Path.home() / \".credentials\" / \"token.json\"\n    folder_id: Optional[str] = None\n    document_ids: Optional[List[str]] = None\n    file_ids: Optional[List[str]] = None\n    recursive: bool = False\n    file_types: Optional[Sequence[str]] = None\n    load_trashed_files: bool = False\n    @root_validator\n    def validate_inputs(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Validate that either folder_id or document_ids is set, but not both.\"\"\"\n        if values.get(\"folder_id\") and (", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1078{"id": "fde2b0a5d1b9-1", "text": "if values.get(\"folder_id\") and (\n            values.get(\"document_ids\") or values.get(\"file_ids\")\n        ):\n            raise ValueError(\n                \"Cannot specify both folder_id and document_ids nor \"\n                \"folder_id and file_ids\"\n            )\n        if (\n            not values.get(\"folder_id\")\n            and not values.get(\"document_ids\")\n            and not values.get(\"file_ids\")\n        ):\n            raise ValueError(\"Must specify either folder_id, document_ids, or file_ids\")\n        file_types = values.get(\"file_types\")\n        if file_types:\n            if values.get(\"document_ids\") or values.get(\"file_ids\"):\n                raise ValueError(\n                    \"file_types can only be given when folder_id is given,\"\n                    \" (not when document_ids or file_ids are given).\"\n                )\n            type_mapping = {\n                \"document\": \"application/vnd.google-apps.document\",\n                \"sheet\": \"application/vnd.google-apps.spreadsheet\",\n                \"pdf\": \"application/pdf\",\n            }\n            allowed_types = list(type_mapping.keys()) + list(type_mapping.values())\n            short_names = \", \".join([f\"'{x}'\" for x in type_mapping.keys()])\n            full_names = \", \".join([f\"'{x}'\" for x in type_mapping.values()])\n            for file_type in file_types:\n                if file_type not in allowed_types:\n                    raise ValueError(\n                        f\"Given file type {file_type} is not supported. \"\n                        f\"Supported values are: {short_names}; and \"\n                        f\"their full-form names: {full_names}\"\n                    )\n            # replace short-form file types by full-form file types\n            def full_form(x: str) -> str:\n                return type_mapping[x] if x in type_mapping else x", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1079{"id": "fde2b0a5d1b9-2", "text": "return type_mapping[x] if x in type_mapping else x\n            values[\"file_types\"] = [full_form(file_type) for file_type in file_types]\n        return values\n    @validator(\"credentials_path\")\n    def validate_credentials_path(cls, v: Any, **kwargs: Any) -> Any:\n        \"\"\"Validate that credentials_path exists.\"\"\"\n        if not v.exists():\n            raise ValueError(f\"credentials_path {v} does not exist\")\n        return v\n    def _load_credentials(self) -> Any:\n        \"\"\"Load credentials.\"\"\"\n        # Adapted from https://developers.google.com/drive/api/v3/quickstart/python\n        try:\n            from google.auth.transport.requests import Request\n            from google.oauth2 import service_account\n            from google.oauth2.credentials import Credentials\n            from google_auth_oauthlib.flow import InstalledAppFlow\n        except ImportError:\n            raise ImportError(\n                \"You must run \"\n                \"`pip install --upgrade \"\n                \"google-api-python-client google-auth-httplib2 \"\n                \"google-auth-oauthlib` \"\n                \"to use the Google Drive loader.\"\n            )\n        creds = None\n        if self.service_account_key.exists():\n            return service_account.Credentials.from_service_account_file(\n                str(self.service_account_key), scopes=SCOPES\n            )\n        if self.token_path.exists():\n            creds = Credentials.from_authorized_user_file(str(self.token_path), SCOPES)\n        if not creds or not creds.valid:\n            if creds and creds.expired and creds.refresh_token:\n                creds.refresh(Request())\n            else:\n                flow = InstalledAppFlow.from_client_secrets_file(\n                    str(self.credentials_path), SCOPES\n                )\n                creds = flow.run_local_server(port=0)\n            with open(self.token_path, \"w\") as token:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1080{"id": "fde2b0a5d1b9-3", "text": "with open(self.token_path, \"w\") as token:\n                token.write(creds.to_json())\n        return creds\n    def _load_sheet_from_id(self, id: str) -> List[Document]:\n        \"\"\"Load a sheet and all tabs from an ID.\"\"\"\n        from googleapiclient.discovery import build\n        creds = self._load_credentials()\n        sheets_service = build(\"sheets\", \"v4\", credentials=creds)\n        spreadsheet = sheets_service.spreadsheets().get(spreadsheetId=id).execute()\n        sheets = spreadsheet.get(\"sheets\", [])\n        documents = []\n        for sheet in sheets:\n            sheet_name = sheet[\"properties\"][\"title\"]\n            result = (\n                sheets_service.spreadsheets()\n                .values()\n                .get(spreadsheetId=id, range=sheet_name)\n                .execute()\n            )\n            values = result.get(\"values\", [])\n            header = values[0]\n            for i, row in enumerate(values[1:], start=1):\n                metadata = {\n                    \"source\": (\n                        f\"https://docs.google.com/spreadsheets/d/{id}/\"\n                        f\"edit?gid={sheet['properties']['sheetId']}\"\n                    ),\n                    \"title\": f\"{spreadsheet['properties']['title']} - {sheet_name}\",\n                    \"row\": i,\n                }\n                content = []\n                for j, v in enumerate(row):\n                    title = header[j].strip() if len(header) > j else \"\"\n                    content.append(f\"{title}: {v.strip()}\")\n                page_content = \"\\n\".join(content)\n                documents.append(Document(page_content=page_content, metadata=metadata))\n        return documents\n    def _load_document_from_id(self, id: str) -> Document:\n        \"\"\"Load a document from an ID.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1081{"id": "fde2b0a5d1b9-4", "text": "\"\"\"Load a document from an ID.\"\"\"\n        from io import BytesIO\n        from googleapiclient.discovery import build\n        from googleapiclient.errors import HttpError\n        from googleapiclient.http import MediaIoBaseDownload\n        creds = self._load_credentials()\n        service = build(\"drive\", \"v3\", credentials=creds)\n        file = service.files().get(fileId=id, supportsAllDrives=True).execute()\n        request = service.files().export_media(fileId=id, mimeType=\"text/plain\")\n        fh = BytesIO()\n        downloader = MediaIoBaseDownload(fh, request)\n        done = False\n        try:\n            while done is False:\n                status, done = downloader.next_chunk()\n        except HttpError as e:\n            if e.resp.status == 404:\n                print(\"File not found: {}\".format(id))\n            else:\n                print(\"An error occurred: {}\".format(e))\n        text = fh.getvalue().decode(\"utf-8\")\n        metadata = {\n            \"source\": f\"https://docs.google.com/document/d/{id}/edit\",\n            \"title\": f\"{file.get('name')}\",\n        }\n        return Document(page_content=text, metadata=metadata)\n    def _load_documents_from_folder(\n        self, folder_id: str, *, file_types: Optional[Sequence[str]] = None\n    ) -> List[Document]:\n        \"\"\"Load documents from a folder.\"\"\"\n        from googleapiclient.discovery import build\n        creds = self._load_credentials()\n        service = build(\"drive\", \"v3\", credentials=creds)\n        files = self._fetch_files_recursive(service, folder_id)\n        # If file types filter is provided, we'll filter by the file type.\n        if file_types:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1082{"id": "fde2b0a5d1b9-5", "text": "if file_types:\n            _files = [f for f in files if f[\"mimeType\"] in file_types]  # type: ignore\n        else:\n            _files = files\n        returns = []\n        for file in files:\n            if file[\"trashed\"] and not self.load_trashed_files:\n                continue\n            elif file[\"mimeType\"] == \"application/vnd.google-apps.document\":\n                returns.append(self._load_document_from_id(file[\"id\"]))  # type: ignore\n            elif file[\"mimeType\"] == \"application/vnd.google-apps.spreadsheet\":\n                returns.extend(self._load_sheet_from_id(file[\"id\"]))  # type: ignore\n            elif file[\"mimeType\"] == \"application/pdf\":\n                returns.extend(self._load_file_from_id(file[\"id\"]))  # type: ignore\n            else:\n                pass\n        return returns\n    def _fetch_files_recursive(\n        self, service: Any, folder_id: str\n    ) -> List[Dict[str, Union[str, List[str]]]]:\n        \"\"\"Fetch all files and subfolders recursively.\"\"\"\n        results = (\n            service.files()\n            .list(\n                q=f\"'{folder_id}' in parents\",\n                pageSize=1000,\n                includeItemsFromAllDrives=True,\n                supportsAllDrives=True,\n                fields=\"nextPageToken, files(id, name, mimeType, parents, trashed)\",\n            )\n            .execute()\n        )\n        files = results.get(\"files\", [])\n        returns = []\n        for file in files:\n            if file[\"mimeType\"] == \"application/vnd.google-apps.folder\":\n                if self.recursive:\n                    returns.extend(self._fetch_files_recursive(service, file[\"id\"]))\n            else:\n                returns.append(file)\n        return returns", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1083{"id": "fde2b0a5d1b9-6", "text": "else:\n                returns.append(file)\n        return returns\n    def _load_documents_from_ids(self) -> List[Document]:\n        \"\"\"Load documents from a list of IDs.\"\"\"\n        if not self.document_ids:\n            raise ValueError(\"document_ids must be set\")\n        return [self._load_document_from_id(doc_id) for doc_id in self.document_ids]\n    def _load_file_from_id(self, id: str) -> List[Document]:\n        \"\"\"Load a file from an ID.\"\"\"\n        from io import BytesIO\n        from googleapiclient.discovery import build\n        from googleapiclient.http import MediaIoBaseDownload\n        creds = self._load_credentials()\n        service = build(\"drive\", \"v3\", credentials=creds)\n        file = service.files().get(fileId=id, supportsAllDrives=True).execute()\n        request = service.files().get_media(fileId=id)\n        fh = BytesIO()\n        downloader = MediaIoBaseDownload(fh, request)\n        done = False\n        while done is False:\n            status, done = downloader.next_chunk()\n        content = fh.getvalue()\n        from PyPDF2 import PdfReader\n        pdf_reader = PdfReader(BytesIO(content))\n        return [\n            Document(\n                page_content=page.extract_text(),\n                metadata={\n                    \"source\": f\"https://drive.google.com/file/d/{id}/view\",\n                    \"title\": f\"{file.get('name')}\",\n                    \"page\": i,\n                },\n            )\n            for i, page in enumerate(pdf_reader.pages)\n        ]\n    def _load_file_from_ids(self) -> List[Document]:\n        \"\"\"Load files from a list of IDs.\"\"\"\n        if not self.file_ids:\n            raise ValueError(\"file_ids must be set\")\n        docs = []", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1084{"id": "fde2b0a5d1b9-7", "text": "raise ValueError(\"file_ids must be set\")\n        docs = []\n        for file_id in self.file_ids:\n            docs.extend(self._load_file_from_id(file_id))\n        return docs\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        if self.folder_id:\n            return self._load_documents_from_folder(\n                self.folder_id, file_types=self.file_types\n            )\n        elif self.document_ids:\n            return self._load_documents_from_ids()\n        else:\n            return self._load_file_from_ids()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/googledrive.html"}1085{"id": "0ceed7c59ff1-0", "text": "Source code for langchain.document_loaders.markdown\n\"\"\"Loader that loads Markdown files.\"\"\"\nfrom typing import List\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class UnstructuredMarkdownLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load markdown files.\"\"\"\n    def _get_elements(self) -> List:\n        from unstructured.__version__ import __version__ as __unstructured_version__\n        from unstructured.partition.md import partition_md\n        # NOTE(MthwRobinson) - enables the loader to work when you're using pre-release\n        # versions of unstructured like 0.4.17-dev1\n        _unstructured_version = __unstructured_version__.split(\"-\")[0]\n        unstructured_version = tuple([int(x) for x in _unstructured_version.split(\".\")])\n        if unstructured_version < (0, 4, 16):\n            raise ValueError(\n                f\"You are on unstructured version {__unstructured_version__}. \"\n                \"Partitioning markdown files is only supported in unstructured>=0.4.16.\"\n            )\n        return partition_md(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/markdown.html"}1086{"id": "a8ac945a1464-0", "text": "Source code for langchain.document_loaders.apify_dataset\n\"\"\"Logic for loading documents from Apify datasets.\"\"\"\nfrom typing import Any, Callable, Dict, List\nfrom pydantic import BaseModel, root_validator\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class ApifyDatasetLoader(BaseLoader, BaseModel):\n    \"\"\"Logic for loading documents from Apify datasets.\"\"\"\n    apify_client: Any\n    dataset_id: str\n    \"\"\"The ID of the dataset on the Apify platform.\"\"\"\n    dataset_mapping_function: Callable[[Dict], Document]\n    \"\"\"A custom function that takes a single dictionary (an Apify dataset item)\n     and converts it to an instance of the Document class.\"\"\"\n    def __init__(\n        self, dataset_id: str, dataset_mapping_function: Callable[[Dict], Document]\n    ):\n        \"\"\"Initialize the loader with an Apify dataset ID and a mapping function.\n        Args:\n            dataset_id (str): The ID of the dataset on the Apify platform.\n            dataset_mapping_function (Callable): A function that takes a single\n                dictionary (an Apify dataset item) and converts it to an instance\n                of the Document class.\n        \"\"\"\n        super().__init__(\n            dataset_id=dataset_id, dataset_mapping_function=dataset_mapping_function\n        )\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate environment.\"\"\"\n        try:\n            from apify_client import ApifyClient\n            values[\"apify_client\"] = ApifyClient()\n        except ImportError:\n            raise ImportError(\n                \"Could not import apify-client Python package. \"\n                \"Please install it with `pip install apify-client`.\"\n            )\n        return values", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/apify_dataset.html"}1087{"id": "a8ac945a1464-1", "text": ")\n        return values\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        dataset_items = self.apify_client.dataset(self.dataset_id).list_items().items\n        return list(map(self.dataset_mapping_function, dataset_items))\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/apify_dataset.html"}1088{"id": "9e44db4ebcd2-0", "text": "Source code for langchain.document_loaders.mediawikidump\n\"\"\"Load Data from a MediaWiki dump xml.\"\"\"\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class MWDumpLoader(BaseLoader):\n    \"\"\"\n    Load MediaWiki dump from XML file\n    Example:\n        .. code-block:: python\n            from langchain.document_loaders import MWDumpLoader\n            loader = MWDumpLoader(\n                file_path=\"myWiki.xml\",\n                encoding=\"utf8\"\n            )\n            docs = loader.load()\n            from langchain.text_splitter import RecursiveCharacterTextSplitter\n            text_splitter = RecursiveCharacterTextSplitter(\n                chunk_size=1000, chunk_overlap=0\n            )\n            texts = text_splitter.split_documents(docs)\n    :param file_path: XML local file path\n    :type file_path: str\n    :param encoding: Charset encoding, defaults to \"utf8\"\n    :type encoding: str, optional\n    \"\"\"\n    def __init__(self, file_path: str, encoding: Optional[str] = \"utf8\"):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n        self.encoding = encoding\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load from file path.\"\"\"\n        import mwparserfromhell\n        import mwxml\n        dump = mwxml.Dump.from_file(open(self.file_path, encoding=self.encoding))\n        docs = []\n        for page in dump.pages:\n            for revision in page:\n                code = mwparserfromhell.parse(revision.text)\n                text = code.strip_code(\n                    normalize=True, collapse=True, keep_template_params=False\n                )\n                metadata = {\"source\": page.title}", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/mediawikidump.html"}1089{"id": "9e44db4ebcd2-1", "text": ")\n                metadata = {\"source\": page.title}\n                docs.append(Document(page_content=text, metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/mediawikidump.html"}1090{"id": "e84f2abb673a-0", "text": "Source code for langchain.document_loaders.facebook_chat\n\"\"\"Loader that loads Facebook chat json dump.\"\"\"\nimport datetime\nimport json\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\ndef concatenate_rows(row: dict) -> str:\n    \"\"\"Combine message information in a readable format ready to be used.\"\"\"\n    sender = row[\"sender_name\"]\n    text = row[\"content\"]\n    date = datetime.datetime.fromtimestamp(row[\"timestamp_ms\"] / 1000).strftime(\n        \"%Y-%m-%d %H:%M:%S\"\n    )\n    return f\"{sender} on {date}: {text}\\n\\n\"\n[docs]class FacebookChatLoader(BaseLoader):\n    \"\"\"Loader that loads Facebook messages json directory dump.\"\"\"\n    def __init__(self, path: str):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        p = Path(self.file_path)\n        with open(p, encoding=\"utf8\") as f:\n            d = json.load(f)\n        text = \"\".join(\n            concatenate_rows(message)\n            for message in d[\"messages\"]\n            if message.get(\"content\") and isinstance(message[\"content\"], str)\n        )\n        metadata = {\"source\": str(p)}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/facebook_chat.html"}1091{"id": "b4c3204db36a-0", "text": "Source code for langchain.document_loaders.gitbook\n\"\"\"Loader that loads GitBook.\"\"\"\nfrom typing import Any, List, Optional\nfrom urllib.parse import urljoin, urlparse\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class GitbookLoader(WebBaseLoader):\n    \"\"\"Load GitBook data.\n    1. load from either a single page, or\n    2. load all (relative) paths in the navbar.\n    \"\"\"\n    def __init__(\n        self,\n        web_page: str,\n        load_all_paths: bool = False,\n        base_url: Optional[str] = None,\n        content_selector: str = \"main\",\n    ):\n        \"\"\"Initialize with web page and whether to load all paths.\n        Args:\n            web_page: The web page to load or the starting point from where\n                relative paths are discovered.\n            load_all_paths: If set to True, all relative paths in the navbar\n                are loaded instead of only `web_page`.\n            base_url: If `load_all_paths` is True, the relative paths are\n                appended to this base url. Defaults to `web_page` if not set.\n        \"\"\"\n        self.base_url = base_url or web_page\n        if self.base_url.endswith(\"/\"):\n            self.base_url = self.base_url[:-1]\n        if load_all_paths:\n            # set web_path to the sitemap if we want to crawl all paths\n            web_paths = f\"{self.base_url}/sitemap.xml\"\n        else:\n            web_paths = web_page\n        super().__init__(web_paths)\n        self.load_all_paths = load_all_paths\n        self.content_selector = content_selector\n[docs]    def load(self) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/gitbook.html"}1092{"id": "b4c3204db36a-1", "text": "[docs]    def load(self) -> List[Document]:\n        \"\"\"Fetch text from one single GitBook page.\"\"\"\n        if self.load_all_paths:\n            soup_info = self.scrape()\n            relative_paths = self._get_paths(soup_info)\n            documents = []\n            for path in relative_paths:\n                url = urljoin(self.base_url, path)\n                print(f\"Fetching text from {url}\")\n                soup_info = self._scrape(url)\n                documents.append(self._get_document(soup_info, url))\n            return [d for d in documents if d]\n        else:\n            soup_info = self.scrape()\n            documents = [self._get_document(soup_info, self.web_path)]\n            return [d for d in documents if d]\n    def _get_document(\n        self, soup: Any, custom_url: Optional[str] = None\n    ) -> Optional[Document]:\n        \"\"\"Fetch content from page and return Document.\"\"\"\n        page_content_raw = soup.find(self.content_selector)\n        if not page_content_raw:\n            return None\n        content = page_content_raw.get_text(separator=\"\\n\").strip()\n        title_if_exists = page_content_raw.find(\"h1\")\n        title = title_if_exists.text if title_if_exists else \"\"\n        metadata = {\"source\": custom_url or self.web_path, \"title\": title}\n        return Document(page_content=content, metadata=metadata)\n    def _get_paths(self, soup: Any) -> List[str]:\n        \"\"\"Fetch all relative paths in the navbar.\"\"\"\n        return [urlparse(loc.text).path for loc in soup.find_all(\"loc\")]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/gitbook.html"}1093{"id": "4ad9e60241dd-0", "text": "Source code for langchain.document_loaders.blockchain\nimport os\nimport re\nimport time\nfrom enum import Enum\nfrom typing import List, Optional\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nclass BlockchainType(Enum):\n    ETH_MAINNET = \"eth-mainnet\"\n    ETH_GOERLI = \"eth-goerli\"\n    POLYGON_MAINNET = \"polygon-mainnet\"\n    POLYGON_MUMBAI = \"polygon-mumbai\"\n[docs]class BlockchainDocumentLoader(BaseLoader):\n    \"\"\"Loads elements from a blockchain smart contract into Langchain documents.\n    The supported blockchains are: Ethereum mainnet, Ethereum Goerli testnet,\n    Polygon mainnet, and Polygon Mumbai testnet.\n    If no BlockchainType is specified, the default is Ethereum mainnet.\n    The Loader uses the Alchemy API to interact with the blockchain.\n    ALCHEMY_API_KEY environment variable must be set to use this loader.\n    The API returns 100 NFTs per request and can be paginated using the\n    startToken parameter.\n    If get_all_tokens is set to True, the loader will get all tokens\n    on the contract.  Note that for contracts with a large number of tokens,\n    this may take a long time (e.g. 10k tokens is 100 requests).\n    Default value is false for this reason.\n    The max_execution_time (sec) can be set to limit the execution time\n    of the loader.\n    Future versions of this loader can:\n        - Support additional Alchemy APIs (e.g. getTransactions, etc.)\n        - Support additional blockain APIs (e.g. Infura, Opensea, etc.)\n    \"\"\"\n    def __init__(\n        self,\n        contract_address: str,", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blockchain.html"}1094{"id": "4ad9e60241dd-1", "text": "\"\"\"\n    def __init__(\n        self,\n        contract_address: str,\n        blockchainType: BlockchainType = BlockchainType.ETH_MAINNET,\n        api_key: str = \"docs-demo\",\n        startToken: str = \"\",\n        get_all_tokens: bool = False,\n        max_execution_time: Optional[int] = None,\n    ):\n        self.contract_address = contract_address\n        self.blockchainType = blockchainType.value\n        self.api_key = os.environ.get(\"ALCHEMY_API_KEY\") or api_key\n        self.startToken = startToken\n        self.get_all_tokens = get_all_tokens\n        self.max_execution_time = max_execution_time\n        if not self.api_key:\n            raise ValueError(\"Alchemy API key not provided.\")\n        if not re.match(r\"^0x[a-fA-F0-9]{40}$\", self.contract_address):\n            raise ValueError(f\"Invalid contract address {self.contract_address}\")\n[docs]    def load(self) -> List[Document]:\n        result = []\n        current_start_token = self.startToken\n        start_time = time.time()\n        while True:\n            url = (\n                f\"https://{self.blockchainType}.g.alchemy.com/nft/v2/\"\n                f\"{self.api_key}/getNFTsForCollection?withMetadata=\"\n                f\"True&contractAddress={self.contract_address}\"\n                f\"&startToken={current_start_token}\"\n            )\n            response = requests.get(url)\n            if response.status_code != 200:\n                raise ValueError(\n                    f\"Request failed with status code {response.status_code}\"\n                )\n            items = response.json()[\"nfts\"]\n            if not items:\n                break\n            for item in items:\n                content = str(item)\n                tokenId = item[\"id\"][\"tokenId\"]\n                metadata = {", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blockchain.html"}1095{"id": "4ad9e60241dd-2", "text": "tokenId = item[\"id\"][\"tokenId\"]\n                metadata = {\n                    \"source\": self.contract_address,\n                    \"blockchain\": self.blockchainType,\n                    \"tokenId\": tokenId,\n                }\n                result.append(Document(page_content=content, metadata=metadata))\n            # exit after the first API call if get_all_tokens is False\n            if not self.get_all_tokens:\n                break\n            # get the start token for the next API call from the last item in array\n            current_start_token = self._get_next_tokenId(result[-1].metadata[\"tokenId\"])\n            if (\n                self.max_execution_time is not None\n                and (time.time() - start_time) > self.max_execution_time\n            ):\n                raise RuntimeError(\"Execution time exceeded the allowed time limit.\")\n        if not result:\n            raise ValueError(\n                f\"No NFTs found for contract address {self.contract_address}\"\n            )\n        return result\n    # add one to the tokenId, ensuring the correct tokenId format is used\n    def _get_next_tokenId(self, tokenId: str) -> str:\n        value_type = self._detect_value_type(tokenId)\n        if value_type == \"hex_0x\":\n            value_int = int(tokenId, 16)\n        elif value_type == \"hex_0xbf\":\n            value_int = int(tokenId[2:], 16)\n        else:\n            value_int = int(tokenId)\n        result = value_int + 1\n        if value_type == \"hex_0x\":\n            return \"0x\" + format(result, \"0\" + str(len(tokenId) - 2) + \"x\")\n        elif value_type == \"hex_0xbf\":\n            return \"0xbf\" + format(result, \"0\" + str(len(tokenId) - 4) + \"x\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blockchain.html"}1096{"id": "4ad9e60241dd-3", "text": "else:\n            return str(result)\n    # A smart contract can use different formats for the tokenId\n    @staticmethod\n    def _detect_value_type(tokenId: str) -> str:\n        if isinstance(tokenId, int):\n            return \"int\"\n        elif tokenId.startswith(\"0x\"):\n            return \"hex_0x\"\n        elif tokenId.startswith(\"0xbf\"):\n            return \"hex_0xbf\"\n        else:\n            return \"hex_0xbf\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/blockchain.html"}1097{"id": "0386d17bc443-0", "text": "Source code for langchain.document_loaders.spreedly\n\"\"\"Loader that fetches data from Spreedly API.\"\"\"\nimport json\nimport urllib.request\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utils import stringify_dict\nSPREEDLY_ENDPOINTS = {\n    \"gateways_options\": \"https://core.spreedly.com/v1/gateways_options.json\",\n    \"gateways\": \"https://core.spreedly.com/v1/gateways.json\",\n    \"receivers_options\": \"https://core.spreedly.com/v1/receivers_options.json\",\n    \"receivers\": \"https://core.spreedly.com/v1/receivers.json\",\n    \"payment_methods\": \"https://core.spreedly.com/v1/payment_methods.json\",\n    \"certificates\": \"https://core.spreedly.com/v1/certificates.json\",\n    \"transactions\": \"https://core.spreedly.com/v1/transactions.json\",\n    \"environments\": \"https://core.spreedly.com/v1/environments.json\",\n}\n[docs]class SpreedlyLoader(BaseLoader):\n    def __init__(self, access_token: str, resource: str) -> None:\n        self.access_token = access_token\n        self.resource = resource\n        self.headers = {\n            \"Authorization\": f\"Bearer {self.access_token}\",\n            \"Accept\": \"application/json\",\n        }\n    def _make_request(self, url: str) -> List[Document]:\n        request = urllib.request.Request(url, headers=self.headers)\n        with urllib.request.urlopen(request) as response:\n            json_data = json.loads(response.read().decode())\n            text = stringify_dict(json_data)\n            metadata = {\"source\": url}", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/spreedly.html"}1098{"id": "0386d17bc443-1", "text": "text = stringify_dict(json_data)\n            metadata = {\"source\": url}\n            return [Document(page_content=text, metadata=metadata)]\n    def _get_resource(self) -> List[Document]:\n        endpoint = SPREEDLY_ENDPOINTS.get(self.resource)\n        if endpoint is None:\n            return []\n        return self._make_request(endpoint)\n[docs]    def load(self) -> List[Document]:\n        return self._get_resource()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/spreedly.html"}1099{"id": "1328de0c8ba3-0", "text": "Source code for langchain.document_loaders.discord\n\"\"\"Load from Discord chat dump\"\"\"\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING, List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nif TYPE_CHECKING:\n    import pandas as pd\n[docs]class DiscordChatLoader(BaseLoader):\n    \"\"\"Load Discord chat logs.\"\"\"\n    def __init__(self, chat_log: pd.DataFrame, user_id_col: str = \"ID\"):\n        \"\"\"Initialize with a Pandas DataFrame containing chat logs.\"\"\"\n        if not isinstance(chat_log, pd.DataFrame):\n            raise ValueError(\n                f\"Expected chat_log to be a pd.DataFrame, got {type(chat_log)}\"\n            )\n        self.chat_log = chat_log\n        self.user_id_col = user_id_col\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load all chat messages.\"\"\"\n        result = []\n        for _, row in self.chat_log.iterrows():\n            user_id = row[self.user_id_col]\n            metadata = row.to_dict()\n            metadata.pop(self.user_id_col)\n            result.append(Document(page_content=user_id, metadata=metadata))\n        return result\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/discord.html"}1100{"id": "c5280bb4056a-0", "text": "Source code for langchain.document_loaders.rtf\n\"\"\"Loader that loads rich text files.\"\"\"\nfrom typing import Any, List\nfrom langchain.document_loaders.unstructured import (\n    UnstructuredFileLoader,\n    satisfies_min_unstructured_version,\n)\n[docs]class UnstructuredRTFLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load rtf files.\"\"\"\n    def __init__(\n        self, file_path: str, mode: str = \"single\", **unstructured_kwargs: Any\n    ):\n        min_unstructured_version = \"0.5.12\"\n        if not satisfies_min_unstructured_version(min_unstructured_version):\n            raise ValueError(\n                \"Partitioning rtf files is only supported in \"\n                f\"unstructured>={min_unstructured_version}.\"\n            )\n        super().__init__(file_path=file_path, mode=mode, **unstructured_kwargs)\n    def _get_elements(self) -> List:\n        from unstructured.partition.rtf import partition_rtf\n        return partition_rtf(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/rtf.html"}1101{"id": "524183d0cf49-0", "text": "Source code for langchain.document_loaders.s3_directory\n\"\"\"Loading logic for loading documents from an s3 directory.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.s3_file import S3FileLoader\n[docs]class S3DirectoryLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from s3.\"\"\"\n    def __init__(self, bucket: str, prefix: str = \"\"):\n        \"\"\"Initialize with bucket and key name.\"\"\"\n        self.bucket = bucket\n        self.prefix = prefix\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            import boto3\n        except ImportError:\n            raise ImportError(\n                \"Could not import boto3 python package. \"\n                \"Please install it with `pip install boto3`.\"\n            )\n        s3 = boto3.resource(\"s3\")\n        bucket = s3.Bucket(self.bucket)\n        docs = []\n        for obj in bucket.objects.filter(Prefix=self.prefix):\n            loader = S3FileLoader(self.bucket, obj.key)\n            docs.extend(loader.load())\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/s3_directory.html"}1102{"id": "8601568e7855-0", "text": "Source code for langchain.document_loaders.bilibili\nimport json\nimport re\nimport warnings\nfrom typing import List, Tuple\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class BiliBiliLoader(BaseLoader):\n    \"\"\"Loader that loads bilibili transcripts.\"\"\"\n    def __init__(self, video_urls: List[str]):\n        \"\"\"Initialize with bilibili url.\"\"\"\n        self.video_urls = video_urls\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load from bilibili url.\"\"\"\n        results = []\n        for url in self.video_urls:\n            transcript, video_info = self._get_bilibili_subs_and_info(url)\n            doc = Document(page_content=transcript, metadata=video_info)\n            results.append(doc)\n        return results\n    def _get_bilibili_subs_and_info(self, url: str) -> Tuple[str, dict]:\n        try:\n            from bilibili_api import sync, video\n        except ImportError:\n            raise ValueError(\n                \"requests package not found, please install it with \"\n                \"`pip install bilibili-api-python`\"\n            )\n        bvid = re.search(r\"BV\\w+\", url)\n        if bvid is not None:\n            v = video.Video(bvid=bvid.group())\n        else:\n            aid = re.search(r\"av[0-9]+\", url)\n            if aid is not None:\n                try:\n                    v = video.Video(aid=int(aid.group()[2:]))\n                except AttributeError:\n                    raise ValueError(f\"{url} is not bilibili url.\")\n            else:\n                raise ValueError(f\"{url} is not bilibili url.\")\n        video_info = sync(v.get_info())", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bilibili.html"}1103{"id": "8601568e7855-1", "text": "video_info = sync(v.get_info())\n        video_info.update({\"url\": url})\n        # Get subtitle url\n        subtitle = video_info.pop(\"subtitle\")\n        sub_list = subtitle[\"list\"]\n        if sub_list:\n            sub_url = sub_list[0][\"subtitle_url\"]\n            result = requests.get(sub_url)\n            raw_sub_titles = json.loads(result.content)[\"body\"]\n            raw_transcript = \" \".join([c[\"content\"] for c in raw_sub_titles])\n            raw_transcript_with_meta_info = (\n                f\"Video Title: {video_info['title']},\"\n                f\"description: {video_info['desc']}\\n\\n\"\n                f\"Transcript: {raw_transcript}\"\n            )\n            return raw_transcript_with_meta_info, video_info\n        else:\n            raw_transcript = \"\"\n            warnings.warn(\n                f\"\"\"\n                No subtitles found for video: {url}.\n                Return Empty transcript.\n                \"\"\"\n            )\n            return raw_transcript, video_info\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/bilibili.html"}1104{"id": "6110a84fab8e-0", "text": "Source code for langchain.document_loaders.arxiv\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utilities.arxiv import ArxivAPIWrapper\n[docs]class ArxivLoader(BaseLoader):\n    \"\"\"Loads a query result from arxiv.org into a list of Documents.\n    Each document represents one Document.\n    The loader converts the original PDF format into the text.\n    \"\"\"\n    def __init__(\n        self,\n        query: str,\n        load_max_docs: Optional[int] = 100,\n        load_all_available_meta: Optional[bool] = False,\n    ):\n        self.query = query\n        self.load_max_docs = load_max_docs\n        self.load_all_available_meta = load_all_available_meta\n[docs]    def load(self) -> List[Document]:\n        arxiv_client = ArxivAPIWrapper(\n            load_max_docs=self.load_max_docs,\n            load_all_available_meta=self.load_all_available_meta,\n        )\n        docs = arxiv_client.load(self.query)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/arxiv.html"}1105{"id": "a737f8740922-0", "text": "Source code for langchain.document_loaders.odt\n\"\"\"Loader that loads Open Office ODT files.\"\"\"\nfrom typing import Any, List\nfrom langchain.document_loaders.unstructured import (\n    UnstructuredFileLoader,\n    validate_unstructured_version,\n)\n[docs]class UnstructuredODTLoader(UnstructuredFileLoader):\n    \"\"\"Loader that uses unstructured to load open office ODT files.\"\"\"\n    def __init__(\n        self, file_path: str, mode: str = \"single\", **unstructured_kwargs: Any\n    ):\n        validate_unstructured_version(min_unstructured_version=\"0.6.3\")\n        super().__init__(file_path=file_path, mode=mode, **unstructured_kwargs)\n    def _get_elements(self) -> List:\n        from unstructured.partition.odt import partition_odt\n        return partition_odt(filename=self.file_path, **self.unstructured_kwargs)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/odt.html"}1106{"id": "7bf057740dfa-0", "text": "Source code for langchain.document_loaders.college_confidential\n\"\"\"Loader that loads College Confidential.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class CollegeConfidentialLoader(WebBaseLoader):\n    \"\"\"Loader that loads College Confidential webpages.\"\"\"\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load webpage.\"\"\"\n        soup = self.scrape()\n        text = soup.select_one(\"main[class='skin-handler']\").text\n        metadata = {\"source\": self.web_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/college_confidential.html"}1107{"id": "7b2c7d0b6fd2-0", "text": "Source code for langchain.document_loaders.whatsapp_chat\nimport re\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\ndef concatenate_rows(date: str, sender: str, text: str) -> str:\n    \"\"\"Combine message information in a readable format ready to be used.\"\"\"\n    return f\"{sender} on {date}: {text}\\n\\n\"\n[docs]class WhatsAppChatLoader(BaseLoader):\n    \"\"\"Loader that loads WhatsApp messages text file.\"\"\"\n    def __init__(self, path: str):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        p = Path(self.file_path)\n        text_content = \"\"\n        with open(p, encoding=\"utf8\") as f:\n            lines = f.readlines()\n        message_line_regex = r\"\"\"\n            \\[?\n            (\n                \\d{1,2}\n                [\\/.]\n                \\d{1,2}\n                [\\/.]\n                \\d{2,4}\n                ,\\s\n                \\d{1,2}\n                :\\d{2}\n                (?:\n                    :\\d{2}\n                )?\n                (?:[ _](?:AM|PM))?\n            )\n            \\]?\n            [\\s-]*\n            ([~\\w\\s]+)\n            [:]+\n            \\s\n            (.+)\n        \"\"\"\n        for line in lines:\n            result = re.match(message_line_regex, line.strip(), flags=re.VERBOSE)\n            if result:\n                date, sender, text = result.groups()\n                text_content += concatenate_rows(date, sender, text)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/whatsapp_chat.html"}1108{"id": "7b2c7d0b6fd2-1", "text": "text_content += concatenate_rows(date, sender, text)\n        metadata = {\"source\": str(p)}\n        return [Document(page_content=text_content, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/whatsapp_chat.html"}1109{"id": "b56f364a632e-0", "text": "Source code for langchain.document_loaders.stripe\n\"\"\"Loader that fetches data from Stripe\"\"\"\nimport json\nimport urllib.request\nfrom typing import List, Optional\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.utils import get_from_env, stringify_dict\nSTRIPE_ENDPOINTS = {\n    \"balance_transactions\": \"https://api.stripe.com/v1/balance_transactions\",\n    \"charges\": \"https://api.stripe.com/v1/charges\",\n    \"customers\": \"https://api.stripe.com/v1/customers\",\n    \"events\": \"https://api.stripe.com/v1/events\",\n    \"refunds\": \"https://api.stripe.com/v1/refunds\",\n    \"disputes\": \"https://api.stripe.com/v1/disputes\",\n}\n[docs]class StripeLoader(BaseLoader):\n    def __init__(self, resource: str, access_token: Optional[str] = None) -> None:\n        self.resource = resource\n        access_token = access_token or get_from_env(\n            \"access_token\", \"STRIPE_ACCESS_TOKEN\"\n        )\n        self.headers = {\"Authorization\": f\"Bearer {access_token}\"}\n    def _make_request(self, url: str) -> List[Document]:\n        request = urllib.request.Request(url, headers=self.headers)\n        with urllib.request.urlopen(request) as response:\n            json_data = json.loads(response.read().decode())\n            text = stringify_dict(json_data)\n            metadata = {\"source\": url}\n            return [Document(page_content=text, metadata=metadata)]\n    def _get_resource(self) -> List[Document]:\n        endpoint = STRIPE_ENDPOINTS.get(self.resource)\n        if endpoint is None:\n            return []\n        return self._make_request(endpoint)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/stripe.html"}1110{"id": "b56f364a632e-1", "text": "if endpoint is None:\n            return []\n        return self._make_request(endpoint)\n[docs]    def load(self) -> List[Document]:\n        return self._get_resource()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/stripe.html"}1111{"id": "04b6b1a4ecc9-0", "text": "Source code for langchain.document_loaders.conllu\n\"\"\"Load CoNLL-U files.\"\"\"\nimport csv\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class CoNLLULoader(BaseLoader):\n    \"\"\"Load CoNLL-U files.\"\"\"\n    def __init__(self, file_path: str):\n        \"\"\"Initialize with file path.\"\"\"\n        self.file_path = file_path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load from file path.\"\"\"\n        with open(self.file_path, encoding=\"utf8\") as f:\n            tsv = list(csv.reader(f, delimiter=\"\\t\"))\n            # If len(line) > 1, the line is not a comment\n            lines = [line for line in tsv if len(line) > 1]\n        text = \"\"\n        for i, line in enumerate(lines):\n            # Do not add a space after a punctuation mark or at the end of the sentence\n            if line[9] == \"SpaceAfter=No\" or i == len(lines) - 1:\n                text += line[1]\n            else:\n                text += line[1] + \" \"\n        metadata = {\"source\": self.file_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/conllu.html"}1112{"id": "6f8f881250ba-0", "text": "Source code for langchain.document_loaders.roam\n\"\"\"Loader that loads Roam directory dump.\"\"\"\nfrom pathlib import Path\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class RoamLoader(BaseLoader):\n    \"\"\"Loader that loads Roam files from disk.\"\"\"\n    def __init__(self, path: str):\n        \"\"\"Initialize with path.\"\"\"\n        self.file_path = path\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        ps = list(Path(self.file_path).glob(\"**/*.md\"))\n        docs = []\n        for p in ps:\n            with open(p) as f:\n                text = f.read()\n            metadata = {\"source\": str(p)}\n            docs.append(Document(page_content=text, metadata=metadata))\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/roam.html"}1113{"id": "25f9a58911d7-0", "text": "Source code for langchain.document_loaders.gcs_file\n\"\"\"Loading logic for loading documents from a GCS file.\"\"\"\nimport os\nimport tempfile\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.unstructured import UnstructuredFileLoader\n[docs]class GCSFileLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from GCS.\"\"\"\n    def __init__(self, project_name: str, bucket: str, blob: str):\n        \"\"\"Initialize with bucket and key name.\"\"\"\n        self.bucket = bucket\n        self.blob = blob\n        self.project_name = project_name\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            from google.cloud import storage\n        except ImportError:\n            raise ValueError(\n                \"Could not import google-cloud-storage python package. \"\n                \"Please install it with `pip install google-cloud-storage`.\"\n            )\n        # Initialise a client\n        storage_client = storage.Client(self.project_name)\n        # Create a bucket object for our bucket\n        bucket = storage_client.get_bucket(self.bucket)\n        # Create a blob object from the filepath\n        blob = bucket.blob(self.blob)\n        with tempfile.TemporaryDirectory() as temp_dir:\n            file_path = f\"{temp_dir}/{self.blob}\"\n            os.makedirs(os.path.dirname(file_path), exist_ok=True)\n            # Download the file to a destination\n            blob.download_to_filename(file_path)\n            loader = UnstructuredFileLoader(file_path)\n            return loader.load()\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/gcs_file.html"}1114{"id": "8dfccd5a9c7b-0", "text": "Source code for langchain.document_loaders.azlyrics\n\"\"\"Loader that loads AZLyrics.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.web_base import WebBaseLoader\n[docs]class AZLyricsLoader(WebBaseLoader):\n    \"\"\"Loader that loads AZLyrics webpages.\"\"\"\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load webpage.\"\"\"\n        soup = self.scrape()\n        title = soup.title.text\n        lyrics = soup.find_all(\"div\", {\"class\": \"\"})[2].text\n        text = title + lyrics\n        metadata = {\"source\": self.web_path}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/azlyrics.html"}1115{"id": "39b241b1c57f-0", "text": "Source code for langchain.document_loaders.ifixit\n\"\"\"Loader that loads iFixit data.\"\"\"\nfrom typing import List, Optional\nimport requests\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\nfrom langchain.document_loaders.web_base import WebBaseLoader\nIFIXIT_BASE_URL = \"https://www.ifixit.com/api/2.0\"\n[docs]class IFixitLoader(BaseLoader):\n    \"\"\"Load iFixit repair guides, device wikis and answers.\n    iFixit is the largest, open repair community on the web. The site contains nearly\n    100k repair manuals, 200k Questions & Answers on 42k devices, and all the data is\n    licensed under CC-BY.\n    This loader will allow you to download the text of a repair guide, text of Q&A's\n    and wikis from devices on iFixit using their open APIs and web scraping.\n    \"\"\"\n    def __init__(self, web_path: str):\n        \"\"\"Initialize with web path.\"\"\"\n        if not web_path.startswith(\"https://www.ifixit.com\"):\n            raise ValueError(\"web path must start with 'https://www.ifixit.com'\")\n        path = web_path.replace(\"https://www.ifixit.com\", \"\")\n        allowed_paths = [\"/Device\", \"/Guide\", \"/Answers\", \"/Teardown\"]\n        \"\"\" TODO: Add /Wiki \"\"\"\n        if not any(path.startswith(allowed_path) for allowed_path in allowed_paths):\n            raise ValueError(\n                \"web path must start with /Device, /Guide, /Teardown or /Answers\"\n            )\n        pieces = [x for x in path.split(\"/\") if x]\n        \"\"\"Teardowns are just guides by a different name\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/ifixit.html"}1116{"id": "39b241b1c57f-1", "text": "\"\"\"Teardowns are just guides by a different name\"\"\"\n        self.page_type = pieces[0] if pieces[0] != \"Teardown\" else \"Guide\"\n        if self.page_type == \"Guide\" or self.page_type == \"Answers\":\n            self.id = pieces[2]\n        else:\n            self.id = pieces[1]\n        self.web_path = web_path\n[docs]    def load(self) -> List[Document]:\n        if self.page_type == \"Device\":\n            return self.load_device()\n        elif self.page_type == \"Guide\" or self.page_type == \"Teardown\":\n            return self.load_guide()\n        elif self.page_type == \"Answers\":\n            return self.load_questions_and_answers()\n        else:\n            raise ValueError(\"Unknown page type: \" + self.page_type)\n[docs]    @staticmethod\n    def load_suggestions(query: str = \"\", doc_type: str = \"all\") -> List[Document]:\n        res = requests.get(\n            IFIXIT_BASE_URL + \"/suggest/\" + query + \"?doctypes=\" + doc_type\n        )\n        if res.status_code != 200:\n            raise ValueError(\n                'Could not load suggestions for \"' + query + '\"\\n' + res.json()\n            )\n        data = res.json()\n        results = data[\"results\"]\n        output = []\n        for result in results:\n            try:\n                loader = IFixitLoader(result[\"url\"])\n                if loader.page_type == \"Device\":\n                    output += loader.load_device(include_guides=False)\n                else:\n                    output += loader.load()\n            except ValueError:\n                continue\n        return output\n[docs]    def load_questions_and_answers(\n        self, url_override: Optional[str] = None\n    ) -> List[Document]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/ifixit.html"}1117{"id": "39b241b1c57f-2", "text": "self, url_override: Optional[str] = None\n    ) -> List[Document]:\n        loader = WebBaseLoader(self.web_path if url_override is None else url_override)\n        soup = loader.scrape()\n        output = []\n        title = soup.find(\"h1\", \"post-title\").text\n        output.append(\"# \" + title)\n        output.append(soup.select_one(\".post-content .post-text\").text.strip())\n        answersHeader = soup.find(\"div\", \"post-answers-header\")\n        if answersHeader:\n            output.append(\"\\n## \" + answersHeader.text.strip())\n        for answer in soup.select(\".js-answers-list .post.post-answer\"):\n            if answer.has_attr(\"itemprop\") and \"acceptedAnswer\" in answer[\"itemprop\"]:\n                output.append(\"\\n### Accepted Answer\")\n            elif \"post-helpful\" in answer[\"class\"]:\n                output.append(\"\\n### Most Helpful Answer\")\n            else:\n                output.append(\"\\n### Other Answer\")\n            output += [\n                a.text.strip() for a in answer.select(\".post-content .post-text\")\n            ]\n            output.append(\"\\n\")\n        text = \"\\n\".join(output).strip()\n        metadata = {\"source\": self.web_path, \"title\": title}\n        return [Document(page_content=text, metadata=metadata)]\n[docs]    def load_device(\n        self, url_override: Optional[str] = None, include_guides: bool = True\n    ) -> List[Document]:\n        documents = []\n        if url_override is None:\n            url = IFIXIT_BASE_URL + \"/wikis/CATEGORY/\" + self.id\n        else:\n            url = url_override\n        res = requests.get(url)\n        data = res.json()\n        text = \"\\n\".join(\n            [\n                data[key]", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/ifixit.html"}1118{"id": "39b241b1c57f-3", "text": "text = \"\\n\".join(\n            [\n                data[key]\n                for key in [\"title\", \"description\", \"contents_raw\"]\n                if key in data\n            ]\n        ).strip()\n        metadata = {\"source\": self.web_path, \"title\": data[\"title\"]}\n        documents.append(Document(page_content=text, metadata=metadata))\n        if include_guides:\n            \"\"\"Load and return documents for each guide linked to from the device\"\"\"\n            guide_urls = [guide[\"url\"] for guide in data[\"guides\"]]\n            for guide_url in guide_urls:\n                documents.append(IFixitLoader(guide_url).load()[0])\n        return documents\n[docs]    def load_guide(self, url_override: Optional[str] = None) -> List[Document]:\n        if url_override is None:\n            url = IFIXIT_BASE_URL + \"/guides/\" + self.id\n        else:\n            url = url_override\n        res = requests.get(url)\n        if res.status_code != 200:\n            raise ValueError(\n                \"Could not load guide: \" + self.web_path + \"\\n\" + res.json()\n            )\n        data = res.json()\n        doc_parts = [\"# \" + data[\"title\"], data[\"introduction_raw\"]]\n        doc_parts.append(\"\\n\\n###Tools Required:\")\n        if len(data[\"tools\"]) == 0:\n            doc_parts.append(\"\\n - None\")\n        else:\n            for tool in data[\"tools\"]:\n                doc_parts.append(\"\\n - \" + tool[\"text\"])\n        doc_parts.append(\"\\n\\n###Parts Required:\")\n        if len(data[\"parts\"]) == 0:\n            doc_parts.append(\"\\n - None\")\n        else:\n            for part in data[\"parts\"]:\n                doc_parts.append(\"\\n - \" + part[\"text\"])", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/ifixit.html"}1119{"id": "39b241b1c57f-4", "text": "doc_parts.append(\"\\n - \" + part[\"text\"])\n        for row in data[\"steps\"]:\n            doc_parts.append(\n                \"\\n\\n## \"\n                + (\n                    row[\"title\"]\n                    if row[\"title\"] != \"\"\n                    else \"Step {}\".format(row[\"orderby\"])\n                )\n            )\n            for line in row[\"lines\"]:\n                doc_parts.append(line[\"text_raw\"])\n        doc_parts.append(data[\"conclusion_raw\"])\n        text = \"\\n\".join(doc_parts)\n        metadata = {\"source\": self.web_path, \"title\": data[\"title\"]}\n        return [Document(page_content=text, metadata=metadata)]\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/ifixit.html"}1120{"id": "ba9bb3e2689c-0", "text": "Source code for langchain.document_loaders.azure_blob_storage_container\n\"\"\"Loading logic for loading documents from an Azure Blob Storage container.\"\"\"\nfrom typing import List\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.azure_blob_storage_file import (\n    AzureBlobStorageFileLoader,\n)\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class AzureBlobStorageContainerLoader(BaseLoader):\n    \"\"\"Loading logic for loading documents from Azure Blob Storage.\"\"\"\n    def __init__(self, conn_str: str, container: str, prefix: str = \"\"):\n        \"\"\"Initialize with connection string, container and blob prefix.\"\"\"\n        self.conn_str = conn_str\n        self.container = container\n        self.prefix = prefix\n[docs]    def load(self) -> List[Document]:\n        \"\"\"Load documents.\"\"\"\n        try:\n            from azure.storage.blob import ContainerClient\n        except ImportError as exc:\n            raise ValueError(\n                \"Could not import azure storage blob python package. \"\n                \"Please install it with `pip install azure-storage-blob`.\"\n            ) from exc\n        container = ContainerClient.from_connection_string(\n            conn_str=self.conn_str, container_name=self.container\n        )\n        docs = []\n        blob_list = container.list_blobs(name_starts_with=self.prefix)\n        for blob in blob_list:\n            loader = AzureBlobStorageFileLoader(\n                self.conn_str, self.container, blob.name  # type: ignore\n            )\n            docs.extend(loader.load())\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/azure_blob_storage_container.html"}1121{"id": "7a1b3976d8c6-0", "text": "Source code for langchain.document_loaders.duckdb_loader\nfrom typing import Dict, List, Optional, cast\nfrom langchain.docstore.document import Document\nfrom langchain.document_loaders.base import BaseLoader\n[docs]class DuckDBLoader(BaseLoader):\n    \"\"\"Loads a query result from DuckDB into a list of documents.\n    Each document represents one row of the result. The `page_content_columns`\n    are written into the `page_content` of the document. The `metadata_columns`\n    are written into the `metadata` of the document. By default, all columns\n    are written into the `page_content` and none into the `metadata`.\n    \"\"\"\n    def __init__(\n        self,\n        query: str,\n        database: str = \":memory:\",\n        read_only: bool = False,\n        config: Optional[Dict[str, str]] = None,\n        page_content_columns: Optional[List[str]] = None,\n        metadata_columns: Optional[List[str]] = None,\n    ):\n        self.query = query\n        self.database = database\n        self.read_only = read_only\n        self.config = config or {}\n        self.page_content_columns = page_content_columns\n        self.metadata_columns = metadata_columns\n[docs]    def load(self) -> List[Document]:\n        try:\n            import duckdb\n        except ImportError:\n            raise ImportError(\n                \"Could not import duckdb python package. \"\n                \"Please install it with `pip install duckdb`.\"\n            )\n        docs = []\n        with duckdb.connect(\n            database=self.database, read_only=self.read_only, config=self.config\n        ) as con:\n            query_result = con.execute(self.query)\n            results = query_result.fetchall()\n            description = cast(list, query_result.description)", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/duckdb_loader.html"}1122{"id": "7a1b3976d8c6-1", "text": "results = query_result.fetchall()\n            description = cast(list, query_result.description)\n            field_names = [c[0] for c in description]\n            if self.page_content_columns is None:\n                page_content_columns = field_names\n            else:\n                page_content_columns = self.page_content_columns\n            if self.metadata_columns is None:\n                metadata_columns = []\n            else:\n                metadata_columns = self.metadata_columns\n            for result in results:\n                page_content = \"\\n\".join(\n                    f\"{column}: {result[field_names.index(column)]}\"\n                    for column in page_content_columns\n                )\n                metadata = {\n                    column: result[field_names.index(column)]\n                    for column in metadata_columns\n                }\n                doc = Document(page_content=page_content, metadata=metadata)\n                docs.append(doc)\n        return docs\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/document_loaders/duckdb_loader.html"}1123{"id": "ea0610033e38-0", "text": "Source code for langchain.llms.databricks\nimport os\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Callable, Dict, List, Optional\nimport requests\nfrom pydantic import BaseModel, Extra, Field, PrivateAttr, root_validator, validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\n__all__ = [\"Databricks\"]\nclass _DatabricksClientBase(BaseModel, ABC):\n    \"\"\"A base JSON API client that talks to Databricks.\"\"\"\n    api_url: str\n    api_token: str\n    def post_raw(self, request: Any) -> Any:\n        headers = {\"Authorization\": f\"Bearer {self.api_token}\"}\n        response = requests.post(self.api_url, headers=headers, json=request)\n        # TODO: error handling and automatic retries\n        if not response.ok:\n            raise ValueError(f\"HTTP {response.status_code} error: {response.text}\")\n        return response.json()\n    @abstractmethod\n    def post(self, request: Any) -> Any:\n        ...\nclass _DatabricksServingEndpointClient(_DatabricksClientBase):\n    \"\"\"An API client that talks to a Databricks serving endpoint.\"\"\"\n    host: str\n    endpoint_name: str\n    @root_validator(pre=True)\n    def set_api_url(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        if \"api_url\" not in values:\n            host = values[\"host\"]\n            endpoint_name = values[\"endpoint_name\"]\n            api_url = f\"https://{host}/serving-endpoints/{endpoint_name}/invocations\"\n            values[\"api_url\"] = api_url\n        return values\n    def post(self, request: Any) -> Any:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1124{"id": "ea0610033e38-1", "text": "return values\n    def post(self, request: Any) -> Any:\n        # See https://docs.databricks.com/machine-learning/model-serving/score-model-serving-endpoints.html\n        wrapped_request = {\"dataframe_records\": [request]}\n        response = self.post_raw(wrapped_request)[\"predictions\"]\n        # For a single-record query, the result is not a list.\n        if isinstance(response, list):\n            response = response[0]\n        return response\nclass _DatabricksClusterDriverProxyClient(_DatabricksClientBase):\n    \"\"\"An API client that talks to a Databricks cluster driver proxy app.\"\"\"\n    host: str\n    cluster_id: str\n    cluster_driver_port: str\n    @root_validator(pre=True)\n    def set_api_url(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        if \"api_url\" not in values:\n            host = values[\"host\"]\n            cluster_id = values[\"cluster_id\"]\n            port = values[\"cluster_driver_port\"]\n            api_url = f\"https://{host}/driver-proxy-api/o/0/{cluster_id}/{port}\"\n            values[\"api_url\"] = api_url\n        return values\n    def post(self, request: Any) -> Any:\n        return self.post_raw(request)\ndef get_repl_context() -> Any:\n    \"\"\"Gets the notebook REPL context if running inside a Databricks notebook.\n    Returns None otherwise.\n    \"\"\"\n    try:\n        from dbruntime.databricks_repl_context import get_context\n        return get_context()\n    except ImportError:\n        raise ValueError(\n            \"Cannot access dbruntime, not running inside a Databricks notebook.\"\n        )\ndef get_default_host() -> str:\n    \"\"\"Gets the default Databricks workspace hostname.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1125{"id": "ea0610033e38-2", "text": "def get_default_host() -> str:\n    \"\"\"Gets the default Databricks workspace hostname.\n    Raises an error if the hostname cannot be automatically determined.\n    \"\"\"\n    host = os.getenv(\"DATABRICKS_HOST\")\n    if not host:\n        try:\n            host = get_repl_context().browserHostName\n            if not host:\n                raise ValueError(\"context doesn't contain browserHostName.\")\n        except Exception as e:\n            raise ValueError(\n                \"host was not set and cannot be automatically inferred. Set \"\n                f\"environment variable 'DATABRICKS_HOST'. Received error: {e}\"\n            )\n    # TODO: support Databricks CLI profile\n    host = host.lstrip(\"https://\").lstrip(\"http://\").rstrip(\"/\")\n    return host\ndef get_default_api_token() -> str:\n    \"\"\"Gets the default Databricks personal access token.\n    Raises an error if the token cannot be automatically determined.\n    \"\"\"\n    if api_token := os.getenv(\"DATABRICKS_API_TOKEN\"):\n        return api_token\n    try:\n        api_token = get_repl_context().apiToken\n        if not api_token:\n            raise ValueError(\"context doesn't contain apiToken.\")\n    except Exception as e:\n        raise ValueError(\n            \"api_token was not set and cannot be automatically inferred. Set \"\n            f\"environment variable 'DATABRICKS_API_TOKEN'. Received error: {e}\"\n        )\n    # TODO: support Databricks CLI profile\n    return api_token\n[docs]class Databricks(LLM):\n    \"\"\"LLM wrapper around a Databricks serving endpoint or a cluster driver proxy app.\n    It supports two endpoint types:\n    * **Serving endpoint** (recommended for both production and development).", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1126{"id": "ea0610033e38-3", "text": "* **Serving endpoint** (recommended for both production and development).\n      We assume that an LLM was registered and deployed to a serving endpoint.\n      To wrap it as an LLM you must have \"Can Query\" permission to the endpoint.\n      Set ``endpoint_name`` accordingly and do not set ``cluster_id`` and\n      ``cluster_driver_port``.\n      The expected model signature is:\n      * inputs::\n          [{\"name\": \"prompt\", \"type\": \"string\"},\n           {\"name\": \"stop\", \"type\": \"list[string]\"}]\n      * outputs: ``[{\"type\": \"string\"}]``\n    * **Cluster driver proxy app** (recommended for interactive development).\n      One can load an LLM on a Databricks interactive cluster and start a local HTTP\n      server on the driver node to serve the model at ``/`` using HTTP POST method\n      with JSON input/output.\n      Please use a port number between ``[3000, 8000]`` and let the server listen to\n      the driver IP address or simply ``0.0.0.0`` instead of localhost only.\n      To wrap it as an LLM you must have \"Can Attach To\" permission to the cluster.\n      Set ``cluster_id`` and ``cluster_driver_port`` and do not set ``endpoint_name``.\n      The expected server schema (using JSON schema) is:\n      * inputs::\n          {\"type\": \"object\",\n           \"properties\": {\n              \"prompt\": {\"type\": \"string\"},\n              \"stop\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}},\n           \"required\": [\"prompt\"]}`\n      * outputs: ``{\"type\": \"string\"}``\n    If the endpoint model signature is different or you want to set extra params,", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1127{"id": "ea0610033e38-4", "text": "If the endpoint model signature is different or you want to set extra params,\n    you can use `transform_input_fn` and `transform_output_fn` to apply necessary\n    transformations before and after the query.\n    \"\"\"\n    host: str = Field(default_factory=get_default_host)\n    \"\"\"Databricks workspace hostname.\n    If not provided, the default value is determined by\n    * the ``DATABRICKS_HOST`` environment variable if present, or\n    * the hostname of the current Databricks workspace if running inside\n      a Databricks notebook attached to an interactive cluster in \"single user\"\n      or \"no isolation shared\" mode.\n    \"\"\"\n    api_token: str = Field(default_factory=get_default_api_token)\n    \"\"\"Databricks personal access token.\n    If not provided, the default value is determined by\n    * the ``DATABRICKS_API_TOKEN`` environment variable if present, or\n    * an automatically generated temporary token if running inside a Databricks\n      notebook attached to an interactive cluster in \"single user\" or\n      \"no isolation shared\" mode.\n    \"\"\"\n    endpoint_name: Optional[str] = None\n    \"\"\"Name of the model serving endpont.\n    You must specify the endpoint name to connect to a model serving endpoint.\n    You must not set both ``endpoint_name`` and ``cluster_id``.\n    \"\"\"\n    cluster_id: Optional[str] = None\n    \"\"\"ID of the cluster if connecting to a cluster driver proxy app.\n    If neither ``endpoint_name`` nor ``cluster_id`` is not provided and the code runs\n    inside a Databricks notebook attached to an interactive cluster in \"single user\"\n    or \"no isolation shared\" mode, the current cluster ID is used as default.\n    You must not set both ``endpoint_name`` and ``cluster_id``.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1128{"id": "ea0610033e38-5", "text": "You must not set both ``endpoint_name`` and ``cluster_id``.\n    \"\"\"\n    cluster_driver_port: Optional[str] = None\n    \"\"\"The port number used by the HTTP server running on the cluster driver node.\n    The server should listen on the driver IP address or simply ``0.0.0.0`` to connect.\n    We recommend the server using a port number between ``[3000, 8000]``.\n    \"\"\"\n    model_kwargs: Optional[Dict[str, Any]] = None\n    \"\"\"Extra parameters to pass to the endpoint.\"\"\"\n    transform_input_fn: Optional[Callable] = None\n    \"\"\"A function that transforms ``{prompt, stop, **kwargs}`` into a JSON-compatible\n    request object that the endpoint accepts.\n    For example, you can apply a prompt template to the input prompt.\n    \"\"\"\n    transform_output_fn: Optional[Callable[..., str]] = None\n    \"\"\"A function that transforms the output from the endpoint to the generated text.\n    \"\"\"\n    _client: _DatabricksClientBase = PrivateAttr()\n    class Config:\n        extra = Extra.forbid\n        underscore_attrs_are_private = True\n    @validator(\"cluster_id\", always=True)\n    def set_cluster_id(cls, v: Any, values: Dict[str, Any]) -> Optional[str]:\n        if v and values[\"endpoint_name\"]:\n            raise ValueError(\"Cannot set both endpoint_name and cluster_id.\")\n        elif values[\"endpoint_name\"]:\n            return None\n        elif v:\n            return v\n        else:\n            try:\n                if v := get_repl_context().clusterId:\n                    return v\n                raise ValueError(\"Context doesn't contain clusterId.\")\n            except Exception as e:\n                raise ValueError(\n                    \"Neither endpoint_name nor cluster_id was set. \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1129{"id": "ea0610033e38-6", "text": "raise ValueError(\n                    \"Neither endpoint_name nor cluster_id was set. \"\n                    \"And the cluster_id cannot be automatically determined. Received\"\n                    f\" error: {e}\"\n                )\n    @validator(\"cluster_driver_port\", always=True)\n    def set_cluster_driver_port(cls, v: Any, values: Dict[str, Any]) -> Optional[str]:\n        if v and values[\"endpoint_name\"]:\n            raise ValueError(\"Cannot set both endpoint_name and cluster_driver_port.\")\n        elif values[\"endpoint_name\"]:\n            return None\n        elif v is None:\n            raise ValueError(\n                \"Must set cluster_driver_port to connect to a cluster driver.\"\n            )\n        elif int(v) <= 0:\n            raise ValueError(f\"Invalid cluster_driver_port: {v}\")\n        else:\n            return v\n    @validator(\"model_kwargs\", always=True)\n    def set_model_kwargs(cls, v: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:\n        if v:\n            assert \"prompt\" not in v, \"model_kwargs must not contain key 'prompt'\"\n            assert \"stop\" not in v, \"model_kwargs must not contain key 'stop'\"\n        return v\n    def __init__(self, **data: Any):\n        super().__init__(**data)\n        if self.endpoint_name:\n            self._client = _DatabricksServingEndpointClient(\n                host=self.host,\n                api_token=self.api_token,\n                endpoint_name=self.endpoint_name,\n            )\n        elif self.cluster_id and self.cluster_driver_port:\n            self._client = _DatabricksClusterDriverProxyClient(\n                host=self.host,\n                api_token=self.api_token,\n                cluster_id=self.cluster_id,\n                cluster_driver_port=self.cluster_driver_port,\n            )\n        else:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1130{"id": "ea0610033e38-7", "text": "cluster_driver_port=self.cluster_driver_port,\n            )\n        else:\n            raise ValueError(\n                \"Must specify either endpoint_name or cluster_id/cluster_driver_port.\"\n            )\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"databricks\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Queries the LLM endpoint with the given prompt and stop sequence.\"\"\"\n        # TODO: support callbacks\n        request = {\"prompt\": prompt, \"stop\": stop}\n        if self.model_kwargs:\n            request.update(self.model_kwargs)\n        if self.transform_input_fn:\n            request = self.transform_input_fn(**request)\n        response = self._client.post(request)\n        if self.transform_output_fn:\n            response = self.transform_output_fn(response)\n        return response\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/databricks.html"}1131{"id": "ac1fd16c24f7-0", "text": "Source code for langchain.llms.huggingface_hub\n\"\"\"Wrapper around HuggingFace APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\nDEFAULT_REPO_ID = \"gpt2\"\nVALID_TASKS = (\"text2text-generation\", \"text-generation\", \"summarization\")\n[docs]class HuggingFaceHub(LLM):\n    \"\"\"Wrapper around HuggingFaceHub  models.\n    To use, you should have the ``huggingface_hub`` python package installed, and the\n    environment variable ``HUGGINGFACEHUB_API_TOKEN`` set with your API token, or pass\n    it as a named parameter to the constructor.\n    Only supports `text-generation`, `text2text-generation` and `summarization` for now.\n    Example:\n        .. code-block:: python\n            from langchain.llms import HuggingFaceHub\n            hf = HuggingFaceHub(repo_id=\"gpt2\", huggingfacehub_api_token=\"my-api-key\")\n    \"\"\"\n    client: Any  #: :meta private:\n    repo_id: str = DEFAULT_REPO_ID\n    \"\"\"Model name to use.\"\"\"\n    task: Optional[str] = None\n    \"\"\"Task to call the model with.\n    Should be a task that returns `generated_text` or `summary_text`.\"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    huggingfacehub_api_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_hub.html"}1132{"id": "ac1fd16c24f7-1", "text": "\"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        huggingfacehub_api_token = get_from_dict_or_env(\n            values, \"huggingfacehub_api_token\", \"HUGGINGFACEHUB_API_TOKEN\"\n        )\n        try:\n            from huggingface_hub.inference_api import InferenceApi\n            repo_id = values[\"repo_id\"]\n            client = InferenceApi(\n                repo_id=repo_id,\n                token=huggingfacehub_api_token,\n                task=values.get(\"task\"),\n            )\n            if client.task not in VALID_TASKS:\n                raise ValueError(\n                    f\"Got invalid task {client.task}, \"\n                    f\"currently only {VALID_TASKS} are supported\"\n                )\n            values[\"client\"] = client\n        except ImportError:\n            raise ValueError(\n                \"Could not import huggingface_hub python package. \"\n                \"Please install it with `pip install huggingface_hub`.\"\n            )\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        return {\n            **{\"repo_id\": self.repo_id, \"task\": self.task},\n            **{\"model_kwargs\": _model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"huggingface_hub\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_hub.html"}1133{"id": "ac1fd16c24f7-2", "text": "self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to HuggingFace Hub's inference endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = hf(\"Tell me a joke.\")\n        \"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        response = self.client(inputs=prompt, params=_model_kwargs)\n        if \"error\" in response:\n            raise ValueError(f\"Error raised by inference API: {response['error']}\")\n        if self.client.task == \"text-generation\":\n            # Text generation return includes the starter text.\n            text = response[0][\"generated_text\"][len(prompt) :]\n        elif self.client.task == \"text2text-generation\":\n            text = response[0][\"generated_text\"]\n        elif self.client.task == \"summarization\":\n            text = response[0][\"summary_text\"]\n        else:\n            raise ValueError(\n                f\"Got invalid task {self.client.task}, \"\n                f\"currently only {VALID_TASKS} are supported\"\n            )\n        if stop is not None:\n            # This is a bit hacky, but I can't figure out a better way to enforce\n            # stop tokens when making calls to huggingface_hub.\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_hub.html"}1134{"id": "4baadf018099-0", "text": "Source code for langchain.llms.self_hosted\n\"\"\"Run model inference on self-hosted remote hardware.\"\"\"\nimport importlib.util\nimport logging\nimport pickle\nfrom typing import Any, Callable, List, Mapping, Optional\nfrom pydantic import Extra\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nlogger = logging.getLogger(__name__)\ndef _generate_text(\n    pipeline: Any,\n    prompt: str,\n    *args: Any,\n    stop: Optional[List[str]] = None,\n    **kwargs: Any,\n) -> str:\n    \"\"\"Inference function to send to the remote hardware.\n    Accepts a pipeline callable (or, more likely,\n    a key pointing to the model on the cluster's object store)\n    and returns text predictions for each document\n    in the batch.\n    \"\"\"\n    text = pipeline(prompt, *args, **kwargs)\n    if stop is not None:\n        text = enforce_stop_tokens(text, stop)\n    return text\ndef _send_pipeline_to_device(pipeline: Any, device: int) -> Any:\n    \"\"\"Send a pipeline to a device on the cluster.\"\"\"\n    if isinstance(pipeline, str):\n        with open(pipeline, \"rb\") as f:\n            pipeline = pickle.load(f)\n    if importlib.util.find_spec(\"torch\") is not None:\n        import torch\n        cuda_device_count = torch.cuda.device_count()\n        if device < -1 or (device >= cuda_device_count):\n            raise ValueError(\n                f\"Got device=={device}, \"\n                f\"device is required to be within [-1, {cuda_device_count})\"\n            )\n        if device < 0 and cuda_device_count > 0:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted.html"}1135{"id": "4baadf018099-1", "text": ")\n        if device < 0 and cuda_device_count > 0:\n            logger.warning(\n                \"Device has %d GPUs available. \"\n                \"Provide device={deviceId} to `from_model_id` to use available\"\n                \"GPUs for execution. deviceId is -1 for CPU and \"\n                \"can be a positive integer associated with CUDA device id.\",\n                cuda_device_count,\n            )\n        pipeline.device = torch.device(device)\n        pipeline.model = pipeline.model.to(pipeline.device)\n    return pipeline\n[docs]class SelfHostedPipeline(LLM):\n    \"\"\"Run model inference on self-hosted remote hardware.\n    Supported hardware includes auto-launched instances on AWS, GCP, Azure,\n    and Lambda, as well as servers specified\n    by IP address and SSH credentials (such as on-prem, or another\n    cloud like Paperspace, Coreweave, etc.).\n    To use, you should have the ``runhouse`` python package installed.\n    Example for custom pipeline and inference functions:\n        .. code-block:: python\n            from langchain.llms import SelfHostedPipeline\n            from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n            import runhouse as rh\n            def load_pipeline():\n                tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n                model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n                return pipeline(\n                    \"text-generation\", model=model, tokenizer=tokenizer,\n                    max_new_tokens=10\n                )\n            def inference_fn(pipeline, prompt, stop = None):\n                return pipeline(prompt)[0][\"generated_text\"]\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            llm = SelfHostedPipeline(\n                model_load_fn=load_pipeline,", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted.html"}1136{"id": "4baadf018099-2", "text": "llm = SelfHostedPipeline(\n                model_load_fn=load_pipeline,\n                hardware=gpu,\n                model_reqs=model_reqs, inference_fn=inference_fn\n            )\n    Example for <2GB model (can be serialized and sent directly to the server):\n        .. code-block:: python\n            from langchain.llms import SelfHostedPipeline\n            import runhouse as rh\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            my_model = ...\n            llm = SelfHostedPipeline.from_pipeline(\n                pipeline=my_model,\n                hardware=gpu,\n                model_reqs=[\"./\", \"torch\", \"transformers\"],\n            )\n    Example passing model path for larger models:\n        .. code-block:: python\n            from langchain.llms import SelfHostedPipeline\n            import runhouse as rh\n            import pickle\n            from transformers import pipeline\n            generator = pipeline(model=\"gpt2\")\n            rh.blob(pickle.dumps(generator), path=\"models/pipeline.pkl\"\n                ).save().to(gpu, path=\"models\")\n            llm = SelfHostedPipeline.from_pipeline(\n                pipeline=\"models/pipeline.pkl\",\n                hardware=gpu,\n                model_reqs=[\"./\", \"torch\", \"transformers\"],\n            )\n    \"\"\"\n    pipeline_ref: Any  #: :meta private:\n    client: Any  #: :meta private:\n    inference_fn: Callable = _generate_text  #: :meta private:\n    \"\"\"Inference function to send to the remote hardware.\"\"\"\n    hardware: Any\n    \"\"\"Remote hardware to send the inference function to.\"\"\"\n    model_load_fn: Callable\n    \"\"\"Function to load the model remotely on the server.\"\"\"\n    load_fn_kwargs: Optional[dict] = None", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted.html"}1137{"id": "4baadf018099-3", "text": "load_fn_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model load function.\"\"\"\n    model_reqs: List[str] = [\"./\", \"torch\"]\n    \"\"\"Requirements to install on hardware to inference the model.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def __init__(self, **kwargs: Any):\n        \"\"\"Init the pipeline with an auxiliary function.\n        The load function must be in global scope to be imported\n        and run on the server, i.e. in a module and not a REPL or closure.\n        Then, initialize the remote inference function.\n        \"\"\"\n        super().__init__(**kwargs)\n        try:\n            import runhouse as rh\n        except ImportError:\n            raise ImportError(\n                \"Could not import runhouse python package. \"\n                \"Please install it with `pip install runhouse`.\"\n            )\n        remote_load_fn = rh.function(fn=self.model_load_fn).to(\n            self.hardware, reqs=self.model_reqs\n        )\n        _load_fn_kwargs = self.load_fn_kwargs or {}\n        self.pipeline_ref = remote_load_fn.remote(**_load_fn_kwargs)\n        self.client = rh.function(fn=self.inference_fn).to(\n            self.hardware, reqs=self.model_reqs\n        )\n[docs]    @classmethod\n    def from_pipeline(\n        cls,\n        pipeline: Any,\n        hardware: Any,\n        model_reqs: Optional[List[str]] = None,\n        device: int = 0,\n        **kwargs: Any,\n    ) -> LLM:\n        \"\"\"Init the SelfHostedPipeline from a pipeline object or string.\"\"\"\n        if not isinstance(pipeline, str):\n            logger.warning(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted.html"}1138{"id": "4baadf018099-4", "text": "if not isinstance(pipeline, str):\n            logger.warning(\n                \"Serializing pipeline to send to remote hardware. \"\n                \"Note, it can be quite slow\"\n                \"to serialize and send large models with each execution. \"\n                \"Consider sending the pipeline\"\n                \"to the cluster and passing the path to the pipeline instead.\"\n            )\n        load_fn_kwargs = {\"pipeline\": pipeline, \"device\": device}\n        return cls(\n            load_fn_kwargs=load_fn_kwargs,\n            model_load_fn=_send_pipeline_to_device,\n            hardware=hardware,\n            model_reqs=[\"transformers\", \"torch\"] + (model_reqs or []),\n            **kwargs,\n        )\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            **{\"hardware\": self.hardware},\n        }\n    @property\n    def _llm_type(self) -> str:\n        return \"self_hosted_llm\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        return self.client(pipeline=self.pipeline_ref, prompt=prompt, stop=stop)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted.html"}1139{"id": "c7eef00c5567-0", "text": "Source code for langchain.llms.mosaicml\n\"\"\"Wrapper around MosaicML APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\nINSTRUCTION_KEY = \"### Instruction:\"\nRESPONSE_KEY = \"### Response:\"\nINTRO_BLURB = (\n    \"Below is an instruction that describes a task. \"\n    \"Write a response that appropriately completes the request.\"\n)\nPROMPT_FOR_GENERATION_FORMAT = \"\"\"{intro}\n{instruction_key}\n{instruction}\n{response_key}\n\"\"\".format(\n    intro=INTRO_BLURB,\n    instruction_key=INSTRUCTION_KEY,\n    instruction=\"{instruction}\",\n    response_key=RESPONSE_KEY,\n)\n[docs]class MosaicML(LLM):\n    \"\"\"Wrapper around MosaicML's LLM inference service.\n    To use, you should have the\n    environment variable ``MOSAICML_API_TOKEN`` set with your API token, or pass\n    it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.llms import MosaicML\n            endpoint_url = (\n                \"https://models.hosted-on.mosaicml.hosting/mpt-7b-instruct/v1/predict\"\n            )\n            mosaic_llm = MosaicML(\n                endpoint_url=endpoint_url,\n                mosaicml_api_token=\"my-api-key\"\n            )\n    \"\"\"\n    endpoint_url: str = (", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/mosaicml.html"}1140{"id": "c7eef00c5567-1", "text": ")\n    \"\"\"\n    endpoint_url: str = (\n        \"https://models.hosted-on.mosaicml.hosting/mpt-7b-instruct/v1/predict\"\n    )\n    \"\"\"Endpoint URL to use.\"\"\"\n    inject_instruction_format: bool = False\n    \"\"\"Whether to inject the instruction format into the prompt.\"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    retry_sleep: float = 1.0\n    \"\"\"How long to try sleeping for if a rate limit is encountered\"\"\"\n    mosaicml_api_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        mosaicml_api_token = get_from_dict_or_env(\n            values, \"mosaicml_api_token\", \"MOSAICML_API_TOKEN\"\n        )\n        values[\"mosaicml_api_token\"] = mosaicml_api_token\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        return {\n            **{\"endpoint_url\": self.endpoint_url},\n            **{\"model_kwargs\": _model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"mosaicml\"\n    def _transform_prompt(self, prompt: str) -> str:\n        \"\"\"Transform prompt.\"\"\"\n        if self.inject_instruction_format:\n            prompt = PROMPT_FOR_GENERATION_FORMAT.format(\n                instruction=prompt,\n            )\n        return prompt", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/mosaicml.html"}1141{"id": "c7eef00c5567-2", "text": "instruction=prompt,\n            )\n        return prompt\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n        is_retry: bool = False,\n    ) -> str:\n        \"\"\"Call out to a MosaicML LLM inference endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = mosaic_llm(\"Tell me a joke.\")\n        \"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        prompt = self._transform_prompt(prompt)\n        payload = {\"input_strings\": [prompt]}\n        payload.update(_model_kwargs)\n        # HTTP headers for authorization\n        headers = {\n            \"Authorization\": f\"{self.mosaicml_api_token}\",\n            \"Content-Type\": \"application/json\",\n        }\n        # send request\n        try:\n            response = requests.post(self.endpoint_url, headers=headers, json=payload)\n        except requests.exceptions.RequestException as e:\n            raise ValueError(f\"Error raised by inference endpoint: {e}\")\n        try:\n            parsed_response = response.json()\n            if \"error\" in parsed_response:\n                # if we get rate limited, try sleeping for 1 second\n                if (\n                    not is_retry\n                    and \"rate limit exceeded\" in parsed_response[\"error\"].lower()\n                ):\n                    import time\n                    time.sleep(self.retry_sleep)\n                    return self._call(prompt, stop, run_manager, is_retry=True)\n                raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/mosaicml.html"}1142{"id": "c7eef00c5567-3", "text": "raise ValueError(\n                    f\"Error raised by inference API: {parsed_response['error']}\"\n                )\n            if \"data\" not in parsed_response:\n                raise ValueError(\n                    f\"Error raised by inference API, no key data: {parsed_response}\"\n                )\n            generated_text = parsed_response[\"data\"]\n        except requests.exceptions.JSONDecodeError as e:\n            raise ValueError(\n                f\"Error raised by inference API: {e}.\\nResponse: {response.text}\"\n            )\n        text = generated_text[0][len(prompt) :]\n        # TODO: replace when MosaicML supports custom stop tokens natively\n        if stop is not None:\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/mosaicml.html"}1143{"id": "03856a0d133a-0", "text": "Source code for langchain.llms.huggingface_text_gen_inference\n\"\"\"Wrapper around Huggingface text generation inference API.\"\"\"\nfrom functools import partial\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import Extra, Field, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\n[docs]class HuggingFaceTextGenInference(LLM):\n    \"\"\"\n    HuggingFace text generation inference API.\n    This class is a wrapper around the HuggingFace text generation inference API.\n    It is used to generate text from a given prompt.\n    Attributes:\n    - max_new_tokens: The maximum number of tokens to generate.\n    - top_k: The number of top-k tokens to consider when generating text.\n    - top_p: The cumulative probability threshold for generating text.\n    - typical_p: The typical probability threshold for generating text.\n    - temperature: The temperature to use when generating text.\n    - repetition_penalty: The repetition penalty to use when generating text.\n    - stop_sequences: A list of stop sequences to use when generating text.\n    - seed: The seed to use when generating text.\n    - inference_server_url: The URL of the inference server to use.\n    - timeout: The timeout value in seconds to use while connecting to inference server.\n    - client: The client object used to communicate with the inference server.\n    Methods:\n    - _call: Generates text based on a given prompt and stop sequences.\n    - _llm_type: Returns the type of LLM.\n    \"\"\"\n    \"\"\"\n    Example:\n        .. code-block:: python\n            # Basic Example (no streaming)\n            llm = HuggingFaceTextGenInference(\n                inference_server_url = \"http://localhost:8010/\",", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_text_gen_inference.html"}1144{"id": "03856a0d133a-1", "text": "inference_server_url = \"http://localhost:8010/\",\n                max_new_tokens = 512,\n                top_k = 10,\n                top_p = 0.95,\n                typical_p = 0.95,\n                temperature = 0.01,\n                repetition_penalty = 1.03,\n            )\n            print(llm(\"What is Deep Learning?\"))\n            \n            # Streaming response example\n            from langchain.callbacks import streaming_stdout\n            \n            callbacks = [streaming_stdout.StreamingStdOutCallbackHandler()]\n            llm = HuggingFaceTextGenInference(\n                inference_server_url = \"http://localhost:8010/\",\n                max_new_tokens = 512,\n                top_k = 10,\n                top_p = 0.95,\n                typical_p = 0.95,\n                temperature = 0.01,\n                repetition_penalty = 1.03,\n                callbacks = callbacks,\n                stream = True\n            )\n            print(llm(\"What is Deep Learning?\"))\n            \n    \"\"\"\n    max_new_tokens: int = 512\n    top_k: Optional[int] = None\n    top_p: Optional[float] = 0.95\n    typical_p: Optional[float] = 0.95\n    temperature: float = 0.8\n    repetition_penalty: Optional[float] = None\n    stop_sequences: List[str] = Field(default_factory=list)\n    seed: Optional[int] = None\n    inference_server_url: str = \"\"\n    timeout: int = 120\n    stream: bool = False\n    client: Any\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_text_gen_inference.html"}1145{"id": "03856a0d133a-2", "text": "@root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that python package exists in environment.\"\"\"\n        try:\n            import text_generation\n            values[\"client\"] = text_generation.Client(\n                values[\"inference_server_url\"], timeout=values[\"timeout\"]\n            )\n        except ImportError:\n            raise ImportError(\n                \"Could not import text_generation python package. \"\n                \"Please install it with `pip install text_generation`.\"\n            )\n        return values\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"hf_textgen_inference\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        if stop is None:\n            stop = self.stop_sequences\n        else:\n            stop += self.stop_sequences\n        if not self.stream:\n            res = self.client.generate(\n                prompt,\n                stop_sequences=stop,\n                max_new_tokens=self.max_new_tokens,\n                top_k=self.top_k,\n                top_p=self.top_p,\n                typical_p=self.typical_p,\n                temperature=self.temperature,\n                repetition_penalty=self.repetition_penalty,\n                seed=self.seed,\n            )\n            # remove stop sequences from the end of the generated text\n            for stop_seq in stop:\n                if stop_seq in res.generated_text:\n                    res.generated_text = res.generated_text[\n                        : res.generated_text.index(stop_seq)\n                    ]\n            text = res.generated_text\n        else:\n            text_callback = None\n            if run_manager:\n                text_callback = partial(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_text_gen_inference.html"}1146{"id": "03856a0d133a-3", "text": "text_callback = None\n            if run_manager:\n                text_callback = partial(\n                    run_manager.on_llm_new_token, verbose=self.verbose\n                )\n            params = {\n                \"stop_sequences\": stop,\n                \"max_new_tokens\": self.max_new_tokens,\n                \"top_k\": self.top_k,\n                \"top_p\": self.top_p,\n                \"typical_p\": self.typical_p,\n                \"temperature\": self.temperature,\n                \"repetition_penalty\": self.repetition_penalty,\n                \"seed\": self.seed,\n            }\n            text = \"\"\n            for res in self.client.generate_stream(prompt, **params):\n                token = res.token\n                is_stop = False\n                for stop_seq in stop:\n                    if stop_seq in token.text:\n                        is_stop = True\n                        break\n                if is_stop:\n                    break\n                if not token.special:\n                    if text_callback:\n                        text_callback(token.text)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_text_gen_inference.html"}1147{"id": "d742a1755ffe-0", "text": "Source code for langchain.llms.writer\n\"\"\"Wrapper around Writer APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\n[docs]class Writer(LLM):\n    \"\"\"Wrapper around Writer large language models.\n    To use, you should have the environment variable ``WRITER_API_KEY`` and\n    ``WRITER_ORG_ID`` set with your API key and organization ID respectively.\n    Example:\n        .. code-block:: python\n            from langchain import Writer\n            writer = Writer(model_id=\"palmyra-base\")\n    \"\"\"\n    writer_org_id: Optional[str] = None\n    \"\"\"Writer organization ID.\"\"\"\n    model_id: str = \"palmyra-instruct\"\n    \"\"\"Model name to use.\"\"\"\n    min_tokens: Optional[int] = None\n    \"\"\"Minimum number of tokens to generate.\"\"\"\n    max_tokens: Optional[int] = None\n    \"\"\"Maximum number of tokens to generate.\"\"\"\n    temperature: Optional[float] = None\n    \"\"\"What sampling temperature to use.\"\"\"\n    top_p: Optional[float] = None\n    \"\"\"Total probability mass of tokens to consider at each step.\"\"\"\n    stop: Optional[List[str]] = None\n    \"\"\"Sequences when completion generation will stop.\"\"\"\n    presence_penalty: Optional[float] = None\n    \"\"\"Penalizes repeated tokens regardless of frequency.\"\"\"\n    repetition_penalty: Optional[float] = None\n    \"\"\"Penalizes repeated tokens according to frequency.\"\"\"\n    best_of: Optional[int] = None\n    \"\"\"Generates this many completions server-side and returns the \"best\".\"\"\"\n    logprobs: bool = False", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/writer.html"}1148{"id": "d742a1755ffe-1", "text": "logprobs: bool = False\n    \"\"\"Whether to return log probabilities.\"\"\"\n    n: Optional[int] = None\n    \"\"\"How many completions to generate.\"\"\"\n    writer_api_key: Optional[str] = None\n    \"\"\"Writer API key.\"\"\"\n    base_url: Optional[str] = None\n    \"\"\"Base url to use, if None decides based on model name.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and organization id exist in environment.\"\"\"\n        writer_api_key = get_from_dict_or_env(\n            values, \"writer_api_key\", \"WRITER_API_KEY\"\n        )\n        values[\"writer_api_key\"] = writer_api_key\n        writer_org_id = get_from_dict_or_env(values, \"writer_org_id\", \"WRITER_ORG_ID\")\n        values[\"writer_org_id\"] = writer_org_id\n        return values\n    @property\n    def _default_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the default parameters for calling Writer API.\"\"\"\n        return {\n            \"minTokens\": self.min_tokens,\n            \"maxTokens\": self.max_tokens,\n            \"temperature\": self.temperature,\n            \"topP\": self.top_p,\n            \"stop\": self.stop,\n            \"presencePenalty\": self.presence_penalty,\n            \"repetitionPenalty\": self.repetition_penalty,\n            \"bestOf\": self.best_of,\n            \"logprobs\": self.logprobs,\n            \"n\": self.n,\n        }\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/writer.html"}1149{"id": "d742a1755ffe-2", "text": "\"\"\"Get the identifying parameters.\"\"\"\n        return {\n            **{\"model_id\": self.model_id, \"writer_org_id\": self.writer_org_id},\n            **self._default_params,\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"writer\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to Writer's completions endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = Writer(\"Tell me a joke.\")\n        \"\"\"\n        if self.base_url is not None:\n            base_url = self.base_url\n        else:\n            base_url = (\n                \"https://enterprise-api.writer.com/llm\"\n                f\"/organization/{self.writer_org_id}\"\n                f\"/model/{self.model_id}/completions\"\n            )\n        response = requests.post(\n            url=base_url,\n            headers={\n                \"Authorization\": f\"{self.writer_api_key}\",\n                \"Content-Type\": \"application/json\",\n                \"Accept\": \"application/json\",\n            },\n            json={\"prompt\": prompt, **self._default_params},\n        )\n        text = response.text\n        if stop is not None:\n            # I believe this is required since the stop tokens\n            # are not enforced by the model parameters\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/writer.html"}1150{"id": "d742a1755ffe-3", "text": "text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/writer.html"}1151{"id": "3938af184e57-0", "text": "Source code for langchain.llms.beam\n\"\"\"Wrapper around Beam API.\"\"\"\nimport base64\nimport json\nimport logging\nimport subprocess\nimport textwrap\nimport time\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, Field, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\nDEFAULT_NUM_TRIES = 10\nDEFAULT_SLEEP_TIME = 4\n[docs]class Beam(LLM):\n    \"\"\"Wrapper around Beam API for gpt2 large language model.\n    To use, you should have the ``beam-sdk`` python package installed,\n    and the environment variable ``BEAM_CLIENT_ID`` set with your client id\n    and ``BEAM_CLIENT_SECRET`` set with your client secret. Information on how\n    to get these is available here: https://docs.beam.cloud/account/api-keys.\n    The wrapper can then be called as follows, where the name, cpu, memory, gpu,\n    python version, and python packages can be updated accordingly. Once deployed,\n    the instance can be called.\n        llm = Beam(model_name=\"gpt2\",\n            name=\"langchain-gpt2\",\n            cpu=8,\n            memory=\"32Gi\",\n            gpu=\"A10G\",\n            python_version=\"python3.8\",\n            python_packages=[\n                \"diffusers[torch]>=0.10\",\n                \"transformers\",\n                \"torch\",\n                \"pillow\",\n                \"accelerate\",\n                \"safetensors\",\n                \"xformers\",],\n            max_length=50)\n        llm._deploy()\n        call_result = llm._call(input)\n    \"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1152{"id": "3938af184e57-1", "text": "llm._deploy()\n        call_result = llm._call(input)\n    \"\"\"\n    model_name: str = \"\"\n    name: str = \"\"\n    cpu: str = \"\"\n    memory: str = \"\"\n    gpu: str = \"\"\n    python_version: str = \"\"\n    python_packages: List[str] = []\n    max_length: str = \"\"\n    url: str = \"\"\n    \"\"\"model endpoint to use\"\"\"\n    model_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Holds any model parameters valid for `create` call not\n    explicitly specified.\"\"\"\n    beam_client_id: str = \"\"\n    beam_client_secret: str = \"\"\n    app_id: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic config.\"\"\"\n        extra = Extra.forbid\n    @root_validator(pre=True)\n    def build_extra(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Build extra kwargs from additional params that were passed in.\"\"\"\n        all_required_field_names = {field.alias for field in cls.__fields__.values()}\n        extra = values.get(\"model_kwargs\", {})\n        for field_name in list(values):\n            if field_name not in all_required_field_names:\n                if field_name in extra:\n                    raise ValueError(f\"Found {field_name} supplied twice.\")\n                logger.warning(\n                    f\"\"\"{field_name} was transfered to model_kwargs.\n                    Please confirm that {field_name} is what you intended.\"\"\"\n                )\n                extra[field_name] = values.pop(field_name)\n        values[\"model_kwargs\"] = extra\n        return values\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1153{"id": "3938af184e57-2", "text": "\"\"\"Validate that api key and python package exists in environment.\"\"\"\n        beam_client_id = get_from_dict_or_env(\n            values, \"beam_client_id\", \"BEAM_CLIENT_ID\"\n        )\n        beam_client_secret = get_from_dict_or_env(\n            values, \"beam_client_secret\", \"BEAM_CLIENT_SECRET\"\n        )\n        values[\"beam_client_id\"] = beam_client_id\n        values[\"beam_client_secret\"] = beam_client_secret\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            \"model_name\": self.model_name,\n            \"name\": self.name,\n            \"cpu\": self.cpu,\n            \"memory\": self.memory,\n            \"gpu\": self.gpu,\n            \"python_version\": self.python_version,\n            \"python_packages\": self.python_packages,\n            \"max_length\": self.max_length,\n            \"model_kwargs\": self.model_kwargs,\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"beam\"\n[docs]    def app_creation(self) -> None:\n        \"\"\"Creates a Python file which will contain your Beam app definition.\"\"\"\n        script = textwrap.dedent(\n            \"\"\"\\\n        import beam\n        # The environment your code will run on\n        app = beam.App(\n            name=\"{name}\",\n            cpu={cpu},\n            memory=\"{memory}\",\n            gpu=\"{gpu}\",\n            python_version=\"{python_version}\",\n            python_packages={python_packages},\n        )\n        app.Trigger.RestAPI(\n            inputs={{\"prompt\": beam.Types.String(), \"max_length\": beam.Types.String()}},\n            outputs={{\"text\": beam.Types.String()}},", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1154{"id": "3938af184e57-3", "text": "outputs={{\"text\": beam.Types.String()}},\n            handler=\"run.py:beam_langchain\",\n        )\n        \"\"\"\n        )\n        script_name = \"app.py\"\n        with open(script_name, \"w\") as file:\n            file.write(\n                script.format(\n                    name=self.name,\n                    cpu=self.cpu,\n                    memory=self.memory,\n                    gpu=self.gpu,\n                    python_version=self.python_version,\n                    python_packages=self.python_packages,\n                )\n            )\n[docs]    def run_creation(self) -> None:\n        \"\"\"Creates a Python file which will be deployed on beam.\"\"\"\n        script = textwrap.dedent(\n            \"\"\"\n        import os\n        import transformers\n        from transformers import GPT2LMHeadModel, GPT2Tokenizer\n        model_name = \"{model_name}\"\n        def beam_langchain(**inputs):\n            prompt = inputs[\"prompt\"]\n            length = inputs[\"max_length\"]\n            tokenizer = GPT2Tokenizer.from_pretrained(model_name)\n            model = GPT2LMHeadModel.from_pretrained(model_name)\n            encodedPrompt = tokenizer.encode(prompt, return_tensors='pt')\n            outputs = model.generate(encodedPrompt, max_length=int(length),\n              do_sample=True, pad_token_id=tokenizer.eos_token_id)\n            output = tokenizer.decode(outputs[0], skip_special_tokens=True)\n            print(output)\n            return {{\"text\": output}}\n        \"\"\"\n        )\n        script_name = \"run.py\"\n        with open(script_name, \"w\") as file:\n            file.write(script.format(model_name=self.model_name))\n    def _deploy(self) -> str:\n        \"\"\"Call to Beam.\"\"\"\n        try:\n            import beam  # type: ignore\n            if beam.__path__ == \"\":\n                raise ImportError\n        except ImportError:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1155{"id": "3938af184e57-4", "text": "if beam.__path__ == \"\":\n                raise ImportError\n        except ImportError:\n            raise ImportError(\n                \"Could not import beam python package. \"\n                \"Please install it with `curl \"\n                \"https://raw.githubusercontent.com/slai-labs\"\n                \"/get-beam/main/get-beam.sh -sSfL | sh`.\"\n            )\n        self.app_creation()\n        self.run_creation()\n        process = subprocess.run(\n            \"beam deploy app.py\", shell=True, capture_output=True, text=True\n        )\n        if process.returncode == 0:\n            output = process.stdout\n            logger.info(output)\n            lines = output.split(\"\\n\")\n            for line in lines:\n                if line.startswith(\" i  Send requests to: https://apps.beam.cloud/\"):\n                    self.app_id = line.split(\"/\")[-1]\n                    self.url = line.split(\":\")[1].strip()\n                    return self.app_id\n            raise ValueError(\n                f\"\"\"Failed to retrieve the appID from the deployment output.\n                Deployment output: {output}\"\"\"\n            )\n        else:\n            raise ValueError(f\"Deployment failed. Error: {process.stderr}\")\n    @property\n    def authorization(self) -> str:\n        if self.beam_client_id:\n            credential_str = self.beam_client_id + \":\" + self.beam_client_secret\n        else:\n            credential_str = self.beam_client_secret\n        return base64.b64encode(credential_str.encode()).decode()\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[list] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call to Beam.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1156{"id": "3938af184e57-5", "text": ") -> str:\n        \"\"\"Call to Beam.\"\"\"\n        url = \"https://apps.beam.cloud/\" + self.app_id if self.app_id else self.url\n        payload = {\"prompt\": prompt, \"max_length\": self.max_length}\n        headers = {\n            \"Accept\": \"*/*\",\n            \"Accept-Encoding\": \"gzip, deflate\",\n            \"Authorization\": \"Basic \" + self.authorization,\n            \"Connection\": \"keep-alive\",\n            \"Content-Type\": \"application/json\",\n        }\n        for _ in range(DEFAULT_NUM_TRIES):\n            request = requests.post(url, headers=headers, data=json.dumps(payload))\n            if request.status_code == 200:\n                return request.json()[\"text\"]\n            time.sleep(DEFAULT_SLEEP_TIME)\n        logger.warning(\"Unable to successfully call model.\")\n        return \"\"\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/beam.html"}1157{"id": "d2d82e8e3bb0-0", "text": "Source code for langchain.llms.gpt4all\n\"\"\"Wrapper for the GPT4All model.\"\"\"\nfrom functools import partial\nfrom typing import Any, Dict, List, Mapping, Optional, Set\nfrom pydantic import Extra, Field, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\n[docs]class GPT4All(LLM):\n    r\"\"\"Wrapper around GPT4All language models.\n    To use, you should have the ``gpt4all`` python package installed, the\n    pre-trained model file, and the model's config information.\n    Example:\n        .. code-block:: python\n            from langchain.llms import GPT4All\n            model = GPT4All(model=\"./models/gpt4all-model.bin\", n_ctx=512, n_threads=8)\n            # Simplest invocation\n            response = model(\"Once upon a time, \")\n    \"\"\"\n    model: str\n    \"\"\"Path to the pre-trained GPT4All model file.\"\"\"\n    backend: Optional[str] = Field(None, alias=\"backend\")\n    n_ctx: int = Field(512, alias=\"n_ctx\")\n    \"\"\"Token context window.\"\"\"\n    n_parts: int = Field(-1, alias=\"n_parts\")\n    \"\"\"Number of parts to split the model into. \n    If -1, the number of parts is automatically determined.\"\"\"\n    seed: int = Field(0, alias=\"seed\")\n    \"\"\"Seed. If -1, a random seed is used.\"\"\"\n    f16_kv: bool = Field(False, alias=\"f16_kv\")\n    \"\"\"Use half-precision for key/value cache.\"\"\"\n    logits_all: bool = Field(False, alias=\"logits_all\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/gpt4all.html"}1158{"id": "d2d82e8e3bb0-1", "text": "logits_all: bool = Field(False, alias=\"logits_all\")\n    \"\"\"Return logits for all tokens, not just the last token.\"\"\"\n    vocab_only: bool = Field(False, alias=\"vocab_only\")\n    \"\"\"Only load the vocabulary, no weights.\"\"\"\n    use_mlock: bool = Field(False, alias=\"use_mlock\")\n    \"\"\"Force system to keep model in RAM.\"\"\"\n    embedding: bool = Field(False, alias=\"embedding\")\n    \"\"\"Use embedding mode only.\"\"\"\n    n_threads: Optional[int] = Field(4, alias=\"n_threads\")\n    \"\"\"Number of threads to use.\"\"\"\n    n_predict: Optional[int] = 256\n    \"\"\"The maximum number of tokens to generate.\"\"\"\n    temp: Optional[float] = 0.8\n    \"\"\"The temperature to use for sampling.\"\"\"\n    top_p: Optional[float] = 0.95\n    \"\"\"The top-p value to use for sampling.\"\"\"\n    top_k: Optional[int] = 40\n    \"\"\"The top-k value to use for sampling.\"\"\"\n    echo: Optional[bool] = False\n    \"\"\"Whether to echo the prompt.\"\"\"\n    stop: Optional[List[str]] = []\n    \"\"\"A list of strings to stop generation when encountered.\"\"\"\n    repeat_last_n: Optional[int] = 64\n    \"Last n tokens to penalize\"\n    repeat_penalty: Optional[float] = 1.3\n    \"\"\"The penalty to apply to repeated tokens.\"\"\"\n    n_batch: int = Field(1, alias=\"n_batch\")\n    \"\"\"Batch size for prompt processing.\"\"\"\n    streaming: bool = False\n    \"\"\"Whether to stream the results or not.\"\"\"\n    context_erase: float = 0.5\n    \"\"\"Leave (n_ctx * context_erase) tokens\n    starting from beginning if the context has run out.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/gpt4all.html"}1159{"id": "d2d82e8e3bb0-2", "text": "starting from beginning if the context has run out.\"\"\"\n    client: Any = None  #: :meta private:\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @staticmethod\n    def _model_param_names() -> Set[str]:\n        return {\n            \"n_ctx\",\n            \"n_predict\",\n            \"top_k\",\n            \"top_p\",\n            \"temp\",\n            \"n_batch\",\n            \"repeat_penalty\",\n            \"repeat_last_n\",\n            \"context_erase\",\n        }\n    def _default_params(self) -> Dict[str, Any]:\n        return {\n            \"n_ctx\": self.n_ctx,\n            \"n_predict\": self.n_predict,\n            \"top_k\": self.top_k,\n            \"top_p\": self.top_p,\n            \"temp\": self.temp,\n            \"n_batch\": self.n_batch,\n            \"repeat_penalty\": self.repeat_penalty,\n            \"repeat_last_n\": self.repeat_last_n,\n            \"context_erase\": self.context_erase,\n        }\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that the python package exists in the environment.\"\"\"\n        try:\n            from gpt4all import GPT4All as GPT4AllModel\n            full_path = values[\"model\"]\n            model_path, delimiter, model_name = full_path.rpartition(\"/\")\n            model_path += delimiter\n            values[\"client\"] = GPT4AllModel(\n                model_name=model_name,\n                model_path=model_path or None,\n                model_type=values[\"backend\"],\n                allow_download=False,\n            )\n            values[\"backend\"] = values[\"client\"].model.model_type\n        except ImportError:\n            raise ValueError(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/gpt4all.html"}1160{"id": "d2d82e8e3bb0-3", "text": "except ImportError:\n            raise ValueError(\n                \"Could not import gpt4all python package. \"\n                \"Please install it with `pip install gpt4all`.\"\n            )\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            \"model\": self.model,\n            **self._default_params(),\n            **{\n                k: v for k, v in self.__dict__.items() if k in self._model_param_names()\n            },\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return the type of llm.\"\"\"\n        return \"gpt4all\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        r\"\"\"Call out to GPT4All's generate method.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: A list of strings to stop generation when encountered.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                prompt = \"Once upon a time, \"\n                response = model(prompt, n_predict=55)\n        \"\"\"\n        text_callback = None\n        if run_manager:\n            text_callback = partial(run_manager.on_llm_new_token, verbose=self.verbose)\n        text = \"\"\n        for token in self.client.generate(prompt, **self._default_params()):\n            if text_callback:\n                text_callback(token)\n            text += token\n        if stop is not None:\n            text = enforce_stop_tokens(text, stop)\n        return text", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/gpt4all.html"}1161{"id": "d2d82e8e3bb0-4", "text": "text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/gpt4all.html"}1162{"id": "48d1c47dda07-0", "text": "Source code for langchain.llms.forefrontai\n\"\"\"Wrapper around ForefrontAI APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\n[docs]class ForefrontAI(LLM):\n    \"\"\"Wrapper around ForefrontAI large language models.\n    To use, you should have the environment variable ``FOREFRONTAI_API_KEY``\n    set with your API key.\n    Example:\n        .. code-block:: python\n            from langchain.llms import ForefrontAI\n            forefrontai = ForefrontAI(endpoint_url=\"\")\n    \"\"\"\n    endpoint_url: str = \"\"\n    \"\"\"Model name to use.\"\"\"\n    temperature: float = 0.7\n    \"\"\"What sampling temperature to use.\"\"\"\n    length: int = 256\n    \"\"\"The maximum number of tokens to generate in the completion.\"\"\"\n    top_p: float = 1.0\n    \"\"\"Total probability mass of tokens to consider at each step.\"\"\"\n    top_k: int = 40\n    \"\"\"The number of highest probability vocabulary tokens to\n    keep for top-k-filtering.\"\"\"\n    repetition_penalty: int = 1\n    \"\"\"Penalizes repeated tokens according to frequency.\"\"\"\n    forefrontai_api_key: Optional[str] = None\n    base_url: Optional[str] = None\n    \"\"\"Base url to use, if None decides based on model name.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/forefrontai.html"}1163{"id": "48d1c47dda07-1", "text": "@root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key exists in environment.\"\"\"\n        forefrontai_api_key = get_from_dict_or_env(\n            values, \"forefrontai_api_key\", \"FOREFRONTAI_API_KEY\"\n        )\n        values[\"forefrontai_api_key\"] = forefrontai_api_key\n        return values\n    @property\n    def _default_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the default parameters for calling ForefrontAI API.\"\"\"\n        return {\n            \"temperature\": self.temperature,\n            \"length\": self.length,\n            \"top_p\": self.top_p,\n            \"top_k\": self.top_k,\n            \"repetition_penalty\": self.repetition_penalty,\n        }\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {**{\"endpoint_url\": self.endpoint_url}, **self._default_params}\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"forefrontai\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to ForefrontAI's complete endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = ForefrontAI(\"Tell me a joke.\")\n        \"\"\"\n        response = requests.post(\n            url=self.endpoint_url,", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/forefrontai.html"}1164{"id": "48d1c47dda07-2", "text": "\"\"\"\n        response = requests.post(\n            url=self.endpoint_url,\n            headers={\n                \"Authorization\": f\"Bearer {self.forefrontai_api_key}\",\n                \"Content-Type\": \"application/json\",\n            },\n            json={\"text\": prompt, **self._default_params},\n        )\n        response_json = response.json()\n        text = response_json[\"result\"][0][\"completion\"]\n        if stop is not None:\n            # I believe this is required since the stop tokens\n            # are not enforced by the model parameters\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/forefrontai.html"}1165{"id": "80df87147829-0", "text": "Source code for langchain.llms.sagemaker_endpoint\n\"\"\"Wrapper around Sagemaker InvokeEndpoint API.\"\"\"\nfrom abc import abstractmethod\nfrom typing import Any, Dict, Generic, List, Mapping, Optional, TypeVar, Union\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nINPUT_TYPE = TypeVar(\"INPUT_TYPE\", bound=Union[str, List[str]])\nOUTPUT_TYPE = TypeVar(\"OUTPUT_TYPE\", bound=Union[str, List[List[float]]])\nclass ContentHandlerBase(Generic[INPUT_TYPE, OUTPUT_TYPE]):\n    \"\"\"A handler class to transform input from LLM to a\n    format that SageMaker endpoint expects. Similarily,\n    the class also handles transforming output from the\n    SageMaker endpoint to a format that LLM class expects.\n    \"\"\"\n    \"\"\"\n    Example:\n        .. code-block:: python\n            class ContentHandler(ContentHandlerBase):\n                content_type = \"application/json\"\n                accepts = \"application/json\"\n                def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n                    input_str = json.dumps({prompt: prompt, **model_kwargs})\n                    return input_str.encode('utf-8')\n                \n                def transform_output(self, output: bytes) -> str:\n                    response_json = json.loads(output.read().decode(\"utf-8\"))\n                    return response_json[0][\"generated_text\"]\n    \"\"\"\n    content_type: Optional[str] = \"text/plain\"\n    \"\"\"The MIME type of the input data passed to endpoint\"\"\"\n    accepts: Optional[str] = \"text/plain\"\n    \"\"\"The MIME type of the response data returned from endpoint\"\"\"\n    @abstractmethod", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1166{"id": "80df87147829-1", "text": "\"\"\"The MIME type of the response data returned from endpoint\"\"\"\n    @abstractmethod\n    def transform_input(self, prompt: INPUT_TYPE, model_kwargs: Dict) -> bytes:\n        \"\"\"Transforms the input to a format that model can accept\n        as the request Body. Should return bytes or seekable file\n        like object in the format specified in the content_type\n        request header.\n        \"\"\"\n    @abstractmethod\n    def transform_output(self, output: bytes) -> OUTPUT_TYPE:\n        \"\"\"Transforms the output from the model to string that\n        the LLM class expects.\n        \"\"\"\nclass LLMContentHandler(ContentHandlerBase[str, str]):\n    \"\"\"Content handler for LLM class.\"\"\"\n[docs]class SagemakerEndpoint(LLM):\n    \"\"\"Wrapper around custom Sagemaker Inference Endpoints.\n    To use, you must supply the endpoint name from your deployed\n    Sagemaker model & the region where it is deployed.\n    To authenticate, the AWS client uses the following methods to\n    automatically load credentials:\n    https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html\n    If a specific credential profile should be used, you must pass\n    the name of the profile from the ~/.aws/credentials file that is to be used.\n    Make sure the credentials / roles used have the required policies to\n    access the Sagemaker endpoint.\n    See: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html\n    \"\"\"\n    \"\"\"\n    Example:\n        .. code-block:: python\n            from langchain import SagemakerEndpoint\n            endpoint_name = (\n                \"my-endpoint-name\"\n            )\n            region_name = (\n                \"us-west-2\"\n            )\n            credentials_profile_name = (\n                \"default\"\n            )", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1167{"id": "80df87147829-2", "text": ")\n            credentials_profile_name = (\n                \"default\"\n            )\n            se = SagemakerEndpoint(\n                endpoint_name=endpoint_name,\n                region_name=region_name,\n                credentials_profile_name=credentials_profile_name\n            )\n    \"\"\"\n    client: Any  #: :meta private:\n    endpoint_name: str = \"\"\n    \"\"\"The name of the endpoint from the deployed Sagemaker model.\n    Must be unique within an AWS Region.\"\"\"\n    region_name: str = \"\"\n    \"\"\"The aws region where the Sagemaker model is deployed, eg. `us-west-2`.\"\"\"\n    credentials_profile_name: Optional[str] = None\n    \"\"\"The name of the profile in the ~/.aws/credentials or ~/.aws/config files, which\n    has either access keys or role information specified.\n    If not specified, the default credential profile or, if on an EC2 instance,\n    credentials from IMDS will be used.\n    See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html\n    \"\"\"\n    content_handler: LLMContentHandler\n    \"\"\"The content handler class that provides an input and\n    output transform functions to handle formats between LLM\n    and the endpoint.\n    \"\"\"\n    \"\"\"\n     Example:\n        .. code-block:: python\n        from langchain.llms.sagemaker_endpoint import LLMContentHandler\n        class ContentHandler(LLMContentHandler):\n                content_type = \"application/json\"\n                accepts = \"application/json\"\n                def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n                    input_str = json.dumps({prompt: prompt, **model_kwargs})\n                    return input_str.encode('utf-8')\n                \n                def transform_output(self, output: bytes) -> str:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1168{"id": "80df87147829-3", "text": "def transform_output(self, output: bytes) -> str:\n                    response_json = json.loads(output.read().decode(\"utf-8\"))\n                    return response_json[0][\"generated_text\"]\n    \"\"\"\n    model_kwargs: Optional[Dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    endpoint_kwargs: Optional[Dict] = None\n    \"\"\"Optional attributes passed to the invoke_endpoint\n    function. See `boto3`_. docs for more info.\n    .. _boto3: <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>\n    \"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that AWS credentials to and python package exists in environment.\"\"\"\n        try:\n            import boto3\n            try:\n                if values[\"credentials_profile_name\"] is not None:\n                    session = boto3.Session(\n                        profile_name=values[\"credentials_profile_name\"]\n                    )\n                else:\n                    # use default credentials\n                    session = boto3.Session()\n                values[\"client\"] = session.client(\n                    \"sagemaker-runtime\", region_name=values[\"region_name\"]\n                )\n            except Exception as e:\n                raise ValueError(\n                    \"Could not load credentials to authenticate with AWS client. \"\n                    \"Please check that credentials in the specified \"\n                    \"profile name are valid.\"\n                ) from e\n        except ImportError:\n            raise ImportError(\n                \"Could not import boto3 python package. \"\n                \"Please install it with `pip install boto3`.\"\n            )\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1169{"id": "80df87147829-4", "text": "@property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        return {\n            **{\"endpoint_name\": self.endpoint_name},\n            **{\"model_kwargs\": _model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"sagemaker_endpoint\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to Sagemaker inference endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = se(\"Tell me a joke.\")\n        \"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        _endpoint_kwargs = self.endpoint_kwargs or {}\n        body = self.content_handler.transform_input(prompt, _model_kwargs)\n        content_type = self.content_handler.content_type\n        accepts = self.content_handler.accepts\n        # send request\n        try:\n            response = self.client.invoke_endpoint(\n                EndpointName=self.endpoint_name,\n                Body=body,\n                ContentType=content_type,\n                Accept=accepts,\n                **_endpoint_kwargs,\n            )\n        except Exception as e:\n            raise ValueError(f\"Error raised by inference endpoint: {e}\")\n        text = self.content_handler.transform_output(response[\"Body\"])\n        if stop is not None:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1170{"id": "80df87147829-5", "text": "text = self.content_handler.transform_output(response[\"Body\"])\n        if stop is not None:\n            # This is a bit hacky, but I can't figure out a better way to enforce\n            # stop tokens when making calls to the sagemaker endpoint.\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/sagemaker_endpoint.html"}1171{"id": "b6ca1e84007f-0", "text": "Source code for langchain.llms.huggingface_pipeline\n\"\"\"Wrapper around HuggingFace Pipeline APIs.\"\"\"\nimport importlib.util\nimport logging\nfrom typing import Any, List, Mapping, Optional\nfrom pydantic import Extra\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nDEFAULT_MODEL_ID = \"gpt2\"\nDEFAULT_TASK = \"text-generation\"\nVALID_TASKS = (\"text2text-generation\", \"text-generation\", \"summarization\")\nlogger = logging.getLogger(__name__)\n[docs]class HuggingFacePipeline(LLM):\n    \"\"\"Wrapper around HuggingFace Pipeline API.\n    To use, you should have the ``transformers`` python package installed.\n    Only supports `text-generation`, `text2text-generation` and `summarization` for now.\n    Example using from_model_id:\n        .. code-block:: python\n            from langchain.llms import HuggingFacePipeline\n            hf = HuggingFacePipeline.from_model_id(\n                model_id=\"gpt2\",\n                task=\"text-generation\",\n                pipeline_kwargs={\"max_new_tokens\": 10},\n            )\n    Example passing pipeline in directly:\n        .. code-block:: python\n            from langchain.llms import HuggingFacePipeline\n            from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n            model_id = \"gpt2\"\n            tokenizer = AutoTokenizer.from_pretrained(model_id)\n            model = AutoModelForCausalLM.from_pretrained(model_id)\n            pipe = pipeline(\n                \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=10\n            )\n            hf = HuggingFacePipeline(pipeline=pipe)\n    \"\"\"\n    pipeline: Any  #: :meta private:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_pipeline.html"}1172{"id": "b6ca1e84007f-1", "text": "\"\"\"\n    pipeline: Any  #: :meta private:\n    model_id: str = DEFAULT_MODEL_ID\n    \"\"\"Model name to use.\"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments passed to the model.\"\"\"\n    pipeline_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments passed to the pipeline.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n[docs]    @classmethod\n    def from_model_id(\n        cls,\n        model_id: str,\n        task: str,\n        device: int = -1,\n        model_kwargs: Optional[dict] = None,\n        pipeline_kwargs: Optional[dict] = None,\n        **kwargs: Any,\n    ) -> LLM:\n        \"\"\"Construct the pipeline object from model_id and task.\"\"\"\n        try:\n            from transformers import (\n                AutoModelForCausalLM,\n                AutoModelForSeq2SeqLM,\n                AutoTokenizer,\n            )\n            from transformers import pipeline as hf_pipeline\n        except ImportError:\n            raise ValueError(\n                \"Could not import transformers python package. \"\n                \"Please install it with `pip install transformers`.\"\n            )\n        _model_kwargs = model_kwargs or {}\n        tokenizer = AutoTokenizer.from_pretrained(model_id, **_model_kwargs)\n        try:\n            if task == \"text-generation\":\n                model = AutoModelForCausalLM.from_pretrained(model_id, **_model_kwargs)\n            elif task in (\"text2text-generation\", \"summarization\"):\n                model = AutoModelForSeq2SeqLM.from_pretrained(model_id, **_model_kwargs)\n            else:\n                raise ValueError(\n                    f\"Got invalid task {task}, \"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_pipeline.html"}1173{"id": "b6ca1e84007f-2", "text": "else:\n                raise ValueError(\n                    f\"Got invalid task {task}, \"\n                    f\"currently only {VALID_TASKS} are supported\"\n                )\n        except ImportError as e:\n            raise ValueError(\n                f\"Could not load the {task} model due to missing dependencies.\"\n            ) from e\n        if importlib.util.find_spec(\"torch\") is not None:\n            import torch\n            cuda_device_count = torch.cuda.device_count()\n            if device < -1 or (device >= cuda_device_count):\n                raise ValueError(\n                    f\"Got device=={device}, \"\n                    f\"device is required to be within [-1, {cuda_device_count})\"\n                )\n            if device < 0 and cuda_device_count > 0:\n                logger.warning(\n                    \"Device has %d GPUs available. \"\n                    \"Provide device={deviceId} to `from_model_id` to use available\"\n                    \"GPUs for execution. deviceId is -1 (default) for CPU and \"\n                    \"can be a positive integer associated with CUDA device id.\",\n                    cuda_device_count,\n                )\n        if \"trust_remote_code\" in _model_kwargs:\n            _model_kwargs = {\n                k: v for k, v in _model_kwargs.items() if k != \"trust_remote_code\"\n            }\n        _pipeline_kwargs = pipeline_kwargs or {}\n        pipeline = hf_pipeline(\n            task=task,\n            model=model,\n            tokenizer=tokenizer,\n            device=device,\n            model_kwargs=_model_kwargs,\n            **_pipeline_kwargs,\n        )\n        if pipeline.task not in VALID_TASKS:\n            raise ValueError(\n                f\"Got invalid task {pipeline.task}, \"\n                f\"currently only {VALID_TASKS} are supported\"\n            )\n        return cls(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_pipeline.html"}1174{"id": "b6ca1e84007f-3", "text": ")\n        return cls(\n            pipeline=pipeline,\n            model_id=model_id,\n            model_kwargs=_model_kwargs,\n            pipeline_kwargs=_pipeline_kwargs,\n            **kwargs,\n        )\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            \"model_id\": self.model_id,\n            \"model_kwargs\": self.model_kwargs,\n            \"pipeline_kwargs\": self.pipeline_kwargs,\n        }\n    @property\n    def _llm_type(self) -> str:\n        return \"huggingface_pipeline\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        response = self.pipeline(prompt)\n        if self.pipeline.task == \"text-generation\":\n            # Text generation return includes the starter text.\n            text = response[0][\"generated_text\"][len(prompt) :]\n        elif self.pipeline.task == \"text2text-generation\":\n            text = response[0][\"generated_text\"]\n        elif self.pipeline.task == \"summarization\":\n            text = response[0][\"summary_text\"]\n        else:\n            raise ValueError(\n                f\"Got invalid task {self.pipeline.task}, \"\n                f\"currently only {VALID_TASKS} are supported\"\n            )\n        if stop is not None:\n            # This is a bit hacky, but I can't figure out a better way to enforce\n            # stop tokens when making calls to huggingface_hub.\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_pipeline.html"}1175{"id": "b6ca1e84007f-4", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/huggingface_pipeline.html"}1176{"id": "26924447613a-0", "text": "Source code for langchain.llms.anyscale\n\"\"\"Wrapper around Anyscale\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\n[docs]class Anyscale(LLM):\n    \"\"\"Wrapper around Anyscale Services.\n    To use, you should have the environment variable ``ANYSCALE_SERVICE_URL``,\n    ``ANYSCALE_SERVICE_ROUTE`` and ``ANYSCALE_SERVICE_TOKEN`` set with your Anyscale\n    Service, or pass it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.llms import Anyscale\n            anyscale = Anyscale(anyscale_service_url=\"SERVICE_URL\",\n                                anyscale_service_route=\"SERVICE_ROUTE\",\n                                anyscale_service_token=\"SERVICE_TOKEN\")\n            # Use Ray for distributed processing\n            import ray\n            prompt_list=[]\n            @ray.remote\n            def send_query(llm, prompt):\n                resp = llm(prompt)\n                return resp\n            futures = [send_query.remote(anyscale, prompt) for prompt in prompt_list]\n            results = ray.get(futures)\n    \"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model. Reserved for future use\"\"\"\n    anyscale_service_url: Optional[str] = None\n    anyscale_service_route: Optional[str] = None\n    anyscale_service_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/anyscale.html"}1177{"id": "26924447613a-1", "text": "@root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        anyscale_service_url = get_from_dict_or_env(\n            values, \"anyscale_service_url\", \"ANYSCALE_SERVICE_URL\"\n        )\n        anyscale_service_route = get_from_dict_or_env(\n            values, \"anyscale_service_route\", \"ANYSCALE_SERVICE_ROUTE\"\n        )\n        anyscale_service_token = get_from_dict_or_env(\n            values, \"anyscale_service_token\", \"ANYSCALE_SERVICE_TOKEN\"\n        )\n        try:\n            anyscale_service_endpoint = f\"{anyscale_service_url}/-/route\"\n            headers = {\"Authorization\": f\"Bearer {anyscale_service_token}\"}\n            requests.get(anyscale_service_endpoint, headers=headers)\n        except requests.exceptions.RequestException as e:\n            raise ValueError(e)\n        values[\"anyscale_service_url\"] = anyscale_service_url\n        values[\"anyscale_service_route\"] = anyscale_service_route\n        values[\"anyscale_service_token\"] = anyscale_service_token\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            \"anyscale_service_url\": self.anyscale_service_url,\n            \"anyscale_service_route\": self.anyscale_service_route,\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"anyscale\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to Anyscale Service endpoint.\n        Args:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/anyscale.html"}1178{"id": "26924447613a-2", "text": ") -> str:\n        \"\"\"Call out to Anyscale Service endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = anyscale(\"Tell me a joke.\")\n        \"\"\"\n        anyscale_service_endpoint = (\n            f\"{self.anyscale_service_url}/{self.anyscale_service_route}\"\n        )\n        headers = {\"Authorization\": f\"Bearer {self.anyscale_service_token}\"}\n        body = {\"prompt\": prompt}\n        resp = requests.post(anyscale_service_endpoint, headers=headers, json=body)\n        if resp.status_code != 200:\n            raise ValueError(\n                f\"Error returned by service, status code {resp.status_code}\"\n            )\n        text = resp.text\n        if stop is not None:\n            # This is a bit hacky, but I can't figure out a better way to enforce\n            # stop tokens when making calls to huggingface_hub.\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/anyscale.html"}1179{"id": "7730cb73735c-0", "text": "Source code for langchain.llms.stochasticai\n\"\"\"Wrapper around StochasticAI APIs.\"\"\"\nimport logging\nimport time\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, Field, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class StochasticAI(LLM):\n    \"\"\"Wrapper around StochasticAI large language models.\n    To use, you should have the environment variable ``STOCHASTICAI_API_KEY``\n    set with your API key.\n    Example:\n        .. code-block:: python\n            from langchain.llms import StochasticAI\n            stochasticai = StochasticAI(api_url=\"\")\n    \"\"\"\n    api_url: str = \"\"\n    \"\"\"Model name to use.\"\"\"\n    model_kwargs: Dict[str, Any] = Field(default_factory=dict)\n    \"\"\"Holds any model parameters valid for `create` call not\n    explicitly specified.\"\"\"\n    stochasticai_api_key: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator(pre=True)\n    def build_extra(cls, values: Dict[str, Any]) -> Dict[str, Any]:\n        \"\"\"Build extra kwargs from additional params that were passed in.\"\"\"\n        all_required_field_names = {field.alias for field in cls.__fields__.values()}\n        extra = values.get(\"model_kwargs\", {})\n        for field_name in list(values):\n            if field_name not in all_required_field_names:\n                if field_name in extra:\n                    raise ValueError(f\"Found {field_name} supplied twice.\")", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/stochasticai.html"}1180{"id": "7730cb73735c-1", "text": "raise ValueError(f\"Found {field_name} supplied twice.\")\n                logger.warning(\n                    f\"\"\"{field_name} was transfered to model_kwargs.\n                    Please confirm that {field_name} is what you intended.\"\"\"\n                )\n                extra[field_name] = values.pop(field_name)\n        values[\"model_kwargs\"] = extra\n        return values\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key exists in environment.\"\"\"\n        stochasticai_api_key = get_from_dict_or_env(\n            values, \"stochasticai_api_key\", \"STOCHASTICAI_API_KEY\"\n        )\n        values[\"stochasticai_api_key\"] = stochasticai_api_key\n        return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            **{\"endpoint_url\": self.api_url},\n            **{\"model_kwargs\": self.model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"stochasticai\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to StochasticAI's complete endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = StochasticAI(\"Tell me a joke.\")\n        \"\"\"\n        params = self.model_kwargs or {}", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/stochasticai.html"}1181{"id": "7730cb73735c-2", "text": "\"\"\"\n        params = self.model_kwargs or {}\n        response_post = requests.post(\n            url=self.api_url,\n            json={\"prompt\": prompt, \"params\": params},\n            headers={\n                \"apiKey\": f\"{self.stochasticai_api_key}\",\n                \"Accept\": \"application/json\",\n                \"Content-Type\": \"application/json\",\n            },\n        )\n        response_post.raise_for_status()\n        response_post_json = response_post.json()\n        completed = False\n        while not completed:\n            response_get = requests.get(\n                url=response_post_json[\"data\"][\"responseUrl\"],\n                headers={\n                    \"apiKey\": f\"{self.stochasticai_api_key}\",\n                    \"Accept\": \"application/json\",\n                    \"Content-Type\": \"application/json\",\n                },\n            )\n            response_get.raise_for_status()\n            response_get_json = response_get.json()[\"data\"]\n            text = response_get_json.get(\"completion\")\n            completed = text is not None\n            time.sleep(0.5)\n        text = text[0]\n        if stop is not None:\n            # I believe this is required since the stop tokens\n            # are not enforced by the model parameters\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/stochasticai.html"}1182{"id": "538c83387d80-0", "text": "Source code for langchain.llms.self_hosted_hugging_face\n\"\"\"Wrapper around HuggingFace Pipeline API to run on self-hosted remote hardware.\"\"\"\nimport importlib.util\nimport logging\nfrom typing import Any, Callable, List, Mapping, Optional\nfrom pydantic import Extra\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.self_hosted import SelfHostedPipeline\nfrom langchain.llms.utils import enforce_stop_tokens\nDEFAULT_MODEL_ID = \"gpt2\"\nDEFAULT_TASK = \"text-generation\"\nVALID_TASKS = (\"text2text-generation\", \"text-generation\", \"summarization\")\nlogger = logging.getLogger(__name__)\ndef _generate_text(\n    pipeline: Any,\n    prompt: str,\n    *args: Any,\n    stop: Optional[List[str]] = None,\n    **kwargs: Any,\n) -> str:\n    \"\"\"Inference function to send to the remote hardware.\n    Accepts a Hugging Face pipeline (or more likely,\n    a key pointing to such a pipeline on the cluster's object store)\n    and returns generated text.\n    \"\"\"\n    response = pipeline(prompt, *args, **kwargs)\n    if pipeline.task == \"text-generation\":\n        # Text generation return includes the starter text.\n        text = response[0][\"generated_text\"][len(prompt) :]\n    elif pipeline.task == \"text2text-generation\":\n        text = response[0][\"generated_text\"]\n    elif pipeline.task == \"summarization\":\n        text = response[0][\"summary_text\"]\n    else:\n        raise ValueError(\n            f\"Got invalid task {pipeline.task}, \"\n            f\"currently only {VALID_TASKS} are supported\"\n        )\n    if stop is not None:\n        text = enforce_stop_tokens(text, stop)\n    return text", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1183{"id": "538c83387d80-1", "text": "text = enforce_stop_tokens(text, stop)\n    return text\ndef _load_transformer(\n    model_id: str = DEFAULT_MODEL_ID,\n    task: str = DEFAULT_TASK,\n    device: int = 0,\n    model_kwargs: Optional[dict] = None,\n) -> Any:\n    \"\"\"Inference function to send to the remote hardware.\n    Accepts a huggingface model_id and returns a pipeline for the task.\n    \"\"\"\n    from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer\n    from transformers import pipeline as hf_pipeline\n    _model_kwargs = model_kwargs or {}\n    tokenizer = AutoTokenizer.from_pretrained(model_id, **_model_kwargs)\n    try:\n        if task == \"text-generation\":\n            model = AutoModelForCausalLM.from_pretrained(model_id, **_model_kwargs)\n        elif task in (\"text2text-generation\", \"summarization\"):\n            model = AutoModelForSeq2SeqLM.from_pretrained(model_id, **_model_kwargs)\n        else:\n            raise ValueError(\n                f\"Got invalid task {task}, \"\n                f\"currently only {VALID_TASKS} are supported\"\n            )\n    except ImportError as e:\n        raise ValueError(\n            f\"Could not load the {task} model due to missing dependencies.\"\n        ) from e\n    if importlib.util.find_spec(\"torch\") is not None:\n        import torch\n        cuda_device_count = torch.cuda.device_count()\n        if device < -1 or (device >= cuda_device_count):\n            raise ValueError(\n                f\"Got device=={device}, \"\n                f\"device is required to be within [-1, {cuda_device_count})\"\n            )\n        if device < 0 and cuda_device_count > 0:", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1184{"id": "538c83387d80-2", "text": ")\n        if device < 0 and cuda_device_count > 0:\n            logger.warning(\n                \"Device has %d GPUs available. \"\n                \"Provide device={deviceId} to `from_model_id` to use available\"\n                \"GPUs for execution. deviceId is -1 for CPU and \"\n                \"can be a positive integer associated with CUDA device id.\",\n                cuda_device_count,\n            )\n    pipeline = hf_pipeline(\n        task=task,\n        model=model,\n        tokenizer=tokenizer,\n        device=device,\n        model_kwargs=_model_kwargs,\n    )\n    if pipeline.task not in VALID_TASKS:\n        raise ValueError(\n            f\"Got invalid task {pipeline.task}, \"\n            f\"currently only {VALID_TASKS} are supported\"\n        )\n    return pipeline\n[docs]class SelfHostedHuggingFaceLLM(SelfHostedPipeline):\n    \"\"\"Wrapper around HuggingFace Pipeline API to run on self-hosted remote hardware.\n    Supported hardware includes auto-launched instances on AWS, GCP, Azure,\n    and Lambda, as well as servers specified\n    by IP address and SSH credentials (such as on-prem, or another cloud\n    like Paperspace, Coreweave, etc.).\n    To use, you should have the ``runhouse`` python package installed.\n    Only supports `text-generation`, `text2text-generation` and `summarization` for now.\n    Example using from_model_id:\n        .. code-block:: python\n            from langchain.llms import SelfHostedHuggingFaceLLM\n            import runhouse as rh\n            gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\")\n            hf = SelfHostedHuggingFaceLLM(", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1185{"id": "538c83387d80-3", "text": "hf = SelfHostedHuggingFaceLLM(\n                model_id=\"google/flan-t5-large\", task=\"text2text-generation\",\n                hardware=gpu\n            )\n    Example passing fn that generates a pipeline (bc the pipeline is not serializable):\n        .. code-block:: python\n            from langchain.llms import SelfHostedHuggingFaceLLM\n            from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n            import runhouse as rh\n            def get_pipeline():\n                model_id = \"gpt2\"\n                tokenizer = AutoTokenizer.from_pretrained(model_id)\n                model = AutoModelForCausalLM.from_pretrained(model_id)\n                pipe = pipeline(\n                    \"text-generation\", model=model, tokenizer=tokenizer\n                )\n                return pipe\n            hf = SelfHostedHuggingFaceLLM(\n                model_load_fn=get_pipeline, model_id=\"gpt2\", hardware=gpu)\n    \"\"\"\n    model_id: str = DEFAULT_MODEL_ID\n    \"\"\"Hugging Face model_id to load the model.\"\"\"\n    task: str = DEFAULT_TASK\n    \"\"\"Hugging Face task (\"text-generation\", \"text2text-generation\" or\n    \"summarization\").\"\"\"\n    device: int = 0\n    \"\"\"Device to use for inference. -1 for CPU, 0 for GPU, 1 for second GPU, etc.\"\"\"\n    model_kwargs: Optional[dict] = None\n    \"\"\"Key word arguments to pass to the model.\"\"\"\n    hardware: Any\n    \"\"\"Remote hardware to send the inference function to.\"\"\"\n    model_reqs: List[str] = [\"./\", \"transformers\", \"torch\"]\n    \"\"\"Requirements to install on hardware to inference the model.\"\"\"\n    model_load_fn: Callable = _load_transformer\n    \"\"\"Function to load the model remotely on the server.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1186{"id": "538c83387d80-4", "text": "\"\"\"Function to load the model remotely on the server.\"\"\"\n    inference_fn: Callable = _generate_text  #: :meta private:\n    \"\"\"Inference function to send to the remote hardware.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    def __init__(self, **kwargs: Any):\n        \"\"\"Construct the pipeline remotely using an auxiliary function.\n        The load function needs to be importable to be imported\n        and run on the server, i.e. in a module and not a REPL or closure.\n        Then, initialize the remote inference function.\n        \"\"\"\n        load_fn_kwargs = {\n            \"model_id\": kwargs.get(\"model_id\", DEFAULT_MODEL_ID),\n            \"task\": kwargs.get(\"task\", DEFAULT_TASK),\n            \"device\": kwargs.get(\"device\", 0),\n            \"model_kwargs\": kwargs.get(\"model_kwargs\", None),\n        }\n        super().__init__(load_fn_kwargs=load_fn_kwargs, **kwargs)\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            **{\"model_id\": self.model_id},\n            **{\"model_kwargs\": self.model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        return \"selfhosted_huggingface_pipeline\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        return self.client(pipeline=self.pipeline_ref, prompt=prompt, stop=stop)\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1187{"id": "538c83387d80-5", "text": "By Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/self_hosted_hugging_face.html"}1188{"id": "2b509f6942e9-0", "text": "Source code for langchain.llms.cohere\n\"\"\"Wrapper around Cohere APIs.\"\"\"\nimport logging\nfrom typing import Any, Dict, List, Optional\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\nlogger = logging.getLogger(__name__)\n[docs]class Cohere(LLM):\n    \"\"\"Wrapper around Cohere large language models.\n    To use, you should have the ``cohere`` python package installed, and the\n    environment variable ``COHERE_API_KEY`` set with your API key, or pass\n    it as a named parameter to the constructor.\n    Example:\n        .. code-block:: python\n            from langchain.llms import Cohere\n            cohere = Cohere(model=\"gptd-instruct-tft\", cohere_api_key=\"my-api-key\")\n    \"\"\"\n    client: Any  #: :meta private:\n    model: Optional[str] = None\n    \"\"\"Model name to use.\"\"\"\n    max_tokens: int = 256\n    \"\"\"Denotes the number of tokens to predict per generation.\"\"\"\n    temperature: float = 0.75\n    \"\"\"A non-negative float that tunes the degree of randomness in generation.\"\"\"\n    k: int = 0\n    \"\"\"Number of most likely tokens to consider at each step.\"\"\"\n    p: int = 1\n    \"\"\"Total probability mass of tokens to consider at each step.\"\"\"\n    frequency_penalty: float = 0.0\n    \"\"\"Penalizes repeated tokens according to frequency. Between 0 and 1.\"\"\"\n    presence_penalty: float = 0.0\n    \"\"\"Penalizes repeated tokens. Between 0 and 1.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/cohere.html"}1189{"id": "2b509f6942e9-1", "text": "\"\"\"Penalizes repeated tokens. Between 0 and 1.\"\"\"\n    truncate: Optional[str] = None\n    \"\"\"Specify how the client handles inputs longer than the maximum token\n    length: Truncate from START, END or NONE\"\"\"\n    cohere_api_key: Optional[str] = None\n    stop: Optional[List[str]] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        cohere_api_key = get_from_dict_or_env(\n            values, \"cohere_api_key\", \"COHERE_API_KEY\"\n        )\n        try:\n            import cohere\n            values[\"client\"] = cohere.Client(cohere_api_key)\n        except ImportError:\n            raise ImportError(\n                \"Could not import cohere python package. \"\n                \"Please install it with `pip install cohere`.\"\n            )\n        return values\n    @property\n    def _default_params(self) -> Dict[str, Any]:\n        \"\"\"Get the default parameters for calling Cohere API.\"\"\"\n        return {\n            \"max_tokens\": self.max_tokens,\n            \"temperature\": self.temperature,\n            \"k\": self.k,\n            \"p\": self.p,\n            \"frequency_penalty\": self.frequency_penalty,\n            \"presence_penalty\": self.presence_penalty,\n            \"truncate\": self.truncate,\n        }\n    @property\n    def _identifying_params(self) -> Dict[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {**{\"model\": self.model}, **self._default_params}\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/cohere.html"}1190{"id": "2b509f6942e9-2", "text": "def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"cohere\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to Cohere's generate endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = cohere(\"Tell me a joke.\")\n        \"\"\"\n        params = self._default_params\n        if self.stop is not None and stop is not None:\n            raise ValueError(\"`stop` found in both the input and default params.\")\n        elif self.stop is not None:\n            params[\"stop_sequences\"] = self.stop\n        else:\n            params[\"stop_sequences\"] = stop\n        response = self.client.generate(model=self.model, prompt=prompt, **params)\n        text = response.generations[0].text\n        # If stop tokens are provided, Cohere's endpoint returns them.\n        # In order to make this consistent with other endpoints, we strip them.\n        if stop is not None or self.stop is not None:\n            text = enforce_stop_tokens(text, params[\"stop_sequences\"])\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/cohere.html"}1191{"id": "d23ef9495344-0", "text": "Source code for langchain.llms.aleph_alpha\n\"\"\"Wrapper around Aleph Alpha APIs.\"\"\"\nfrom typing import Any, Dict, List, Optional, Sequence\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\n[docs]class AlephAlpha(LLM):\n    \"\"\"Wrapper around Aleph Alpha large language models.\n    To use, you should have the ``aleph_alpha_client`` python package installed, and the\n    environment variable ``ALEPH_ALPHA_API_KEY`` set with your API key, or pass\n    it as a named parameter to the constructor.\n    Parameters are explained more in depth here:\n    https://github.com/Aleph-Alpha/aleph-alpha-client/blob/c14b7dd2b4325c7da0d6a119f6e76385800e097b/aleph_alpha_client/completion.py#L10\n    Example:\n        .. code-block:: python\n            from langchain.llms import AlephAlpha\n            alpeh_alpha = AlephAlpha(aleph_alpha_api_key=\"my-api-key\")\n    \"\"\"\n    client: Any  #: :meta private:\n    model: Optional[str] = \"luminous-base\"\n    \"\"\"Model name to use.\"\"\"\n    maximum_tokens: int = 64\n    \"\"\"The maximum number of tokens to be generated.\"\"\"\n    temperature: float = 0.0\n    \"\"\"A non-negative float that tunes the degree of randomness in generation.\"\"\"\n    top_k: int = 0\n    \"\"\"Number of most likely tokens to consider at each step.\"\"\"\n    top_p: float = 0.0\n    \"\"\"Total probability mass of tokens to consider at each step.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1192{"id": "d23ef9495344-1", "text": "\"\"\"Total probability mass of tokens to consider at each step.\"\"\"\n    presence_penalty: float = 0.0\n    \"\"\"Penalizes repeated tokens.\"\"\"\n    frequency_penalty: float = 0.0\n    \"\"\"Penalizes repeated tokens according to frequency.\"\"\"\n    repetition_penalties_include_prompt: Optional[bool] = False\n    \"\"\"Flag deciding whether presence penalty or frequency penalty are\n    updated from the prompt.\"\"\"\n    use_multiplicative_presence_penalty: Optional[bool] = False\n    \"\"\"Flag deciding whether presence penalty is applied\n    multiplicatively (True) or additively (False).\"\"\"\n    penalty_bias: Optional[str] = None\n    \"\"\"Penalty bias for the completion.\"\"\"\n    penalty_exceptions: Optional[List[str]] = None\n    \"\"\"List of strings that may be generated without penalty,\n    regardless of other penalty settings\"\"\"\n    penalty_exceptions_include_stop_sequences: Optional[bool] = None\n    \"\"\"Should stop_sequences be included in penalty_exceptions.\"\"\"\n    best_of: Optional[int] = None\n    \"\"\"returns the one with the \"best of\" results\n    (highest log probability per token)\n    \"\"\"\n    n: int = 1\n    \"\"\"How many completions to generate for each prompt.\"\"\"\n    logit_bias: Optional[Dict[int, float]] = None\n    \"\"\"The logit bias allows to influence the likelihood of generating tokens.\"\"\"\n    log_probs: Optional[int] = None\n    \"\"\"Number of top log probabilities to be returned for each generated token.\"\"\"\n    tokens: Optional[bool] = False\n    \"\"\"return tokens of completion.\"\"\"\n    disable_optimizations: Optional[bool] = False\n    minimum_tokens: Optional[int] = 0\n    \"\"\"Generate at least this number of tokens.\"\"\"\n    echo: bool = False\n    \"\"\"Echo the prompt in the completion.\"\"\"\n    use_multiplicative_frequency_penalty: bool = False", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1193{"id": "d23ef9495344-2", "text": "\"\"\"Echo the prompt in the completion.\"\"\"\n    use_multiplicative_frequency_penalty: bool = False\n    sequence_penalty: float = 0.0\n    sequence_penalty_min_length: int = 2\n    use_multiplicative_sequence_penalty: bool = False\n    completion_bias_inclusion: Optional[Sequence[str]] = None\n    completion_bias_inclusion_first_token_only: bool = False\n    completion_bias_exclusion: Optional[Sequence[str]] = None\n    completion_bias_exclusion_first_token_only: bool = False\n    \"\"\"Only consider the first token for the completion_bias_exclusion.\"\"\"\n    contextual_control_threshold: Optional[float] = None\n    \"\"\"If set to None, attention control parameters only apply to those tokens that have\n    explicitly been set in the request.\n    If set to a non-None value, control parameters are also applied to similar tokens.\n    \"\"\"\n    control_log_additive: Optional[bool] = True\n    \"\"\"True: apply control by adding the log(control_factor) to attention scores.\n    False: (attention_scores - - attention_scores.min(-1)) * control_factor\n    \"\"\"\n    repetition_penalties_include_completion: bool = True\n    \"\"\"Flag deciding whether presence penalty or frequency penalty\n    are updated from the completion.\"\"\"\n    raw_completion: bool = False\n    \"\"\"Force the raw completion of the model to be returned.\"\"\"\n    aleph_alpha_api_key: Optional[str] = None\n    \"\"\"API key for Aleph Alpha API.\"\"\"\n    stop_sequences: Optional[List[str]] = None\n    \"\"\"Stop sequences to use.\"\"\"\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1194{"id": "d23ef9495344-3", "text": "\"\"\"Validate that api key and python package exists in environment.\"\"\"\n        aleph_alpha_api_key = get_from_dict_or_env(\n            values, \"aleph_alpha_api_key\", \"ALEPH_ALPHA_API_KEY\"\n        )\n        try:\n            import aleph_alpha_client\n            values[\"client\"] = aleph_alpha_client.Client(token=aleph_alpha_api_key)\n        except ImportError:\n            raise ImportError(\n                \"Could not import aleph_alpha_client python package. \"\n                \"Please install it with `pip install aleph_alpha_client`.\"\n            )\n        return values\n    @property\n    def _default_params(self) -> Dict[str, Any]:\n        \"\"\"Get the default parameters for calling the Aleph Alpha API.\"\"\"\n        return {\n            \"maximum_tokens\": self.maximum_tokens,\n            \"temperature\": self.temperature,\n            \"top_k\": self.top_k,\n            \"top_p\": self.top_p,\n            \"presence_penalty\": self.presence_penalty,\n            \"frequency_penalty\": self.frequency_penalty,\n            \"n\": self.n,\n            \"repetition_penalties_include_prompt\": self.repetition_penalties_include_prompt,  # noqa: E501\n            \"use_multiplicative_presence_penalty\": self.use_multiplicative_presence_penalty,  # noqa: E501\n            \"penalty_bias\": self.penalty_bias,\n            \"penalty_exceptions\": self.penalty_exceptions,\n            \"penalty_exceptions_include_stop_sequences\": self.penalty_exceptions_include_stop_sequences,  # noqa: E501\n            \"best_of\": self.best_of,\n            \"logit_bias\": self.logit_bias,\n            \"log_probs\": self.log_probs,\n            \"tokens\": self.tokens,\n            \"disable_optimizations\": self.disable_optimizations,\n            \"minimum_tokens\": self.minimum_tokens,\n            \"echo\": self.echo,", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1195{"id": "d23ef9495344-4", "text": "\"minimum_tokens\": self.minimum_tokens,\n            \"echo\": self.echo,\n            \"use_multiplicative_frequency_penalty\": self.use_multiplicative_frequency_penalty,  # noqa: E501\n            \"sequence_penalty\": self.sequence_penalty,\n            \"sequence_penalty_min_length\": self.sequence_penalty_min_length,\n            \"use_multiplicative_sequence_penalty\": self.use_multiplicative_sequence_penalty,  # noqa: E501\n            \"completion_bias_inclusion\": self.completion_bias_inclusion,\n            \"completion_bias_inclusion_first_token_only\": self.completion_bias_inclusion_first_token_only,  # noqa: E501\n            \"completion_bias_exclusion\": self.completion_bias_exclusion,\n            \"completion_bias_exclusion_first_token_only\": self.completion_bias_exclusion_first_token_only,  # noqa: E501\n            \"contextual_control_threshold\": self.contextual_control_threshold,\n            \"control_log_additive\": self.control_log_additive,\n            \"repetition_penalties_include_completion\": self.repetition_penalties_include_completion,  # noqa: E501\n            \"raw_completion\": self.raw_completion,\n        }\n    @property\n    def _identifying_params(self) -> Dict[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {**{\"model\": self.model}, **self._default_params}\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"alpeh_alpha\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to Aleph Alpha's completion endpoint.\n        Args:\n            prompt: The prompt to pass into the model.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1196{"id": "d23ef9495344-5", "text": "Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = alpeh_alpha(\"Tell me a joke.\")\n        \"\"\"\n        from aleph_alpha_client import CompletionRequest, Prompt\n        params = self._default_params\n        if self.stop_sequences is not None and stop is not None:\n            raise ValueError(\n                \"stop sequences found in both the input and default params.\"\n            )\n        elif self.stop_sequences is not None:\n            params[\"stop_sequences\"] = self.stop_sequences\n        else:\n            params[\"stop_sequences\"] = stop\n        request = CompletionRequest(prompt=Prompt.from_text(prompt), **params)\n        response = self.client.complete(model=self.model, request=request)\n        text = response.completions[0].completion\n        # If stop tokens are provided, Aleph Alpha's endpoint returns them.\n        # In order to make this consistent with other endpoints, we strip them.\n        if stop is not None or self.stop_sequences is not None:\n            text = enforce_stop_tokens(text, params[\"stop_sequences\"])\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/aleph_alpha.html"}1197{"id": "6afab3fba6de-0", "text": "Source code for langchain.llms.deepinfra\n\"\"\"Wrapper around DeepInfra APIs.\"\"\"\nfrom typing import Any, Dict, List, Mapping, Optional\nimport requests\nfrom pydantic import Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.llms.utils import enforce_stop_tokens\nfrom langchain.utils import get_from_dict_or_env\nDEFAULT_MODEL_ID = \"google/flan-t5-xl\"\n[docs]class DeepInfra(LLM):\n    \"\"\"Wrapper around DeepInfra deployed models.\n    To use, you should have the ``requests`` python package installed, and the\n    environment variable ``DEEPINFRA_API_TOKEN`` set with your API token, or pass\n    it as a named parameter to the constructor.\n    Only supports `text-generation` and `text2text-generation` for now.\n    Example:\n        .. code-block:: python\n            from langchain.llms import DeepInfra\n            di = DeepInfra(model_id=\"google/flan-t5-xl\",\n                                deepinfra_api_token=\"my-api-key\")\n    \"\"\"\n    model_id: str = DEFAULT_MODEL_ID\n    model_kwargs: Optional[dict] = None\n    deepinfra_api_token: Optional[str] = None\n    class Config:\n        \"\"\"Configuration for this pydantic object.\"\"\"\n        extra = Extra.forbid\n    @root_validator()\n    def validate_environment(cls, values: Dict) -> Dict:\n        \"\"\"Validate that api key and python package exists in environment.\"\"\"\n        deepinfra_api_token = get_from_dict_or_env(\n            values, \"deepinfra_api_token\", \"DEEPINFRA_API_TOKEN\"\n        )\n        values[\"deepinfra_api_token\"] = deepinfra_api_token\n        return values\n    @property", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/deepinfra.html"}1198{"id": "6afab3fba6de-1", "text": "return values\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\n            **{\"model_id\": self.model_id},\n            **{\"model_kwargs\": self.model_kwargs},\n        }\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of llm.\"\"\"\n        return \"deepinfra\"\n    def _call(\n        self,\n        prompt: str,\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n    ) -> str:\n        \"\"\"Call out to DeepInfra's inference API endpoint.\n        Args:\n            prompt: The prompt to pass into the model.\n            stop: Optional list of stop words to use when generating.\n        Returns:\n            The string generated by the model.\n        Example:\n            .. code-block:: python\n                response = di(\"Tell me a joke.\")\n        \"\"\"\n        _model_kwargs = self.model_kwargs or {}\n        res = requests.post(\n            f\"https://api.deepinfra.com/v1/inference/{self.model_id}\",\n            headers={\n                \"Authorization\": f\"bearer {self.deepinfra_api_token}\",\n                \"Content-Type\": \"application/json\",\n            },\n            json={\"input\": prompt, **_model_kwargs},\n        )\n        if res.status_code != 200:\n            raise ValueError(\"Error raised by inference API\")\n        text = res.json()[0][\"generated_text\"]\n        if stop is not None:\n            # I believe this is required since the stop tokens\n            # are not enforced by the model parameters\n            text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/deepinfra.html"}1199{"id": "6afab3fba6de-2", "text": "text = enforce_stop_tokens(text, stop)\n        return text\nBy Harrison Chase\n    \n      \u00a9 Copyright 2023, Harrison Chase.\n      \n  Last updated on May 28, 2023.", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/deepinfra.html"}1200{"id": "df62d22deb92-0", "text": "Source code for langchain.llms.ai21\n\"\"\"Wrapper around AI21 APIs.\"\"\"\nfrom typing import Any, Dict, List, Optional\nimport requests\nfrom pydantic import BaseModel, Extra, root_validator\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.llms.base import LLM\nfrom langchain.utils import get_from_dict_or_env\nclass AI21PenaltyData(BaseModel):\n    \"\"\"Parameters for AI21 penalty data.\"\"\"\n    scale: int = 0\n    applyToWhitespaces: bool = True\n    applyToPunctuations: bool = True\n    applyToNumbers: bool = True\n    applyToStopwords: bool = True\n    applyToEmojis: bool = True\n[docs]class AI21(LLM):\n    \"\"\"Wrapper around AI21 large language models.\n    To use, you should have the environment variable ``AI21_API_KEY``\n    set with your API key.\n    Example:\n        .. code-block:: python\n            from langchain.llms import AI21\n            ai21 = AI21(model=\"j2-jumbo-instruct\")\n    \"\"\"\n    model: str = \"j2-jumbo-instruct\"\n    \"\"\"Model name to use.\"\"\"\n    temperature: float = 0.7\n    \"\"\"What sampling temperature to use.\"\"\"\n    maxTokens: int = 256\n    \"\"\"The maximum number of tokens to generate in the completion.\"\"\"\n    minTokens: int = 0\n    \"\"\"The minimum number of tokens to generate in the completion.\"\"\"\n    topP: float = 1.0\n    \"\"\"Total probability mass of tokens to consider at each step.\"\"\"\n    presencePenalty: AI21PenaltyData = AI21PenaltyData()\n    \"\"\"Penalizes repeated tokens.\"\"\"\n    countPenalty: AI21PenaltyData = AI21PenaltyData()", "source": "https://python.langchain.com/en/latest/_modules/langchain/llms/ai21.html"}

Showing the first 1,200 of 4235 lines. Download the file for the rest.