codekingpro/portable-devtools

1Metadata-Version: 2.42Name: langsmith3Version: 0.8.34Summary: Client library to connect to the LangSmith Observability and Evaluation Platform.5Project-URL: Homepage, https://smith.langchain.com/6Project-URL: Documentation, https://docs.smith.langchain.com/7Project-URL: Repository, https://github.com/langchain-ai/langsmith-sdk8Author-email: LangChain <support@langchain.dev>9License: MIT10Keywords: evaluation,langchain,langsmith,language,llm,nlp,platform,tracing,translation11Requires-Python: >=3.1012Requires-Dist: httpx<1,>=0.23.013Requires-Dist: orjson>=3.9.14; platform_python_implementation != 'PyPy'14Requires-Dist: packaging>=23.215Requires-Dist: pydantic<3,>=216Requires-Dist: requests-toolbelt>=1.0.017Requires-Dist: requests>=2.0.018Requires-Dist: uuid-utils<1.0,>=0.12.019Requires-Dist: xxhash>=3.0.020Requires-Dist: zstandard>=0.23.021Provides-Extra: claude-agent-sdk22Requires-Dist: claude-agent-sdk>=0.1.0; (python_version >= '3.10') and extra == 'claude-agent-sdk'23Provides-Extra: google-adk24Requires-Dist: google-adk>=1.0.0; extra == 'google-adk'25Requires-Dist: wrapt>=1.16.0; extra == 'google-adk'26Provides-Extra: langsmith-pyo327Requires-Dist: langsmith-pyo3>=0.1.0rc2; extra == 'langsmith-pyo3'28Provides-Extra: openai-agents29Requires-Dist: openai-agents>=0.0.3; extra == 'openai-agents'30Provides-Extra: otel31Requires-Dist: opentelemetry-api>=1.30.0; extra == 'otel'32Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.30.0; extra == 'otel'33Requires-Dist: opentelemetry-sdk>=1.30.0; extra == 'otel'34Provides-Extra: pytest35Requires-Dist: pytest>=7.0.0; extra == 'pytest'36Requires-Dist: rich>=13.9.4; extra == 'pytest'37Requires-Dist: vcrpy>=7.0.0; extra == 'pytest'38Provides-Extra: sandbox39Requires-Dist: websockets>=15.0; extra == 'sandbox'40Provides-Extra: strands-agents41Requires-Dist: opentelemetry-api>=1.30.0; extra == 'strands-agents'42Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.30.0; extra == 'strands-agents'43Requires-Dist: opentelemetry-sdk>=1.30.0; extra == 'strands-agents'44Requires-Dist: strands-agents-tools>=0.2.0; extra == 'strands-agents'45Requires-Dist: strands-agents>=0.1.0; extra == 'strands-agents'46Provides-Extra: vcr47Requires-Dist: vcrpy>=7.0.0; extra == 'vcr'48Description-Content-Type: text/markdown49 50# LangSmith Client SDK51 52[![Release Notes](https://img.shields.io/github/release/langchain-ai/langsmith-sdk?logo=python)](https://github.com/langchain-ai/langsmith-sdk/releases)53[![Python Downloads](https://img.shields.io/pypi/dm/langsmith)](https://pypi.org/project/langsmith/)54 55This package contains the Python client for interacting with the [LangSmith platform](https://smith.langchain.com/).56 57To install:58 59```bash60pip install -U langsmith61export LANGSMITH_TRACING=true62export LANGSMITH_API_KEY=ls_...63```64 65Then trace:66 67```python68import openai69from langsmith.wrappers import wrap_openai70from langsmith import traceable71 72# Auto-trace LLM calls in-context73client = wrap_openai(openai.Client())74 75@traceable # Auto-trace this function76def pipeline(user_input: str):77    result = client.chat.completions.create(78        messages=[{"role": "user", "content": user_input}],79        model="gpt-3.5-turbo"80    )81    return result.choices[0].message.content82 83pipeline("Hello, world!")84```85 86See the resulting nested trace [🌐 here](https://smith.langchain.com/public/b37ca9b1-60cd-4a2a-817e-3c4e4443fdc0/r).87 88LangSmith helps you and your team develop and evaluate language models and intelligent agents. It is compatible with any LLM application.89 90> **Cookbook:** For tutorials on how to get more value out of LangSmith, check out the [Langsmith Cookbook](https://github.com/langchain-ai/langsmith-cookbook/tree/main) repo.91 92A typical workflow looks like:93 941. Set up an account with LangSmith.952. Log traces while debugging and prototyping.963. Run benchmark evaluations and continuously improve with the collected data.97 98We'll walk through these steps in more detail below.99 100## 1. Connect to LangSmith101 102Sign up for [LangSmith](https://smith.langchain.com/) using your GitHub, Discord accounts, or an email address and password. If you sign up with an email, make sure to verify your email address before logging in.103 104Then, create a unique API key on the [Settings Page](https://smith.langchain.com/settings), which is found in the menu at the top right corner of the page.105 106> [!NOTE]107> Save the API Key in a secure location. It will not be shown again.108 109## 2. Log Traces110 111You can log traces natively using the LangSmith SDK or within your LangChain application.112 113### Logging Traces with LangChain114 115LangSmith seamlessly integrates with the Python LangChain library to record traces from your LLM applications.116 1171. **Copy the environment variables from the Settings Page and add them to your application.**118 119Tracing can be activated by setting the following environment variables or by manually specifying the LangChainTracer.120 121```python122import os123os.environ["LANGSMITH_TRACING"] = "true"124os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"125# os.environ["LANGSMITH_ENDPOINT"] = "https://eu.api.smith.langchain.com" # If signed up in the EU region126os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"127# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not set128# os.environ["LANGSMITH_WORKSPACE_ID"] = "<YOUR-WORKSPACE-ID>" # Required for org-scoped API keys129```130 131> **Tip:** Projects are groups of traces. All runs are logged to a project. If not specified, the project is set to `default`.132 1332. **Run an Agent, Chain, or Language Model in LangChain**134 135If the environment variables are correctly set, your application will automatically connect to the LangSmith platform.136 137```python138from langchain_core.runnables import chain139 140@chain141def add_val(x: dict) -> dict:142    return {"val": x["val"] + 1}143 144add_val({"val": 1})145```146 147### Logging Traces Outside LangChain148 149You can still use the LangSmith development platform without depending on any150LangChain code.151 1521. **Copy the environment variables from the Settings Page and add them to your application.**153 154```python155import os156os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"157os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"158# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not set159```160 1612. **Log traces**162 163The easiest way to log traces using the SDK is via the `@traceable` decorator. Below is an example.164 165```python166from datetime import datetime167from typing import List, Optional, Tuple168 169import openai170from langsmith import traceable171from langsmith.wrappers import wrap_openai172 173client = wrap_openai(openai.Client())174 175@traceable176def argument_generator(query: str, additional_description: str = "") -> str:177    return client.chat.completions.create(178        [179            {"role": "system", "content": "You are a debater making an argument on a topic."180             f"{additional_description}"181             f" The current time is {datetime.now()}"},182            {"role": "user", "content": f"The discussion topic is {query}"}183        ]184    ).choices[0].message.content185 186 187 188@traceable189def argument_chain(query: str, additional_description: str = "") -> str:190    argument = argument_generator(query, additional_description)191    # ... Do other processing or call other functions...192    return argument193 194argument_chain("Why is blue better than orange?")195```196 197Alternatively, you can manually log events using the `Client` directly or using a `RunTree`, which is what the traceable decorator is meant to manage for you!198 199A RunTree tracks your application. Each RunTree object is required to have a `name` and `run_type`. These and other important attributes are as follows:200 201- `name`: `str` - used to identify the component's purpose202- `run_type`: `str` - Currently one of "llm", "chain" or "tool"; more options will be added in the future203- `inputs`: `dict` - the inputs to the component204- `outputs`: `Optional[dict]` - the (optional) returned values from the component205- `error`: `Optional[str]` - Any error messages that may have arisen during the call206 207```python208from langsmith.run_trees import RunTree209 210parent_run = RunTree(211    name="My Chat Bot",212    run_type="chain",213    inputs={"text": "Summarize this morning's meetings."},214    # project_name= "Defaults to the LANGSMITH_PROJECT env var"215)216parent_run.post()217# .. My Chat Bot calls an LLM218child_llm_run = parent_run.create_child(219    name="My Proprietary LLM",220    run_type="llm",221    inputs={222        "prompts": [223            "You are an AI Assistant. The time is XYZ."224            " Summarize this morning's meetings."225        ]226    },227)228child_llm_run.post()229child_llm_run.end(230    outputs={231        "generations": [232            "I should use the transcript_loader tool"233            " to fetch meeting_transcripts from XYZ"234        ]235    }236)237child_llm_run.patch()238# ..  My Chat Bot takes the LLM output and calls239# a tool / function for fetching transcripts ..240child_tool_run = parent_run.create_child(241    name="transcript_loader",242    run_type="tool",243    inputs={"date": "XYZ", "content_type": "meeting_transcripts"},244)245child_tool_run.post()246# The tool returns meeting notes to the chat bot247child_tool_run.end(outputs={"meetings": ["Meeting1 notes.."]})248child_tool_run.patch()249 250child_chain_run = parent_run.create_child(251    name="Unreliable Component",252    run_type="tool",253    inputs={"input": "Summarize these notes..."},254)255child_chain_run.post()256 257try:258    # .... the component does work259    raise ValueError("Something went wrong")260    child_chain_run.end(outputs={"output": "foo"}261    child_chain_run.patch()262except Exception as e:263    child_chain_run.end(error=f"I errored again {e}")264    child_chain_run.patch()265    pass266# .. The chat agent recovers267 268parent_run.end(outputs={"output": ["The meeting notes are as follows:..."]})269res = parent_run.patch()270res.result()271```272 273## Create a Dataset from Existing Runs274 275Once your runs are stored in LangSmith, you can convert them into a dataset.276For this example, we will do so using the Client, but you can also do this using277the web interface, as explained in the [LangSmith docs](https://docs.smith.langchain.com/).278 279```python280from langsmith import Client281 282client = Client()283dataset_name = "Example Dataset"284# We will only use examples from the top level AgentExecutor run here,285# and exclude runs that errored.286runs = client.list_runs(287    project_name="my_project",288    execution_order=1,289    error=False,290)291 292dataset = client.create_dataset(dataset_name, description="An example dataset")293for run in runs:294    client.create_example(295        inputs=run.inputs,296        outputs=run.outputs,297        dataset_id=dataset.id,298    )299```300 301## Evaluating Runs302 303Check out the [LangSmith Testing & Evaluation dos](https://docs.smith.langchain.com/evaluation) for up-to-date workflows.304 305For generating automated feedback on individual runs, you can run evaluations directly using the LangSmith client.306 307```python308from typing import Optional309from langsmith.evaluation import StringEvaluator310 311 312def jaccard_chars(output: str, answer: str) -> float:313    """Naive Jaccard similarity between two strings."""314    prediction_chars = set(output.strip().lower())315    answer_chars = set(answer.strip().lower())316    intersection = prediction_chars.intersection(answer_chars)317    union = prediction_chars.union(answer_chars)318    return len(intersection) / len(union)319 320 321def grader(run_input: str, run_output: str, answer: Optional[str]) -> dict:322    """Compute the score and/or label for this run."""323    if answer is None:324        value = "AMBIGUOUS"325        score = 0.5326    else:327        score = jaccard_chars(run_output, answer)328        value = "CORRECT" if score > 0.9 else "INCORRECT"329    return dict(score=score, value=value)330 331evaluator = StringEvaluator(evaluation_name="Jaccard", grading_function=grader)332 333runs = client.list_runs(334    project_name="my_project",335    execution_order=1,336    error=False,337)338for run in runs:339    client.evaluate_run(run, evaluator)340```341 342## Integrations343 344LangSmith easily integrates with your favorite LLM framework.345 346## OpenAI SDK347 348<!-- markdown-link-check-disable -->349 350We provide a convenient wrapper for the [OpenAI SDK](https://platform.openai.com/docs/api-reference).351 352In order to use, you first need to set your LangSmith API key.353 354```shell355export LANGSMITH_API_KEY=<your-api-key>356```357 358Next, you will need to install the LangSmith SDK:359 360```shell361pip install -U langsmith362```363 364After that, you can wrap the OpenAI client:365 366```python367from openai import OpenAI368from langsmith import wrappers369 370client = wrappers.wrap_openai(OpenAI())371```372 373Now, you can use the OpenAI client as you normally would, but now everything is logged to LangSmith!374 375```python376client.chat.completions.create(377    model="gpt-4",378    messages=[{"role": "user", "content": "Say this is a test"}],379)380```381 382Oftentimes, you use the OpenAI client inside of other functions.383You can get nested traces by using this wrapped client and decorating those functions with `@traceable`.384See [this documentation](https://docs.smith.langchain.com/tracing/faq/logging_and_viewing) for more documentation how to use this decorator385 386```python387from langsmith import traceable388 389@traceable(name="Call OpenAI")390def my_function(text: str):391    return client.chat.completions.create(392        model="gpt-4",393        messages=[{"role": "user", "content": f"Say {text}"}],394    )395 396my_function("hello world")397```398 399## Instructor400 401We provide a convenient integration with [Instructor](https://jxnl.github.io/instructor/), largely by virtue of it essentially just using the OpenAI SDK.402 403In order to use, you first need to set your LangSmith API key.404 405```shell406export LANGSMITH_API_KEY=<your-api-key>407```408 409Next, you will need to install the LangSmith SDK:410 411```shell412pip install -U langsmith413```414 415After that, you can wrap the OpenAI client:416 417```python418from openai import OpenAI419from langsmith import wrappers420 421client = wrappers.wrap_openai(OpenAI())422```423 424After this, you can patch the OpenAI client using `instructor`:425 426```python427import instructor428 429client = instructor.patch(OpenAI())430```431 432Now, you can use `instructor` as you normally would, but now everything is logged to LangSmith!433 434```python435from pydantic import BaseModel436 437 438class UserDetail(BaseModel):439    name: str440    age: int441 442 443user = client.chat.completions.create(444    model="gpt-3.5-turbo",445    response_model=UserDetail,446    messages=[447        {"role": "user", "content": "Extract Jason is 25 years old"},448    ]449)450```451 452Oftentimes, you use `instructor` inside of other functions.453You can get nested traces by using this wrapped client and decorating those functions with `@traceable`.454See [this documentation](https://docs.smith.langchain.com/tracing/faq/logging_and_viewing) for more documentation how to use this decorator455 456```python457@traceable()458def my_function(text: str) -> UserDetail:459    return client.chat.completions.create(460        model="gpt-3.5-turbo",461        response_model=UserDetail,462        messages=[463            {"role": "user", "content": f"Extract {text}"},464        ]465    )466 467 468my_function("Jason is 25 years old")469```470 471## Pytest Plugin472 473The LangSmith pytest plugin lets Python developers define their datasets and evaluations as pytest test cases.474See [online docs](https://docs.smith.langchain.com/evaluation/how_to_guides/pytest) for more information.475 476This plugin is installed as part of the LangSmith SDK, and is enabled by default.477See also official pytest docs: [How to install and use plugins](https://docs.pytest.org/en/stable/how-to/plugins.html)478 479## Additional Documentation480 481To learn more about the LangSmith platform, check out the [docs](https://docs.smith.langchain.com/).482 483 484# License485 486The LangSmith SDK is licensed under the [MIT License](../LICENSE).487 488The copyright information for certain dependencies' are reproduced in their corresponding COPYRIGHT.txt files in this repo, including the following:489 490- [uuid-utils](docs/templates/uuid-utils/COPYRIGHT.txt)491- [zstandard](docs/templates/zstandard/COPYRIGHT.txt)492