codekingpro/portable-devtools
114k
1�
2 ��j,; � �� � S r SSKJr SSKrSSKrSSKJrJrJrJ r J3r4J r J r SSK
Jr SSK
Jr SSK
Jr SSK
Jr \( a SS KJr SS5KJr " S S 5 r\" 5 r " S
S5 r " S S5 r \ " 5 r S/r g)a< Make approximate assertions as "expectations" on test results.6 7This module is designed to be used within test cases decorated with the8`@pytest.mark.decorator` decorator9 10It allows you to log scores about a test case and optionally make assertions that log as11"expectation" feedback to LangSmith.12 13Example:14 ```python15 import pytest16 from langsmith import expect17 18 19 @pytest.mark.langsmith20 def test_output_semantically_close():21 response = oai_client.chat.completions.create(22 model="gpt-3.5-turbo",23 messages=[24 {"role": "system", "content": "You are a helpful assistant."},25 {"role": "user", "content": "Say hello!"},26 ],27 )28 response_txt = response.choices[0].message.content29 # Intended usage30 expect.embedding_distance(31 prediction=response_txt,32 reference="Hello!",33 ).to_be_less_than(0.9)34 35 # Score the test case36 matcher = expect.edit_distance(37 prediction=response_txt,38 reference="Hello!",39 )40 # Apply an assertion and log 'expectation' feedback to LangSmith41 matcher.to_be_less_than(1)42 43 # You can also directly make assertions on values directly44 expect.value(response_txt).to_contain("Hello!")45 # Or using a custom check46 expect.value(response_txt).against(lambda x: "Hello" in x)47 48 # You can even use this for basic metric logging within tests49 50 expect.score(0.8)51 expect.score(0.7, key="similarity").to_be_greater_than(0.7)52 ```53� )� annotationsN)�
TYPE_CHECKING�Any�Callable�Literal�Optional�Union�overload��client)� run_helpers)� run_trees)�utils)�EditDistanceConfig)�EmbeddingConfigc �, � \ rS rSrSrSS jrSS jrSrg) � _NULL_SENTRY�L z�A sentinel singleton class used to distinguish omitted keyword arguments54from those passed in with the value None (which may have different behavior).55c � � g)NF� ��selfs �WD:\code\apps\devtools\python\user_packages\Python313\site-packages\langsmith/_expect.py�__bool__�_NULL_SENTRY.__bool__Q s � �� c � � g)N� NOT_GIVENr r s r �__repr__�_NULL_SENTRY.__repr__T s � �r r N)�returnzLiteral[False])r! �str)�__name__�56__module__� __qualname__�__firstlineno__�__doc__r r �__static_attributes__r r r r r L s � ���r r c � � \ rS rSrSr S SS jjrSSS jjrSS jrSS jrSS jr SS jr57SSS58 jjr SS jr SS jr
SS
jrSS jrSrg) �_Matcher�[ z4A class for making assertions on expectation values.Nc � � Xl X l X0l U=( d [ R " SS9U l [ R " 5 U l U R ( a U R R U l59 g UU l60 g )N� �� max_workers) �_client�key�value�ls_utils�ContextThreadPoolExecutor� _executor�rh�get_current_run_tree�_rt�trace_id�_run_id)r r r1 r2 r5 �run_ids r �__init__�_Matcher.__init__^ sX � � � ����61�"�W�h�&H�&H�UV�&W����*�*�,���,0�H�H�t�x�x�(�(�� �&�� r c 62 � � [ R " 5 ( d� U R ( d [ R " 5 U l U R63 R
U R R U R SUUU R ( a U R R OS U R ( a U R R OS S9 g g )N� expectation)r; r1 �score�comment�64session_id�65start_time) r3 �test_tracking_is_disabledr0 �rt�get_cached_clientr5 �submit�create_feedbackr: r8 rB rC )r r@ �messages r �_submit_feedback�_Matcher._submit_feedbackm s� � ��1�1�3�3��<�<� !� 3� 3� 5�� � �N�N� !� !�� � �,�,��|�|�!�� �26�(�(�4�8�8�.�.��26�(�(�4�8�8�.�.��
"�
� 4r c �� � U( d U5 eU R SSU R SU 3S9 g ! [ a# nU R S[ U5 5 US eS nAff = f)N� z Success: �.)rI r )rJ r1 �AssertionError�repr)r � conditionrI � method_name�es r �_assert�_Matcher._assert{ sd � � � � %�g� %�9� � !� !�!�y����66�!�K�=�-Q� !� R�� � � � !� !�!�T�!�W� -��� �� �s �.1 �67A � A�A c �~ � U R U R U: SU R SU SU R 3S5 g)z�Assert that the expectation value is less than the given value.68 69Args:70 value: The value to compare against.71 72Raises:73 AssertionError: If the expectation value is not less than the given value.74� Expected z to be less than �75, but got �to_be_less_thanN�rT r2 r1 �r r2 s r rY �_Matcher.to_be_less_than� s> � �
� � � �J�J�� �����z�!2�5�'��D�J�J�<� P� � 76r c �~ � U R U R U:� SU R SU SU R 3S5 g)z�Assert that the expectation value is greater than the given value.77 78Args:79 value: The value to compare against.80 81Raises:82 AssertionError: If the expectation value is not83 greater than the given value.84rW z to be greater than rX �to_be_greater_thanNrZ r[ s r r^ �_Matcher.to_be_greater_than� s> � �
� � � �J�J�� �����z�!5�e�W�J�t�z�z�l� S� � 85r c � � U R XR s=: =( a U: Os SU R SU SU SU R 3S5 g)a Assert that the expectation value is between the given min and max values.86 87Args:88 min_value: The minimum value (exclusive).89 max_value: The maximum value (exclusive).90 91Raises:92 AssertionError: If the expectation value is not between the min and max.93rW z to be between z and rX �
to_be_betweenNrZ )r � min_value� max_values r ra �_Matcher.to_be_between� sP � �
� � � �94�95� .� .�Y� .�����z� �� �5�� � L� �96�97�|�
%� � 98r c � � U R [ U R U5 [ X5 :H SU R SU SU R 3S5 g)a3 Assert that the expectation value is approximately equal to the given value.99 100Args:101 value: The value to compare against.102 precision: The number of decimal places to round to for comparison.103 104Raises:105 AssertionError: If the rounded expectation value106 does not equal the rounded given value.107rW z to be approximately rX �to_be_approximatelyN)rT �roundr2 r1 )r r2 � precisions r rf � _Matcher.to_be_approximately� sJ � �
� � � �$�*�*�i� (�E�%�,C� C�����z�!6�u�g�Z��108�109�|� T� !� 110r c �~ � U R U R U:H SU R SU SU R 3S5 g)z�Assert that the expectation value equals the given value.111 112Args:113 value: The value to compare against.114 115Raises:116 AssertionError: If the expectation value does117 not exactly equal the given value.118rW z to be equal to rX �to_equalNrZ r[ s r rk �_Matcher.to_equal� s> � �
� � � �J�J�%� �����z�!1�%��119�4�:�:�,� O� � 120r c �v � U R U R SL SU R SU R 3S5 g)zrAssert that the expectation value is `None`.121 122Raises:123 AssertionError: If the expectation value is not `None`.124NrW z to be None, but got �125to_be_nonerZ r s r rn �_Matcher.to_be_none� s8 � �
� � � �J�J�$� �����z�!6�t�z�z�l� C� � 126r c �d � U R XR ; SU R SU S3S5 g)z�Assert that the expectation value contains the given value.127 128Args:129 value: The value to check for containment.130 131Raises:132 AssertionError: If the expectation value does not contain the given value.133rW z to contain z, but it does not�134to_containNrZ r[ s r rq �_Matcher.to_contain� s4 � �
� � � �Z�Z� �����z� �e�W�4E� F� � 135r c � � [ R " U5 nU R U" U R 5 SU SU R 3S5 g)z�Assert the expectation value against a custom function.136 137Args:138 func: A custom function that takes the expectation value as input.139 140Raises:141 AssertionError: If the custom function returns False.142z143Assertion z failed for �againstN)�inspect� signaturerT r2 r1 )r �func�func_signatures r rt �_Matcher.against� sD � � !�*�*�4�0�� � � � ���� �� �(� �T�X�X�J� ?� � 144r )r0 r5 r8 r: r1 r2 )NN)145r �Optional[ls_client.Client]r1 r" r2 r r5 z,Optional[ls_utils.ContextThreadPoolExecutor]r; �
Optional[str]�N)r@ �intrI r{ r! �None)rQ �boolrI r" rR r" r! r~ )r2 �floatr! r~ )rb r� rc r� r! r~ )� )r2 r� rh r} r! r~ )r! r~ )r2 r r! r~ )rw r r! r~ )r# r$ r% r& r' r< rJ rT rY r^ ra rf rk rn rq rt r( r r r r* r* [ s} � �>� CG� $�
A�*�
A� �
A� �
A�146 @�
A� �
A� � �
147� 148� 149�"150�"151� 152153�
154� 155r r* c �� � \ rS rSrSrSS.SS jjrSS. SS jjrSS. SS jjrSS jrS156SSS . SS jjr \157SS
j5 r \158SS j5 r \ S4 SS jjr SS jr
Srg)�_Expecti z1A class for setting expectations on test results.Nr c � � Xl [ R " SS9U l [ R159 " U R R SS9 g )Nr- r. T)�wait)r0 r3 r4 �executor�atexit�register�shutdown�r r s r r< �_Expect.__init__ s1 � � � � �:�:�q�I��
�����
�
�.�.�T�:r ��configc �( � SSK Jn U=( d 0 nUR S5 ( a SOSnU" US9nUR XS9nXVR S.nU R S UUS160U S UR 3S .5 [
U R S XpR S
9$ )a� Compute the embedding distance between the prediction and reference.161 162This logs the embedding distance to LangSmith and returns a `_Matcher` instance163for making assertions on the distance value.164 165By default, this uses the OpenAI API for computing embeddings.166 167Args:168 prediction: The predicted string to compare.169 reference: The reference string to compare against.170 config: Optional configuration for the embedding distance evaluator.171 172 Supported options:173 174 - `encoder`: A custom encoder function to encode the list of input175 strings to embeddings.176 177 Defaults to the OpenAI API.178 - `metric`: The distance metric to use for comparison.179 180 Supported values: `'cosine'`, `'euclidean'`, `'manhattan'`,181 `'chebyshev'`, `'hamming'`.182 183Returns:184 A `_Matcher` instance for the embedding distance value.185 186 187Example:188 ```python189 expect.embedding_distance(190 prediction="hello",191 reference="hi",192 ).to_be_less_than(1.0)193 ```194r )�EmbeddingDistance�encoder�custom�openair� ��195prediction� reference)r� �metric�embedding_distance�Using z196, Metric: �r@ � source_inforA �r5 ) �'langsmith._internal._embedding_distancer� �get�evaluate�distancerJ r* r0 r� ) r r� r� r� r� � encoder_func� evaluatorr@ �src_infos r r� �_Expect.embedding_distance s� � �T N�� �2��#)�:�:�i�#8�#8�x�h� �%�V�4� ��"�"�j�"�N�� +�7I�7I�J�� � � � � � '� #�L�>��I�<N�<N�;O�P�
� 197� � �L�L�.�� � �198� 199r c �& � SSK Jn U=( d 0 nUR S5 =( d SnUR SS5 nU" US9nUR XS9nXVS .n U R S200UU S U S U 3S
.5 [ U R S201UU R S9$ )a� Compute the string distance between the prediction and reference.202 203This logs the string distance (Damerau-Levenshtein) to LangSmith and returns204a `_Matcher` instance for making assertions on the distance value.205 206This depends on the `rapidfuzz` package for string distance computation.207 208Args:209 prediction: The predicted string to compare.210 reference: The reference string to compare against.211 config: Optional configuration for the string distance evaluator.212 213 Supported options:214 215 - `metric`: The distance metric to use for comparison.216 217 Supported values: `'damerau_levenshtein'`, `'levenshtein'`,218 `'jaro'`, `'jaro_winkler'`, `'hamming'`, `'indel'`.219 - `normalize_score`: Whether to normalize the score between `0` and `1`.220 221Returns:222 A `_Matcher` instance for the string distance value.223 224Examples:225 ```python226 expect.edit_distance("hello", "helo").to_be_less_than(1)227 ```228r )� EditDistancer� �damerau_levenshtein�normalize_scoreTr� r� )r� � normalize�
edit_distancer� z
, Normalize: r� r� )�"langsmith._internal._edit_distancer� r� r� rJ r* r0 r� )229r r� r� r� r� r� r� r� r@ r� s230 r r� �_Expect.edit_distanceF s� � �F D�� �2�����H�%�>�)>���J�J� 0�$�7� � ��/� ��"�"�j�"�N�� $�=�� � � � � � '� #�F�8�=�� �D�
� 231� � �L�L� � ��m�m� 232� 233r c �@ � [ U R SXR S9$ )z�Create a `_Matcher` instance for making assertions on the given value.234 235Args:236 value: The value to make assertions on.237 238Returns:239 A `_Matcher` instance for the given value.240 241Example:242 ```python243 expect.value(10).to_be_less_than(20)244 ```245r2 r� )r* r0 r� r[ s r r2 �
_Expect.value� s � � �� � �g�u�
�
�N�Nr r@ )r1 �
source_run_idrA c �r � U R UUSS0UUS.5 [ U R X!U R S9$ )aN Log a numeric score to LangSmith.246 247Args:248 score: The score value to log.249 key: The key to use for logging the score. Defaults to `'score'`.250 251Example:252 ```python253 expect.score(0.8) # doctest: +ELLIPSIS254 <langsmith._expect._Matcher object at ...>255 256 expect.score(0.8, key="similarity").to_be_greater_than(0.7)257 ```258�methodz expect.score)r@ r� r� rA r� )rJ r* r0 r� )r r@ r1 r� rA s r r@ �
_Expect.score� sD � �,
� � � � � (�.� 9�!.�"�
� 259� �� � �c�D�M�M�J�Jr c � � g r| r r[ s r �__call__�_Expect.__call__� s � �36r c � � g r| r r� s r r� r� � s � �CFr c �L � [ US9nU[ La UR U5 $ U$ )Nr )r� r r2 )r r2 r �expecteds r r� r� � s* � � �&�)�� � � !��>�>�%�(� (��r c �J � [ R " 5 nU( a UR OS n[ R " 5 ( d_ U R260 ( d [ R " 5 U l U R R " U R261 R 4XAS.UD6 g g )N)r; r1 ) r6 r7 r9 r3 rD r0 rE rF r� rG rH )r r1 �results� current_runr; s r rJ �_Expect._submit_feedback� sy � ��-�-�/� �)4��%�%�$���1�1�3�3��<�<� !� 3� 3� 5�� � �M�M� � �� � �,�,�
�5;�
�HO�
� 4r )r0 r� )r rz )r� r" r� r" r� zOptional[EmbeddingConfig]r! r* )r� r" r� r" r� z Optional[EditDistanceConfig]r! r* )r2 r r! r* )262r@ zUnion[float, int, bool]r1 r" r� zOptional[ls_client.ID_TYPE]rA r{ r! r* )r zls_client.Clientr! r� )r rz r2 z
Optional[Any]r! zUnion[_Expect, _Matcher])r1 r" r� �dict)r# r$ r% r& r' r< r� r� r2 r@ r263 r� r rJ r( r r r r� r� s � �;�?C� ;� -1� ;264��;265� �;266�267 *� ;268� 269�
;270�D 04� 8271��8272� �8273�274 -� 8275� 276�
8277�tO�( �59�!%�
K�&� K� � K�278 3� K� �
K� 279� K�F �6� �6�
�F� �F� )�-1� � +� � � �280 281"� �r r� �expect) r' �282__future__r r� ru �typingr r r r r r r283 � langsmithr � ls_clientr
r6 r rE r r3 r� r r� r r r r* r� r� �__all__r r r �<module>r� sy ��0�d #�
� �� � � *� '� %� '��E�G� � �
�N� �c284� c285�LK� K�\ 286��� �*�r