codekingpro/portable-devtools
114k
1�
2 ��jN � �@ � S SK r S SKrS SKrS SKrS SKrS SKJr S SKJr S SK J3r4J r S SK r
S SKJr S SKJr \ " S5 rSS\S \S5\S \S \46S
jjrS\S\\
R, S \4S jrS\7\ S\S \8\\ 4S jrSS\S- S \4S jjrS \ \ 4S jr S\S \4S jr g)� N)�Path)�islice)�Iterable�TypeVar)�NDArray)�9NumpyArray�T� input_array�p�dim�eps�returnc �x � [ R R XUSS9n[ R " XC5 nX- nU$ )NT)�ord�axis�keepdims)�np�linalg�norm�maximum)r10 r r r
r �normalized_arrays �\D:\code\apps\devtools\python\user_packages\Python313\site-packages\fastembed/common/utils.py� normalizer s7 � �
�9�9�>�>�+�3��>� F�D�
�:�:�d� �D�"�)�� �� �attention_maskc �D � [ R " USS9R [ R 5 n[ R " USSU R11 S 45 n[ R " X- SS9n[ R " USS9nU[ R " US5 - nU$ )N�����)r � g��&� .>)r � expand_dims�astype�int64�tile�shape�sumr )r12 r �input_mask_expanded�sum_embeddings�sum_mask�pooled_embeddingss r � mean_poolingr) s� � � �.�.� �b�A�H�H����R�� �'�'�"5��1�k�>O�>O�PR�>S�7T�U���V�V�K� =�A�F�N��v�v�)��2�H�&����H�d�)C�C�� � r �iterable�sizec # � # � [ U 5 nU( a2 [ [ X!5 5 n[ U5 S:X a gUv � U( a M1 gg7f)z:13>>> list(iter_batch([1,2,3,4,5], 3))14[[1, 2, 3], [4, 5]]15r N)�iter�listr �len)r* r+ � source_iter�bs r �16iter_batchr2 # s< � � �17 �x�.�K�18� �� �*� +�� �q�6�Q�;� ��� �+�s �AA�A� cache_dirc �� � U cT [ R R [ R " 5 S5 n[ [ R " SU5 5 nO [ U 5 nUR SSS9 U$ )z*19Define the cache directory for fastembed20�fastembed_cache�FASTEMBED_CACHE_PATHT)�parents�exist_ok)�os�path�join�tempfile�21gettempdirr �getenv�mkdir)r3 �default_cache_dir�22cache_paths r �define_cache_dirrB 0 sb � � �� �G�G�L�L��)<�)<�)>�@Q� R���"�)�)�$:�<M�N�O�23��)�_�24����T�D��1� �r c �T � [ S [ [ R 5 5 5 $ )Nc 3 � # � U HE n[ R " [ U5 5 R S 5 ( d M8 [ U5 v � MG g7f)�PN)� unicodedata�category�chr�25startswith)�.0�is r � <genexpr>�&get_all_punctuation.<locals>.<genexpr>? s9 � � � �-�1��1E�1E�c�!�f�1M�1X�1X�Y\�1]���A���-�s26 �7A�A)�set�range�sys�27maxunicode� r r �get_all_punctuationrS > s% � � � � �c�n�n�-�� � r �textc �L � [ R " SSU [ R S9$ )Nz[^\w\s]� )�flags)�re�sub�UNICODE)rT s r �remove_non_alphanumericr[ D s � �
�6�6�*�c�4�r�z�z� :�:r )� r g�-���q=)N) r9 rP rX r< rF �pathlibr � itertoolsr �typingr r �numpyr � numpy.typingr �fastembed.common.typesr r �int�floatr r! r) r. r2 �strrB rN rS r[ rR r r �<module>rf s� �� � 28� � � � � � $� � � -� �C�L�� �:� �#� �� �e� �Xb� � �j� �'�"�(�(�:K� �PZ� �29��!�� 30�C� 31�H�T�!�W�4E� 32� ��d�33� �d� � �S��X� � ;�#� ;�#� ;r