CintraAI/code-chunker
5
1�
2 ���f� � �X � d dl mZmZ d dlmZ d dlmZ G d� de� Z G d� de� Zy) � )�ABC�abstractmethod)�3CodeParser)� count_tokensc �d � e Zd Zdd�Zed� � Zed� � Zed� � Zed� � Z ed� � Z4y) �Chunkerc � � || _ y �N)�
encoding_name)�selfr s �.c:\Users\joesh\.vscode\code-chunker\Chunker.py�__init__zChunker.__init__ s5 � � *��� c � � y r6 � )r �content� token_limits r
�chunkz
Chunker.chunk7 � � � r c � � y r8 r )r �chunked_content� chunk_numbers r
� get_chunkzChunker.get_chunk r r c � � | j � D ]5 \ }}t d|� d�� t d� t |� t d� �7 y )NzChunk �:z(========================================)�items�print)�chunksr �9chunk_codes r
� print_chunkszChunker.print_chunks sC � �(.� � �� � $�L�*� �F�<�.��*� +� �(�O� �*� � �(�O� r c �@ � dj | j � � S �N�10)�join�values)r s r
� consolidate_chunks_into_filez$Chunker.consolidate_chunks_into_file s � ��y�y�� � � �)�)r c �: � | j d� }t |� S r" )�split�len)�consolidated_chunks�liness r
� count_lineszChunker.count_lines s � �#�)�)�$�/���5�z�r N�zgpt-4) �__name__�11__module__� __qualname__r r r r � staticmethodr r& r, r r r
r r sk � �+� �
� �
� �
� �
� � � � � �*� �*� �� �r r c �2 � � e Zd Zd� fd� Zdefd�Zd� Z� xZS )� CodeChunkerc �2 �� t �| � |� || _ y r12 )�superr �file_extension)r r6 r � __class__s �r
r zCodeChunker.__init__% s �� �
��� �'� ,��r �returnc �F � t | j � }i }d}d}|j d� }d}d} d}13t |j || j � � } t |j || j � � } g }
| D ]> }|dz14 }d }|| v r |}|dz }|| v r� |r|
j
|� �.|
j
|� �@ t t |
� � } |t |� k �r|| }t || j � }||z |kD r�|| v r|}n+t t | D �cg c]15 }||k s� |�� c}|16�� |17� }||18k( r|| vr ||z
}|dz
}n�||19k( r||k( r ||z
}|dz
}n�||20k( r<|| v r8dj ||21| � }|j � r22||| < | dz
} d}|}23|dz
}n?dj ||24| � }|j � r25||| < | dz
} |}d}|}26n27||z
}|dz
}|t |� k r��dj ||28d � }|j � r||| < |S c c}w )N� r r# � )�default)r r6 r( �sorted� get_lines_for_points_of_interest�get_lines_for_comments�append�setr) r r �maxr$ �strip)r �coder � code_parserr �
current_chunk� token_countr+ �ir �29start_line� breakpoints�comments�adjusted_breakpoints�bp� current_line�highest_comment_line�line�new_token_count� stop_line�x�current_chunk_codes r
r zCodeChunker.chunk) s� � � ��!4�!4�5� ����
�� ��30�31�4� ��
��� ��32� �[� I� I�$�PT�Pc�Pc� d�e� ��+�<�<�T�4�CV�CV�W�X�� !� � � �B� ��6�L�#'� � �(�*�'3�$� ��!� � �(�*� $�$�+�+�,@�A�$�+�+��� � �S�!5� 6�7� ��#�e�*�n���8�D� *�4��1C�1C� D�O��_�,�{�:� � �#� !�I� #�C�K�(I�q�1�q�5��(I�S]�$^�`j� k�I� �33�*�q� �/C� �?�2�K���F�A� �*�,��i� � �?�2�K���F�A� �*�,��k�1A�$(�I�I�e�J�y�.I�$J�M�$�*�*�,�/<��|�,�$��)� �"#�K�!"�J���F�A� %)�I�I�e�J�y�.I�$J�M�$�*�*�,�/<��|�,�$��)� �!�A�"#�K�!*�J� ��.� ��Q���] �#�e�*�n�b "�Y�Y�u�Z�[�'9� :�� � #� #� %�#5�F�<� ��
��Y )Js � 34H �(H c � � || S r35 r )r �chunked_codebaser s r
r zCodeChunker.get_chunk{ s
� � � �-�-r r- )r. r/ r0 r �dictr r �
__classcell__)r7 s @r
r3 r3 $ s �� �-�P�$� P�d.r r3 N)�abcr r r �utilsr r r3 r r r
�<module>r[ s( �� #� !� ��c� �<X.�'� X.r