Team Ai
Apppublic

documentExtractionag051/ExtractDocument

sourceHugging Faceupdated 9mo agoView on Hugging Face
0likes
invoice_post_processing.cpython-313.pyc154 linesDownload Raw Back to __pycache__
1�

2�`i�e����SSKrSSKrSSKJrJrJrJr SSKJrJ	r	J3r4 \R"\RSS9 \R"\5rSSjrSrS	rS5rSrSSjrSS
jrg)�N)�List�Dict�Any�Optional�)�compute_global_bounds�Rect�TextRectz"%(levelname)s:%(name)s:%(message)s)�level�formatc	���[RS5 Uc0nURSS5n6URSS5nURSS5nURSS	5n
URS7/5nU(aUOSSSS
SS.nU(aUOSnU(a[R"U5/nO[R"S5/n[U5n[
XU5n[XU5n[UU5nURU/5n[UUUSU5n[RS[U5SUS35 [UUUUUUUU8UUU
US9n[RS[U5S35 SU0US'U$)a�9Specialized post-processing pipeline for Invoice documents.10 11Fixes applied:12- Multi-line cell text is now ordered top-to-bottom, then left-to-right.13- Rows with invalid Material codes are removed based on multiple regex opt.14 15Args:16    column_headers: Dict mapping field names to header text in document17    anchor_column: Column name used for row detection18    anchor_pattern: Regex pattern string for validating anchor column values19    advanced_options: Dict for special logic options:20        - stop_text: Text marking end of table on last page21        - enable_continuation: Whether to enable continuation column22        - continuation_marker: Text after which continuation starts23z.Running invoice specialized post-processing...�	stop_textz
Total Amount:�enable_continuationF�continuation_column�Description�continuation_markerzRef.:�table_end_markers�
Material CodezQty.z24Unit PricezTotal Net Value)rr�Quantity�25unit_price�total_pricez^\d{8}$rzIdentified z rows based on anchor column '�')�rows�
column_spaces�space_to_lines�cumulative_heights�MATERIAL_CODE_PATTERNS�	ocr_elems�
anchor_columnrrrr�table_boundariesz26Extracted z table rows�table�tables)�logger�info�get�re�compile�_calculate_cumulative_heights�_create_column_spaces�_find_table_boundaries�_assign_lines_to_spaces�_identify_rows�len�_extract_table_rows)r�validation_data�27field_name�anchor_keywords�column_headersr�anchor_pattern�advanced_options�page_dimensions�kwargsrrrrr�COLUMN_HEADERS�
ANCHOR_COLUMNrrrr r�anchor_col_linesr�28table_rowss                         �uC:\Users\poona\OneDrive\Documents\nextlifyai\Document_Automation\Document_Extraction\utils\invoice_post_processing.py�extract_invoice_tablesr<s���8�K�K�@�A����� �$�$�[�/�B�I�*�.�.�/D�e�L��*�.�.�/D�m�T��*�.�.�/D�g�N��)�,�,�-@�"�E��(6�^�(�$��"�(�<�N�&3�M��M��"$�*�*�^�"<�!=��"$�*�*�Z�"8�!9��7��G��*�)�n�U�M�.�i�J[�\��-�Y�
�F�N�&�)�)�-��<������q�!��	�D��K�K�+�c�$�i�[�(F�}�o�UV�W�X�%�
�#�%�-�5��#��/�/�/�)�
�J��K�K�*�S��_�-�[�9�:�")�*� 5�O�H����c�l�0nSn[UR55HnX!U'XSnX$-
nM U$)z;Calculate cumulative page heights for multi-page documents.rr)�sorted�keys)r5r�cumulative_height�page_num�heights     r;r(r(wsL�������?�/�/�1�2��'8�8�$� �*�1�-���#��3�29�r=c�6^�UR5Vs0sHo3S_M nnU(a [SUR555OSnUVs/sH)nURS:XdMURS:XdM'UPM+ nn[30R
S5 [31R
SU35 [32R
S[U5S	35 [33RS345 UHgn[35RSURR5SURS
URSURS
UR3365 Mi SmU4Sjn0n	UR5GHup�SnSn
SnUH'nU"URU5unnUU
:�dM!Un
UnUnM) U(a�U(a�[!URSURUS9XJ'UURR5UU
URS
UR3S.X�'[37R
SU38SURR5SUSU
SURS
URS3
5 M�[39R#SU40SUS35 GM [%SUR555nUR5VVs/sH
unnUbMUPM nnn[41R
S5 [42R
SUS[U5S35 U(av[43R#S S!R'U535 [44R
S"5 UH4n[45R
S#URR5S35 M6 U$s snfs snfs snnf)$a46Create vertical column spaces based on header text positions.47 48Uses a multi-pass matching strategy:491. Exact match (case-sensitive)502. Case-insensitive match513. Partial/contains match (header text in element or element in header)524. Normalized match (strip whitespace, lowercase)53 54Args:55    ocr_elems: List of OCR TextRect elements56    page_dimensions: Dict of {page_num: (width, height)}57    column_headers: Dict mapping column keys to header text patterns58 59Returns:60    Dict mapping column keys to Rect objects representing column spaces61Nc3�*# �UH	upUv� M g7f�N�)�.0�widthrCs   r;�	<genexpr>�(_create_column_spaces.<locals>.<genexpr>�s���O�6N�]�U�&�6N���rr�LINEz=== Column Header Detection ===zLooking for headers: �Found z LINE elements on page 1zPage 1 LINE elements:z  'z' at x=�-�, y=c�x�SRUR5R5R55$)zENormalize text for comparison: lowercase, strip, collapse whitespace.� )�join�lower�strip�split)�texts r;�_normalize_text�._create_column_spaces.<locals>._normalize_text�s)���x�x��62�63��*�*�,�2�2�4�5�5r=c�8>�UR5nUR5nX#:XagUR5UR5:XagT"U5nT"U5nXE:XagXT;agXE;agURU5(agURU5(agg)	z^64Multi-strategy header matching.65 66Returns: (match_type, confidence) where confidence is 0-10067)�exact�d)�case_insensitive�_)�68normalized�Z)�contains_header�P)�
contains_elem�K)�starts_with�F)�
header_starts�A�Nr)rUrT�69startswith)�	elem_text�header_text�70elem_clean�header_clean�	elem_norm�header_normrXs      �r;�
_match_header�,_create_column_spaces.<locals>._match_header�s�����_�_�&�71�"�(�(�*���%�!������!3�!3�!5�5�+�$�J�/�	�%�l�3���#�%��#�*��#�(�����,�,�&��!�!�)�,�,�(�r=)�x1�y1�x2�y2)rl�matched_text�72match_type�73confidence�x_rangeu
✓ Matched 'z' -> 'z' (type=z, conf=z%, x=�)u✗ No match found for column 'z' with header 'rc3�.# �UHocMSv� M
 g7f�NrrG)rH�spaces  r;rJrK�s���Q�%;�E�a�a�%;�s��	z === Header Detection Summary ===�/z columnszMissing columns: �, z/Available LINE texts on page 1 (for debugging):�  - ')r@�max�valuesrB�74block_typer#r$r-�debugrWrUrsrurtrv�itemsr	�warning�sumrS)rr5r2�keyr�max_page_height�elem�page1_linesrq�matched_headers�col_keyrl�75best_match�best_confidence�	best_elemrxry�found_countr~�missing_colsrXs                    @r;r)r)�sW���$+9�*=�*=�*?�@�*?�3�$�Y�*?�M�@�Sb�c�O�o�6L�6L�6N�O�O�hi�O�#��"���=�=�A��	
�"&�/�/�V�";�	
����76�K�K�1�2�77�K�K�'��'7�8�9�78�K�K�&��[�)�*�*B�C�D��L�L�(�)������s�4�9�9�?�?�,�-�W�T�W�W�I�Q�t�w�w�i�t�D�G�G�9�TU�VZ�V]�V]�U^�_�`��6�#�L�O� .� 4� 4� 6����79����	��D�%2�4�9�9�k�%J�"�J�80��O�+�",��'�81� �	� ��)�%)��<�<���<�<�"�	&�M�"� +� )��� 4� 4� 6�(�-�'�l�l�^�1�Y�\�\�N�;�(�O�$�
�K�K���y��y�~�~�/C�/C�/E�.F�G�#��G�O�+<�E�)�,�,��q�QZ�Q]�Q]�P^�^_�a�
�82
�N�N�<�W�I�_�U`�Ta�ab�c�d�?!7�D�Q�]�%9�%9�%;�Q�Q�K�*7�*=�*=�*?�Q�*?�J�C��5�C�*?�L�Q�83�K�K�2�4�84�K�K�&��
�Q�s�>�':�&;�8�D�E�����*�4�9�9�\�+B�*C�D�E����E�F��D��K�K�%��	�	��� 1�2�!�4�5� ����YA����zRs#�N�N�'N�9N�;85N�	Nc
��0nUR5H�unupVSUS.X4'U(dMUnUH�nURU:XdMURS:XdM'UH_n	X�R;dMURU:a:URS-86n[87R
SUSU	SURSU35  M� M� XsUS	'M� U$)88z�89Find the table content boundaries for each page.90Returns dict: {page_num: {'top': y1, 'bottom': y2}}91 92The bottom boundary is determined by finding footer markers.93r)�top�bottomrM�94zPage z: Found footer marker 'z' at y=z, setting table bottom to r�)r�rBr�rWrtr#r$)95rr5r�96boundariesrB�97page_width�page_height�98footer_topr��markers99          r;r*r*s����J�/>�/D�/D�/F�+��+�:��!� 100�101��102!��!�103��D��}�}��(�T�_�_��-F�/�F����*��7�7�Z�/�)-���2��J�"�K�K�%��z�9P�QW�PX�Y.�.2�g�g�Y�6P�Q[�P\�)^�_��0��*4�8��X�&�/0G�2�r=c�F�UR5Vs0sHo"/_M nnUHunURS:waMSupVUR5H$upxUcM104URU5n	X�:�dM"X�pVM& U(dMZUS:�dMbX5R	U5 Mw U$s snf)z;Assign each LINE element to its best-matching column space.rMrir)r@r�r��intersection_pct�append)105rrr�r�	line_elem�106best_space�best_intersectionr��107space_rect�intersections108          r;r+r+*s���)6�);�);�)=�>�)=�#�2�g�)=�N�>��	����6�)��(/�%�109�#0�#6�#6�#8��G��!��%�6�6�y�A�L��/�0<�:�$9��:�+�a�/��&�-�-�i�8�� ���%?s�Bc��[RS[U5SUR35 UHRnURR5nUR
U5n[RSUSU(aSOS35 MT UVs/sH4oBR
URR55(dM2UPM6 nn[RS[U5S35 U(a3[RS	UVs/sHoDRPM sn35 O�[RS110URS35 [RS5 US
SHNn[RSURR5SURSURS35 MP 0/p�UH.nURUR/5RU5 M0 [UR55H�up�[USS9nXup�U(a
X�;aX:SnOUn[U5H�unnURnUS-[U5:aUUS-RnOUn[!SUU
USSS9n[RSU111SUR"SUR35 U	RU112US.5 M� M� U	$s snfs snf)z�113Identify row boundaries based on material code positions.114 115Args:116    table_boundaries: Dict with table content boundaries per page117zTesting z lines against anchor pattern: z  Line 'z' -> �MATCHzno matchrNz valid anchor column valueszValid material lines: zNo lines matched pattern 'rzAvailable anchor column values:N�r�z' (page rPr{c��UR$rF�rt��ls r;�<lambda>� _identify_rows.<locals>.<lambda>bs��1�4�4r=�r�r�rr�ROWg�?)rsrtrurvr�ryzCreated row rect: page=z, x1=z, y1=)rB�row_rect)r#r$r-�patternrWrU�matchr�r�rBrt�118setdefaultr�r?r��	enumerater	rs)�material_code_linesr5r�r �linerWr��valid_material_lines�
page_to_linesrrB�lines�lines_sortedr�r��table_bottom�irtrvr�s                    r;r,r,As����K�K�(�3�2�3�4�4S�T[�Tc�Tc�Sd�e�f�#���y�y��� ���
�
�d�#�����x��v�U�e�7��*L�M�N�$�-��,��
�
�d�i�i�o�o�>O�0P��,����K�K�&��1�2�3�3N�O�P�����,�DX�-Y�DX�D�i�i�DX�-Y�,Z�[�\�	���3�G�O�O�3D�A�F�G����8�:�'���,�D��K�K�%��	�	��� 1�2�(�4�=�=�/��d�g�g�Y�VW�X�Y�-��b�4�$��� � �����3�:�:�4�@�%�"�-�"5�"5�"7�8����e��8��"1�";��119��� <�+�5�h�?�L�&�L� ��.�G�A�t����B��1�u�s�<�(�(�!�!�a�%�(�+�+��"���q�R�J�2�%�\_�`�H��K�K�1�(��5����
�U�S[�S^�S^�R_�`�a��K�K�X�8�D�E�/�9�,�K��Q��.Zs�1K�K�K120c���/nU(dU$[SU55n
[RSU
35 [USS9n[	U5GHtunnUSnUSnURUS5n0nUR
5GH�unn[SU35 UcMS	nU(aUU;aUUS121nURnUU
:Xa^U(aW[URU/5SS9nUH6nUUR;dMURU:XdM'URS-122n O /nURU/5HVnURU:waMURU5S::aM,U(aURU:�aMEURU5 MX /nUH/nURU5nUS:�dMURUU45 M1 /nUH[unn S
n!UH3n"U"ULaM123U"SU:waMU"SRU5n#U#U :�dM1Sn! O U!(dMJURU5 M] /n$U(Ga�UU	:XGa}S
n%X�S-S	Hn&U&SU:XdMSn%[ M U%(Ga�US-n'Sn(S	n)U(a/U'U;a)UU'RSS5n(UU'RS1245n)S	n*X�S-S	Hn&U&SU':XdMU&SRn+ O Sn,U125(aKUHEn-U-RU':XdMU-RS:XdM'U126U-R;dM9U-Rn, O W+b�URU/5H�nURU':waMURU5S::aM,URU+:�aM>U)(aURU):�aMWU127(aU,(aURU,::aMwU$RU5 M� U$(a.[RS[!U$5SUSUSU'SU+S35 O�URU/5HvnURW':waMURU5S::aM,W)(aURU):�aMEU128(aW,(aURU,::aMeU$RU5 Mx U$(a+[RS[!U$5SUSUSW'S3	5 UU$-n.[U.SS9n/SR#SU/55n0[USS9n1U1(ab[%SU155n2[%SU155U-n3[SU155n4[S U155U-n5U2S!U3S!U4U2-129S!U5U3-1303n6OS"n6U0U6S#.UU'GM� URU05RS$S%5R'5n7[RS&US'U7S(35 URU5 GMw [RS)[!U535 U$)*a�131Extract cell values for each table row by finding line elements at row-column intersections.132 133Args:134    stop_text: Text that marks end of table on last page135    enable_continuation: Whether to look for continuation on next page136    continuation_column: Which column may continue to next page137    continuation_marker: Text after which continuation starts138    table_boundaries: Dict with table content boundaries per page (to filter out footer)139c3�*# �UH	oSv� M g7f)rBNrG)rH�rows  r;rJ�&_extract_table_rows.<locals>.<genexpr>�s���!B�T�c�j�/�T�rLzHighest page row number: c�*�USUSR4$)NrBr�r�)�rs r;r��%_extract_table_rows.<locals>.<lambda>�s��a�140�m�Q�z�]�=M�=M�-Nr=r�rBr�rzProcessing column: Nr�c��UR$rFr�r�s r;r�r��s��!�$�$r=r�TFrr�rMrNz continuation lines for z from page z	 to page z (above next row at y=r{z (no more rows on next page)c�\�UR[URS5UR4$r})rB�roundrtrsr�s r;r�r�Gs��q�z�z�5����q�>�1�4�4�@r=rRc3�T# �UHoRR5v� M  g7frF)rWrU�rHr�s  r;rJr�Js��� J�9I�A�������9I�s�&(c�F�[URS5UR4$r})r�rtrsr�s r;r�r�Os��u�Q�T�T�1�~�q�t�t�4r=c3�8# �UHoRv� M g7frF)rsr�s  r;rJr�S����9�'8�!���'8���c3�8# �UHoRv� M g7frFr�r�s  r;rJr�Tr�r�c3�8# �UHoRv� M g7frF)rur�s  r;rJr�Ur�r�c3�8# �UHoRv� M g7frF)rvr�s  r;rJr�Vr�r�r�z0, 0, -1, -1)�value�boundsr��zValidating anchor column 'z141' value: 'rz'Final table row count after filtering: )r�r#r$r?r�r%r��printrvrWrBrtr�r��brea_y1kr�r-rS�minrU)8rrrrrrrrrrrr r:�highest_page_row_number�sorted_rows�row_idxr�rBr��
height_offset�row_datar��col_rect�page_table_bottom�effective_col_rect_y2�anchor_linesr��candidate_linesr��scored_lines�row_intersection�142cell_lines�score�best_row_for_line�	other_row�other_intersection�continuation_lines�is_last_row_on_page�143future_row�
next_page_num�next_page_table_top�next_page_table_bottom�next_row�next_row_y1�marker_line_y2r��all_lines_for_text�all_lines_sorted�	cell_text�cell_lines_sortedrsrtrurvr��anchor_values8                                                        r;r.r.zsU��<�J����!�!B�T�!B�B��144�K�K�+�,C�+D�E�F���#N�O�K�!�+�.�����z�?���z�?��*�.�.�x��;�
���!.�!4�!4�!6��G�X��'��y�1�2����145!%���H�0@�$@�$4�X�$>�x�$H�!�146%-�K�K�!��2�2�y�%�"�&�&�}�b�9�&� ��)�D� �D�I�I�-��=�=�H�4�48�G�G�b�L�1�!�	)�!�O�#�'�'���4���:�:��)���,�,�Q�/�1�4��$����1B�)B���&�&�q�)�5��L�'��#+�#<�#<�T�#B� �#�a�'� �'�'��/?�(@�A�(��J�+���e�$(�!�!,�I� �C�'� � ��,��8� �)2�:�)>�)O�)O�PT�)U�&�)�E�1�,1�)��"-�%�$��%�%�d�+� ,�$"$��"�w�2E�'E�&*�#�"-��k�l�";�J�!�*�-��9�.3�+� �#<�147'�$,�q�L�M�+,�'�-1�*�'�M�=M�,M�.>�}�.M�.Q�.Q�RW�YZ�.[�+�1A�-�1P�1T�1T�U]�1^�.� $�H�&1�A�+�,�&?�148�%�j�1�]�B�*4�Z�*@�*C�*C�K�!�'@�&'�N�*�$-�D�#�}�}�
�=�$�/�/�U[�B[�#6�$�)�)�#C�59�W�W�N�$)�	%.�#�.�!/�!3�!3�G�R�!@�A� �z�z�]�:� (�'�8�8��;�q�@� (� �t�t�{�2� (�5�!�$�$�BX�:X� (�2�~�!�$�$�R`�J`� (�.�5�5�a�8�"A�"*����$�S�);�%<�$=�=U�V]�U^�_)�)1�149�)�M�?�J`�al�`m�mn�p���,�/�/���<���:�:��6�$�#�4�4�Q�7�1�<�$�1�a�d�d�>T�6T�$�.�>�a�d�d�n�F\�$�*�1�1�!�4�=�*����$�S�);�%<�$=�=U�V]�U^�_)�)1�150�)�M�?�Jf�h��",�.@�!@��%�"�@� ��151��� J�9I� J�J�I�!'��4�!��152!��9�'8�9�9���9�'8�9�9�M�I���9�'8�9�9���9�'8�9�9�M�I���4�r�"��R��R��y��2��7�)�<��'��#� �!�H�W��e"7�t �|�|�M�2�6�:�:�7�B�G�M�M�O�����0���z�,��WX�Y�Z�	���(�#�O/�R�K�K�9�#�j�/�9J�K�L��r=)�custom_field_nameNNNNNNrF)rNFNNN)�loggingr&�typingrrrr�geometry_utilsrr	r153�basicConfig�INFO�	getLogger�__name__r#r<r(r)r*r+r,r.rGr=r;�<module>r�s����	�,�,�A�A����'�,�,�/S�T�	�	�	�8�	$��#�������h�V154�~�B"�J�.6�@"������vr=