Team Ai
Datasetpublic

Brunobkr/llama.cpp_AlgMor24_github

ΩFFFΣLLIa • llama.cpp • AlgMor24 ██████╗ ███████╗███████╗███████╗██╗ ██╗ ██╗ █████╗ ██╔═══██╗██╔════╝██╔════╝██╔════╝██║ ██║ ██║██╔══██╗ ██║ ██║█████╗ █████╗ █████╗ ██║ ██║ ██║███████║ ██║ ██║██╔══╝ ██╔══╝ ██╔══╝ ██║ ██║ ██║██╔══██║ ╚██████╔╝██║ ██║ ███████╗███████╗███████╗██║██║ ██║ ╚═════╝ ╚═╝ ╚═╝ ╚══════╝╚══════╝╚══════╝╚═╝╚═╝ ╚═╝ High-Performance LLM / VLM Inference & Autonomous Agentic Ecosystem… See the full description on the dataset page: https://huggingface.co/datasets/Brunobkr/llama.cpp_AlgMor24_github.

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes3.1kdownloads
granite.cpython-313.pyc169 linesDownload Raw Back to __pycache__
1�

2�pzj��H�SSKJr SSKrSSKJrJrJrJr SSKr\(aSSKJ	r	 SSK3JrJrJ
r
Jr SSKJr SSKJr \R&"S	5"S4S\55r\R&"SS
5"SS\55r\R&"S5"SS\55r\R&"SS5"SS\\55r\R&"S5"SS\55r\R&"S5"SS\55r\R&"S5"SS\55rg) �)�annotationsN)�Any�Callable�Iterable�
TYPE_CHECKING)�Tensor�)�MmprojModel�	ModelBase�gguf�logger)�5LlamaModel)�Mamba2Model�GraniteForCausalLM��r^�\rSrSrSr\RRrU4Sjr	\6SU4Sjj5rSrU=r
$)�GraniteModel��'Conversion for IBM's GraniteForCausalLM���>�URRSS5=n(a[R"SU5 [TU]5 URR
S5=n(a2URRU5 [R"SU5 URR
S5=n(a2URRU5 [R"SU5 URR
S5=n(a2URRU5 [R"S	U5 URR
S75=n(a2URRU5 [R"SU5 URR
S5(d URR
S
5(a�[RUR5n[UR 5Vs/sHnSPM nn[#U5Hun	upz nU	S:XaMX�U8'M URR%U5 ggs snf)��Granite uses standard llama parameters with the following differences:9 10- No head_dim support11- New multiplier params:12    - attention_scale13    - embedding_scale14    - residual_scale15- logits_scaling16�head_dimN�.Ignoring head_dim (%s) from config for Granite�attention_multiplier�$gguf: (granite) attention_scale = %s�embedding_multiplier�$gguf: (granite) embedding_scale = %s�residual_multiplier�#gguf: (granite) residual_scale = %s�logits_scaling�!gguf: (granite) logits_scale = %s�spatial_target_layers�deepstack_layer_map�����r)�hparams�popr
�warning�super�set_gguf_parameters�get�gguf_writer�add_attention_scale�info�add_embedding_scale�add_residual_scale�add_logit_scale�Granite4VisionMmprojModel�get_normalized_projector_map�range�block_count�	enumerate�add_deepstack_mapping)�selfr�attention_scale�embedding_scale�residual_scale�logits_scale�normalized_projector_map�_�deepstack_mapping_arr�proj_idx�	llm_layer�	__class__s           ��9/home/userk21/llama.cpp_algmor24_v2/conversion/granite.pyr*� GraniteModel.set_gguf_parameterss�����|�|�'�'�17�D�9�9�8�9��N�N�K�X�V�
��#�%�#�l�l�.�.�/E�F�F�?�F����0�0��A��K�K�>��P�"�l�l�.�.�/E�F�F�?�F����0�0��A��K�K�>��P�!�\�\�-�-�.C�D�D�>�D����/�/��?��K�K�=�~�N��<�<�+�+�,<�=�=�<�=����,�,�\�:��K�K�;�\�J��<�<���3�4�4����8H�8H�I^�8_�8_�'@�']�']�^b�^j�^j�'k�$�16�t�7G�7G�1H�$I�1H�A�R�1H�!�$I�2;�<T�2U�.��.�1��A��q�=��3;�i�0�3V�
���2�2�3H�I�9`��$Is�I��|>�Uup#URS5(dSU;dSU;dSU;ag[TU]	U5$)N�encoder.�image_�layerwise_projectors�spatial_projectors��18startswithr)�filter_tensors��cls�item�name�genrBs    �rCrM�GraniteModel.filter_tensors>sF����	��
�O�O�Z�)�)��4��%��-�#�t�+���w�%�d�+�+���rP� tuple[str, Callable[[], Tensor]]�return�'tuple[str, Callable[[], Tensor]] | None)�__name__�19__module__�__qualname__�__firstlineno__�__doc__r�20MODEL_ARCH�GRANITE�21model_archr*�classmethodrM�__static_attributes__�
__classcell__�rBs@rCrrs0���1����(�(�J�&J�P�22,��23,rTr�GraniteMoeForCausalLM�GraniteMoeSharedForCausalLM��h^�\rSrSrSr\RRrU4Sjr	SU4Sjjr24SrU=r$)�GraniteMoeModel�L�*Conversion for IBM's GraniteMoeForCausalLM���>�[TU]5 URRS5=n(a3URRU5 [R"SU5 gg)�[GraniteMoeShared uses GraniteMoe parameters plus the following:25- shared_intermediate_size26�shared_intermediate_size�8gguf: (granitemoeshared) shared_feed_forward_length = %sN)r)r*r&r+r,�%add_expert_shared_feed_forward_lengthr
r.)r8�shared_feed_forward_lengthrBs  �rCr*�#GraniteMoeModel.set_gguf_parametersQsW���	��#�%�)-���)9�)9�:T�)U�U�%�U����B�B�C]�^��K�K�R�Tn�o�VrT�#�@># �URS5(a�URSnURSSU-:XdS5eURUSS9upV[R27"XUR
[RRU5U5Shv�N [R28"XUR
[RRU5U5Shv�N g[URRS55nURS	5(GagURS29nURSSU-:XdS5eURUSS9upVU(a�[R30"XUR
[RRU5U5Shv�N [R31"XUR
[RRU5U5Shv�N g[R32"XUR
[RRU5U5Shv�N [R33"XUR
[RR U5U5Shv�N gU(d_URS5(aI[R34"XUR
[RR"U5U5Shv�N g[$TU]XU5Shv�N gGNoGN*GNdGNN�N�N.N7f)
�In modeling_granitemoe, the JetMoe implementation of parallel experts35is used. This essentially merges w1 and w3 into a single tensor with 2x36the hidden size that is then split during forward. To keep compatibility37with existing mixtral support, we pull them apart here.38�$block_sparse_moe.input_linear.weight�intermediate_size�������4Merged FFN tensor size must be 2 * intermediate_size��dimN�num_local_experts�shared_mlp.input_linear.weightrp�;Merged FFN tensor size must be 2 * shared_intermediate_size�shared_mlp.output_linear.weight)�endswithr&�shape�splitr�modify_tensors�format_tensor_namer�MODEL_TENSOR�FFN_GATE_EXP�39FFN_UP_EXP�boolr+�FFN_GATE_SHEXP�FFN_UP_SHEXP�FFN_GATE�FFN_UP�FFN_DOWNr))	r8�40data_torchrQ�bid�ffn_dim�gate�up�has_expertsrBs	        �rCr��GraniteMoeModel.modify_tensorsZs������=�=�?�@�@��l�l�#6�7�G��#�#�B�'�1�w�;�6�n�8n�n�6�!�'�'��R�'�8�H�D� �/�/��D�<S�<S�TX�Te�Te�Tr�Tr�tw�<x�z}�~�~�~� �/�/��$�:Q�:Q�RV�Rc�Rc�Rn�Rn�ps�:t�vy�z�z�z���4�<�<�+�+�,?�@�A���=�=�9�:�:��l�l�#=�>�G��#�#�B�'�1�w�;�6�u�8u�u�6�!�'�'��R�'�8�H�D��$�3�3�D�t�?V�?V�W[�Wh�Wh�Ww�Ww�y|�?}�@C�D�D�D�$�3�3�D�d�>U�>U�VZ�Vg�Vg�Vt�Vt�vy�>z�|�A�A�A�� �/�/��D�<S�<S�TX�Te�Te�Tn�Tn�ps�<t�vy�z�z�z� �/�/��$�:Q�:Q�RV�Rc�Rc�Rj�Rj�lo�:p�ru�v�v�v���t�}�}�-N�O�O� �/�/��$�BY�BY�Z^�Zk�Zk�Zt�Zt�vy�Bz�|�A�
A�
A���7�)�*�C�@�@�@�-
�z�D�A�z�v�
A�	A�s��BL�L41�AL�#L
�$CL�,L�-AL�4L�5AL�=L�>AL�L�A%L�+L�,L�L�L�
L�L�L�L�L�L�LrU�r�rrQ�strr��42int | NonerX�Iterable[tuple[str, Tensor]])
rZr[r\r]r^rr_�GRANITE_MOErar*r�rcrdres@rCrjrjLs'���4����,�,�J�p�!A�!ArTrj�GraniteSwitchForCausalLM��^�\rSrSrSr\RRrSr	U4Sjr43U4SjrS44SjrSSSjjr
S
SjrS	rU=r$)�GraniteSwitchModel�~ژDense, all-attention Granite with N per-token embedded LoRA adapters, stacked45over the adapter dim with a zero adapter at slot 0 (N = num_adapters + 1).F���>�[TU]"U0UD6 URS-46Ul[R"UR47UR5Ul[URS5Ul	[URS5Ul48URS-Ul[URS5n[URS5nURRS5=(d4 URRS5=(d URSU-nX0l
X@l[U5UlX0R-UlX@R-Ulg)	Nr	�num_adapters�
max_lora_rank�num_attention_heads�num_key_value_heads�projection_head_dimr�hidden_size)r)�__init__r5r�get_tensor_name_mapra�49tensor_map�intr&�_n_adapters�_max_lora_rank�_n_slotsr+�_n_head�50_n_kv_head�	_head_dim�_q_size�_kv_size)r8�args�kwargs�n_head�	n_kv_headrrBs      �rCr��GraniteSwitchModel.__init__�s$���
���$�)�&�)��+�+�a�/����2�2�4�?�?�D�DT�DT�U����t�|�|�N�;�<���!�$�,�,��"?�@����(�(�1�,��
��T�\�\�"7�8�9������%:�;�<�	��L�L���2�3�
7��|�|���51�+�
7����]�+�v�5�	�52��#���X�������.���!�N�N�2��
rT��>�[TU]5 URRS5(dURRS5 URR
UR5 URRUR5 URRURS5 URRURS5 [URRSS55nURRU5 [R"SURURUR U5 g)Nrr�adapter_token_ids�adapter_substitute_token_ids�control_token_gaing.@�Pgguf: (graniteswitch) num_adapters=%s max_lora_rank=%s n_slots=%s router_gain=%s)r)r*r&r+r,�add_expert_used_count�add_adapter_countr��add_adapter_lora_rankr��add_adapter_token_ids_activate� add_adapter_token_ids_substitute�float�add_adapter_router_gainr
r.r�)r8�router_gainrBs  �rCr*�&GraniteSwitchModel.set_gguf_parameters�s ���
��#�%��|�|��� 3�4�4����2�2�1�5����*�*�4�+;�+;�<����.�.�t�/B�/B�C����7�7����EX�8Y�Z����9�9�$�,�,�Ge�:f�g��D�L�L�,�,�-A�4�H�I�����0�0��=����f�hl�hx�hx�z~�{N�{N�PT�P]�P]�_j�	krT���URS5n[R"USS5n[R"X2/SS9R	5$�Nr	rr})�squeeze�torch�53zeros_like�cat�54contiguous)r8�data�a�zeros    rC�_lora_a�GraniteSwitchModel._lora_a�sA���L�L��O������"�1��&���y�y�$���*�5�5�7�7rT�55�F�URS5nUbL[R"[URS5Vs/sHo@RX4X"5PM snSS9n[R"USS5n[R"XS/SS9R5$s snfr�)	r�r��stackr4r��permuter�r�r�)r8r��permute_n_head�b�ir�s      rC�_lora_b�GraniteSwitchModel._lora_b�s����L�L��O���%����Y^�_`�_f�_f�gh�_i�Yj�k�Yj�TU�\�\�!�$��O�Yj�k�qr�s�A�����"�1��&���y�y�$���*�5�5�7�7��ls�B�#��# �[RnURS5SnURS5(dURS5(dUS;agSU;Ga�UR	S5(a�URUR56URUR/SS	9upgnURX`RUR5nURXpRUR5n[R"XgU/SS	9n	URURU5U	4v� gS57U;ai[URSS5S5n58UR UR"UR$S.U59nURX�S
S9UR'U54v� gSU;a�[URSS5S5n60UR UR4UR"UR4UR$S4S.U61up�URX�SS9UR)X54v� g[+SU35eSU;a�UR	S5(a!URUR,U5U4v� gUR	S5(a/URUR,US
S9UR'U54v� gUR	S5(a/URUR,USS9UR)U54v� g[+SU35eSU;GaOUR.Sn
UR	S5(aTURX�/SS	9up�URUR0U5U4v� URUR2U5U4v� gS62U;a^[URSS5S5n63UR0UR2S.U64nURX�S
S9UR'U54v� gSU;a^[URSS5S5n65UR0UR2S.U66nURX�SS9UR)U54v� g[+SU35eSU;a�UR	S5(a!URUR4U5U4v� gUR	S5(a/URUR4US
S9UR'U54v� gUR	S5(a/URUR4USS9UR)U54v� g[+SU35eUbFSU;a@SU;dSU;a4SU;aUR6OUR8nURX�5U4v� gUS;a URUR:5U4v� gUS ;a URUR<5U4v� gUS!:Xag[+S"U<S#US$35e7f)%N�.r%�
model.switch.�switch.)r��control_to_substitute_lut�self_attn.qkv_proj�base_layer.weightrr}�lora_A_slices.r	)rr	r{�.lora_a)�suffix�lora_B_slices.�.lora_b�Unexpected qkv_proj tensor: �self_attn.o_proj�lora_A�lora_B�Unexpected o_proj tensor: �shared_mlp.input_linearrp)rr	�+Unexpected shared_mlp.input_linear tensor: �shared_mlp.output_linear�,Unexpected shared_mlp.output_linear tensor: �.layers.�input_layernorm�post_attention_layernorm)�model.embed_tokens.weight�embed_tokens.weight)�model.norm.weight�norm.weight�lm_head.weight� graniteswitch: unhandled tensor � (bid=�))rr�r�rLr�r�r�r�r�r�r�r�r��ATTN_QKVr��rsplit�ATTN_Q�ATTN_K�ATTN_Vr�r��67ValueError�ATTN_OUTr&r�r�r��	ATTN_NORM�FFN_NORM�68TOKEN_EMBD�OUTPUT_NORM)r8r�rQr��T�bare�q�k�v�fused�slot�key�ph�ffnr�r�s                rCr��!GraniteSwitchModel.modify_tensors�s���������z�z�#��r�"���O�O�O�,�,����	�0J�0J��I�I���4�'��}�}�0�1�1�$�*�*�D�L�L�$�-�-����+W�]^�*�_���a��L�L��L�L�$�,�,�?���L�L��O�O�T�_�_�E���	�	�1��)��3���.�.�q�z�z�3�?��G�G���4�'��4�;�;�s�A�.�q�1�2���(�(�q�x�x�A�H�H�=�d�C���.�.�s�	�.�J�D�L�L�Yc�Ld�e�e���4�'��4�;�;�s�A�.�q�1�2�����$�,�,�/����$�/�/�2����$�'���	���69�.�.�s�	�.�J�D�L�L�Yc�Lh�i�i���;�D�6�B�C�C���%��}�}�0�1�1��.�.�q�z�z�3�?��L�L���}�}�X�&�&��.�.�q�z�z�3�y�.�Q�SW�S_�S_�`j�Sk�l�l���}�}�X�&�&��.�.�q�z�z�3�y�.�Q�SW�S_�S_�`j�Sk�l�l���9�$��@�A�A�$��,��,�,�9�:�C��}�}�0�1�1�%�+�+�S�J�A�+�>����.�.�q�z�z�3�?��F�F��.�.�q�x�x��=�r�B�B���4�'��4�;�;�s�A�.�q�1�2���*�*����2�4�8���.�.�s�	�.�J�D�L�L�Yc�Ld�e�e���4�'��4�;�;�s�A�.�q�1�2���*�*����2�4�8���.�.�s�	�.�J�D�L�L�Yc�Ld�e�e���J�4�&�Q�R�R�%��-��}�}�0�1�1��.�.�q�z�z�3�?��L�L���}�}�X�&�&��.�.�q�z�z�3�y�.�Q�SW�S_�S_�`j�Sk�l�l���}�}�X�&�&��.�.�q�z�z�3�y�.�Q�SW�S_�S_�`j�Sk�l�l���K�D�6�R�S�S��?�z�T�1���%�)C�t�)K�!2�d�!:�!�+�+��70�71�C��*�*�3�4�j�A�A���G�G��*�*�1�<�<�8�*�E�E���7�7��*�*�1�=�=�9�:�F�F���#�#���;�D�8�6�#��a�P�Q�Q�s�W$W&)72r�r�r�r�r�r�r�r�r5r�)r�rrXr)N)r�rr�r�rXrr�)rZr[r\r]r^rr_�GRANITE_SWITCHra�undo_permuter�r*r�r�r�rcrdres@rCr�r�~sB���R����/�/�J��L�3�.
k�8�8�_R�_RrTr��GraniteMoeHybridForCausalLM�BambaForCausalLM��^�\rSrSrSr\RRrSr	U4Sjr73SrSSjrSSjr
SrS	rS74rU=r$)
�GraniteHybridModeli�vGraniteHybrid is a hybrid SSM + Attention model that uses Mamba2 SSM75layers and optionally uses MoE w/ a shared expertT���>�S/Ul[TU]"U0UD6 UR5Ul[UR5Vs/sHnX0R;dMUPM snUlUR(d�URSS/SS9nU(a[RRO[RRnXPl
[RUURlURR#5 URURSS3SS	/5UlURS76S/5UlURSS
/5UR$-Ulgs snf)N�mamba�num_experts_per_tok�num_experts_per_tokenT��optionalrr�r��d_model�n_groups�77num_groups�expand�	num_heads)�hparam_prefixesr)r��get_attn_layers�_attn_layersr4r5�_ssm_layers�find_hparamrr_r�r`ra�MODEL_ARCH_NAMESr,�arch�add_architecturer'�n_group�d_inner)r8r�r�r�r��new_archrBs      �rCr��GraniteHybridModel.__init__%sW���!(�y���
���$�)�&�)�!�0�0�2����T�-�-�.�78�.�!��)�)�)�
�.�79�������*�*�,A�CZ�+[�fj�*�k�K�����+�+����'�'�
�80'�O�$(�$9�$9�(�$C�D���!����-�-�/��'�'�D�,@�,@��,C�+D�I�)N�P]�_h�(i�j����'�'��\�(B�C����'�'��;�(?�@�4�<�<�O����781s�E1�E1����URRS5=n(a'[U5VVs/sHup#US:XdM
UPM snn$URRS/5nU(d|URRS5nU(dS5eURRS5nUcS5e[UR5Vs/sHnX%-U:XdM
UPM nnU$s snnfs snf)N�layer_types�	attention�attn_layer_indices�attn_layer_period�3Didn't find attn_layer_indices or attn_layer_period�attn_layer_offset�4No attention layer offset set with attn_layer_period)r&r+r6r4r5)r8r:r��typ�attn_layers�attn_period�attn_offsets       rCr-�"GraniteHybridModel.get_attn_layersLs����,�,�*�*�=�9�9�;�9� )�+� 6�� 6�f�a��+�%�� 6��
��l�l�&�&�';�R�@����,�,�*�*�+>�?�K��U� U�U�;��,�,�*�*�+>�?�K��*�b�,b�b�*� ��!1�!1�2��2�a��?�k�1��2�
�����!��s�C+�C+�C1�!C1��^�/nURHmURU4SjU55 M  [U5U-n[R"X/UQ70UD6$)N�3�H># �UHnSRTU/5v� M g7f)r>N)�join)�.0r�pfxs  �rC�	<genexpr>�1GraniteHybridModel.find_hparam.<locals>.<genexpr>ds(������A����#�q��"�"��s�")r,�extend�listrr0)r8�keysr�r��prefixedrLs     @rCr0�GraniteHybridModel.find_hparamas\������'�'�C��O�O����
�(�82�D�z�H�$���&�&�t�C�D�C�F�C�CrT�#�# �URS5(dSU;a[RXX#5Shv�N gX0R;a [R"XX#5Shv�N gX0R83;a[RXX#5Shv�N g[R"XX#5Shv�N gN�NUN)N7f)Nrx�84shared_mlp)r�rjr�r/rr.r)r8r�rQr�s    rCr��!GraniteHybridModel.modify_tensorsks�����M�M�@�A�A��t�#�&�5�5�d��R�R�R���"�"�"�"�1�1�$�D�N�N�N��
�%�%�
%�&�5�5�d��R�R�R���+�+�D�d�H�H�H�
S�85
O�
S�H�sE�6C�B;�.C�'B=�(-C�B?�C�5C�6C�=C�?C�C���[RU5 URRUR	SS/55 URRUR	/SQ55 URR
UR5 URRUR5 URRUR	SS/55 UR	SS/5n[UR5Vs/sHo"UR;aUOSPM nnURRS	5=n(aURR!U5 URR#U5 S86URS;=(d UR$(+nURR'U5 U(dURR)S5 UR	S
/SS9=(d SnURRS5S;dS5eURU-S:XdSURSU35egs snf)�GThis method merges params from both parents and some that are87specific to this model. The result is some duplication of how the params88get set. The following warnings are expected during conversion:89 90WARNING:Duplicated key name 'granitehybrid.attention.head_count_kv'91WARNING:Duplicated key name 'granitehybrid.context_length'92�conv_kernel�d_conv)�93state_size�d_state�	state_dim�ssm_state_size�n_headsr+r��	n_head_kvr�attn_rotary_embr�
architecturesi�d_headTr%�@�94hidden_act)N�silu�Only SILU activation supported�SSM inner size � not a multiple of head dim N)rjr*r,�add_ssm_conv_kernelr0�add_ssm_state_size�add_ssm_group_countr4�add_ssm_inner_sizer5�add_ssm_time_step_rankr4r5r.r&r+�add_rope_dimension_count�add_head_count_kvr/�add_rope_scaling_finetuned�add_context_length)r8�
head_count_kvr��head_count_kv_vec�rope_dim�use_roperes       rCr*�&GraniteHybridModel.set_gguf_parameters|s��	�+�+�D�1�	
���,�,�T�-=�-=�}�h�>W�-X�Y����+�+�D�,<�,<�=u�,v�w����,�,�T�\�\�:����+�+�D�L�L�9�	
���/�/��0@�0@�)�[�AY�0Z�[��(�(�*?��)M�N�
�DI�$�JZ�JZ�D[�95�D[�q�$�"3�"3�3�M��:�D[�	�96��|�|�'�'�(9�:�:�8�:����5�5�h�?����*�*�+<�=�
�$�,�,��"?�?�
$��#�#�#�	�	
���3�3�H�=�����/�/��6��!�!�8�*�t�!�<�B����|�|����-��?�a�Aa�a�?��|�|�f�$��)�o�_�T�\�\�N�Jf�gm�fn�+o�o�)��'97s�I��t�UR(aSOSURS'[R"U5 g)N�r	�pad_vocab_size_multiple)r/r&r�	set_vocab�r8s rCr~�GraniteHybridModel.set_vocab�s)��7;�7G�7G�!�Q����.�/����d�#rT)r.r/r5r'r,rar4)rQ�
Iterable[str]rXrr�)rZr[r\r]r^rr_�GRANITE_HYBRIDrarr�r-r0r�r*r~rcrdres@rCrrsE���9����/�/�J��L�%P�N�*D�I�"(p�T$�$rTr�%GraniteSpeechForConditionalGeneration��^�\rSrSr%SrSrSrS\S'S
SjrU4Sjr	U4S	jr98\SU4S99jj5rSU4Sjjr
SrU=r$)�GraniteSpeechMmprojModeli�FTN�list[dict[str, Tensor]] | None�_batch_norm_tensors��8�URRS5$)N�encoder_config)�
global_configr+rs rC�get_audio_config�)GraniteSpeechMmprojModel.get_audio_config�s���!�!�%�%�&6�7�7rT��>�URceURnUSUS'USUS-US'USUS'USUS'[TU]	5 URR	[100RR5 URRUS	5 URRS1015 URRUS5 URRUS5 URRUS
5 URnURRUS5 URRUS5 URR!USS5 g)N�102hidden_dimr��feedforward_multryr+r��103num_layers�num_hidden_layers�	input_dim��h㈵��>�context_size�conv_kernel_size�max_pos_emb�window_size�downsample_rate�projector_config)�
hparams_audior)r*r,�add_clip_projector_typer�VisionProjectorType�GRANITE_SPEECH�add_audio_num_mel_bins�!add_audio_attention_layernorm_eps�add_audio_chunk_size�add_audio_conv_kernel_size�add_audio_max_pos_embr��add_audio_projector_window_size�#add_audio_projector_downsample_rate�add_audio_projector_head_count)r8r��prBs   �rCr*�,GraniteSpeechMmprojModel.set_gguf_parameters�sg����!�!�-�-�-������\�?��-��!"�<��1�5G�3H�!H��104��#$�[�>��105� �!"�<���106��
��#�%����0�0��1I�1I�1X�1X�Y����/�/��+��?����:�:�4�@����-�-�a��.?�@����3�3�A�6H�4I�J����.�.�q��/?�@��������8�8��=�9I�J����<�<�Q�?P�=Q�R����7�7��:L�8M�Nc�8d�erT��>�SU;dSU;a&SU;a SU;a[RR$[TU]XX45$)N�encoder�	projector�.conv�.weight)r�GGMLQuantizationType�F32r)�tensor_force_quant)r8rQ�new_namer��n_dimsrBs     �rCr��+GraniteSpeechMmprojModel.tensor_force_quant�sD�������t� 3��$��9��#4��0�0�4�4�4��w�)�$�#�F�FrT��D>�Uup#SU;dSU;ag[TU]U5$)N�attention_dists�num_batches_tracked)r)rMrNs    �rCrM�'GraniteSpeechMmprojModel.filter_tensors�s-����	����$�(=��(E���w�%�d�+�+rT�#��># �SU;Ga*SU;Ga#URc*[UR5Vs/sHn0PM snUlUceXRUU'[URU5S:agSUS3nURUUS3nURUUS3nURUUS3nURUUS3n	S	n107U[R108"X�-5-nXxU--109n[TU]USUS1103U5Shv�N [TU]USUS3U5Shv�N gSU;acURS
SS9up�[TU]X�RSS5U5Shv�N [TU]X�RSS5U5Shv�N gSU;dSU;aJURS5(a4URS:Xa$URS
S:XaURS
5nSU;aJURS5(a4URS:Xa$URSS:XaURS5n[TU]XU5Shv�N gs snfGNPGN6N�N�N7f)N�111batch_norm�encoder.layers.��.conv.batch_normr��.bias�
.running_mean�.running_varr��.conv.batch_norm.weight�.conv.batch_norm.bias�.attn.to_kv.weightr{rr}�to_kv�to_k�to_v�up_conv�	down_conv�r	�112depth_conv)r�r4r5�lenr��sqrtr)r��chunk�replacer��ndimr�r�)r8r�rQr�r>�prefix�weight�bias�running_mean�running_var�epsr�r��k_weight�v_weightrBs               �rCr��'GraniteSpeechMmprojModel.modify_tensors�s������4��$5��$=��'�'�/�8=�d�>N�>N�8O�+P�8O�1�B�8O�+P��(��?�"�?�2<�$�$�S�)�$�/��4�+�+�C�0�1�A�5��&�s�e�+;�<�F��-�-�c�2�f�X�W�3E�F�F��+�+�C�0�F�8�5�1A�B�D��3�3�C�8�F�8�=�9Q�R�L��2�2�3�7�6�(�,�8O�P�K��C�����K�$5�6�6�A��a�'�'�A��w�-�a�?�3�%�G^�1_�ad�e�e�e��w�-�a�?�3�%�G\�1]�_b�c�c�c���4�'�!+�!1�!1�!��!1�!;��H��w�-�h���W�f�8U�WZ�[�[�[��w�-�h���W�f�8U�WZ�[�[�[�������!4�$�-�-�	�:R�:R����!�#�113�(8�(8��(;�q�(@�'�/�/��2�114��4��D�M�M�)�$<�$<����!�#�115�(8�(8��(;�q�(@�'�/�/��2�116��7�)�*�C�@�@�@��?,Q�
f�c�117
\�[�	A�sg�3I.�I�CI.�I"�I.�/I%�0AI.�0I(�1'I.�I*�B>I.�I,�I.�%I.�(I.�*I.�,I.)r�)rX�dict[str, Any] | NonerVr�)rZr[r\r]�has_vision_encoder�has_audio_encoderr��__annotations__r�r*r�rbrMr�rcrdres@rCr�r��sL�������:>��7�>�8�f�,G��,��,�#A�#ArTr��)GraniteSpeechPlusForConditionalGeneration��4^�\rSrSrSrSrSrU4SjrSrU=r	$)�GraniteSpeechPlusMmprojModeli��YConversion for GraniteSpeechPlus - extends GraniteSpeech with feature layer concatenationFT�118�>�URce[TU]	5 URRS5=n(a�URRU5 [R"SU35 URSnU[U5S--nURSSnXC:wa[SUSUS	US119U35egg)N�cat_hidden_layers�gguf: audio feature_layers = r�r	r��encoder_hidden_size�Projector encoder_hidden_size (�2) does not match expected concatenated dimension (�). Expected: hidden_dim (� ) * (len(feature_layers) + 1) = )r�r)r*r+r,�add_audio_feature_layersr
r.r�r�r)r8�feature_layersr��expected_dim�
projector_dimrBs     �rCr*�0GraniteSpeechPlusMmprojModel.set_gguf_parameterss�����!�!�-�-�-�
��#�%�"�/�/�3�3�4G�H�H�>�H����5�5�n�E��K�K�7��7G�H�I��+�+�L�9�J�%��^�)<�q�)@�A�L� �.�.�/A�B�CX�Y�M��,� �5�m�_�E8�8D�~�F-�-7�L�8X�Ye�Xf�h���-�IrTrU)120rZr[r\r]r^r�r�r*rcrdres@rCr�r��s���c������rTr��&Granite4VisionForConditionalGeneration��z^�\rSrSrSrSr\S121Sj5rU4SjrU4Sjr	\122SU4Sjj5rSU4SjjrS	r
U=r$)
r2iTF����URS/5nURS/5nUSSnUSSn/nU(aH[[U55H0unupxUS:aXG-nUS:aX8-nURXxSU45 M2 U(aDURSS	5n	U	S:aXI-n	[U5Hup�URX�S123U12445 M [	[USS95$)
��Normalize both deepstack and spatial projector maps to the form:125(vision_layer, llm_layer, <type>, type_index)126 127This is then used to populate the following mappings:128- vision_feature_layers (mmproj hparam): ordered list of all129  vision_layer values where order corresponds with the order of the130  stacked projector tensors131  NOTE: Values may appear multiple times for spatial projectors132- tensor_prefix_map (mmproj tensors): mapping from tensor prefixes to133  the index of the corresponding projector in the stacked tensors134- deepstack_layer_arr (llm hparam): per-text-layer array indicating135  which input vision feature should be injected at that layer136  (-1 if none)137 138Output: (vision_layer, llm_layer, <type>, type_index)139r$r#�text_configr��
vision_configr�	layerwise�spatial_vision_layerr%�spatial���US$)Nr	rU)�entrys rC�<lambda>�HGranite4VisionMmprojModel.get_normalized_projector_map.<locals>.<lambda>Ds���a�rT)r)r+r6�sorted�appendrP)r��
deepstack_map�spatial_layers�
n_text_layers�n_vision_layersr=�
deepstack_idx�vision_layerrAr��spatial_idxs           rCr3�6Granite4VisionMmprojModel.get_normalized_projector_map s��$&�)�)�*?��D�
�&�*�*�+B�B�G��%�m�4�5H�I�
�'��8�9L�M��#%� ��<E�f�]�F[�<\�8�
�8���!�#�#2�#A�L��q�=� -� 9�I�(�/�/��+�Wd�0e�f�=]��#0�#4�#4�5K�R�#P� �#�a�'�'6�'M�$�*3�N�*C�&��(�/�/�1E�R[�]h�0i�j�+D��F�3�:P�R�S�SrT�140�>�[T	U]"U0UD6 URUR5n[	U5Ul[
U5VVVVs0sHunu pVnSUSU3U_M snnnnUlUVVs/sH	un  oXPM snnUlUVVVs/sHu pVnUS:XaUOSPM snnnUl	gs snnnnfs snnfs snnnf)N�model.�_projectors.r�r%)141r)r�r3r�r��_n_projr6�_tensor_prefix_map�_vision_feature_layers�_spatial_offsets)142r8r�r�r=r@r>�	proj_type�type_idxrrBs143         �rCr��"Granite4VisionMmprojModel.__init__Fs����
���$�)�&�)�#'�#D�#D�T�EW�EW�#X� ��3�4���:C�C[�9\�#144�9\�5��5�1�a�H��Y�K�|�H�:�6��@�9\�#145���Rj�&j�Qi�8M��a��A�|�Qi�&j��#�.F�!146�-E�)��1��"�Y�.�H�B�6�-E�!147����#148��'k��!149s�
B6150�6B>�C��0>�URce[T151U]	5 URR	[152RR5 URRURRSS55 URRS5 URRURRSS55 URSnURS5n[U5S:Xd153S	U35eUVs/sHn[!U5PM snupEURR#U5 URR%U5 URR'UR(5 URR+UR,5 URRS1545=n(a<UVVV	s/sHupxX�4Ho�PM M nnnn	URR/U5 ggs snfs sn	nnf)N�layer_norm_epsg�����ư>T�155image_sizei�r��/r{�!Invalid 'downsample_rate' value: �image_grid_pinpoints)�hparams_visionr)r*r,r�rr��GRANITE4_VISION�"add_vision_attention_layernorm_epsr&r+�add_vision_use_gelu�add_vision_preproc_image_sizer�r�r�r��add_vision_projector_query_side� add_vision_projector_window_side�add_vision_feature_layersr�add_vision_spatial_offsetsr�add_vision_image_grid_pinpoints)r8�ds_rate�ds_partsr��156query_side�window_side�	pinpoints�h�w�valrBs          �rCr*�-Granite4VisionMmprojModel.set_gguf_parametersUs�����"�"�.�.�.�
��#�%����0�0��1I�1I�1Y�1Y�Z�	
���;�;�D�L�L�<L�<L�M]�_c�<d�e����,�,�T�2�	
���6�6�t�|�|�7G�7G��VY�7Z�[��$�$�%6�7���=�=��%���8�}��!�P�%F�w�i�#P�P�!�3;�"<�8�a�3�q�6�8�"<��157����8�8��D����9�9�+�F�	
���2�2�4�3N�3N�O�	
���3�3�D�4I�4I�J��*�*�.�.�/E�F�F�9�F�)2�E����q�f�s��f���I�E����<�<�Y�G�G��#=��Fs�H�H��d>�Uup#SU;dURS5(ag[TU]	U5$)N�vision_model.head�lm_headrK)rOrPrQr>rBs    �rCrM�(Granite4VisionMmprojModel.filter_tensorsvs3�������4�'�4�?�?�9�+E�+E���w�%�d�+�+rT�#�j># �SnURR5HupVURU5(dMUn O UGbJ[[R158"SU55nUVs/sHn[
URS55PM n	n[U	5S:�a[U	5S::dS5eUn159[U	5S:XaAUSUSRS5S[U1605-X'SRS5SS-nOqUn161USUSRS5SX'SRS5SUSRS5S-[U1625-X'SRS5SS-n[TU]1XU1635Shv�N g[TU]1XU5Shv�N gs snfN#N7f)N�\.([0-9]+)\.r	r{�6Must have at least 1 and at most 2 ids in tensor namesr)
r�itemsrLrP�re�finditerr��groupr��spanr�r)r�)
r8r�rQr��
projector_idxr�r@�164id_matches�m�all_ids�new_bidr�rBs
            �rCr��(Granite4VisionMmprojModel.modify_tensors}s�����
� $� 7� 7� =� =� ?��F����v�&�&� (�
��!@��$��b�k�k�/�4�@�A�J�0:�;�165�1�s�1�7�7�1�:��166�G�;��w�<�1�$��W���):�t�<t�t�:�#�G��7�|�q� �� 9��A��!3�!3�A�!6�q�!9�:�S��\�I�D�\]�Q^�Qc�Qc�de�Qf�gh�Qi�Qj�Lk�k��'��� 9��A��!3�!3�A�!6�q�!9�:�T�Q�-�BT�BT�UV�BW�XY�BZ�[e�fg�[h�[m�[m�no�[p�qr�[s�=t�t�wz�|C�xD�D�GK�WX�LY�L^�L^�_`�La�bc�Ld�Le�Gf�f���w�-�j�G�L�L�L���7�)�*�C�@�@�@��<�
M�@�s5�6F3�.F3�+#F*�C=F3�F/�F3�$F1�%F3�1F3)rrrr)r��dictrX�list[tuple[int, int, str, int]]rVr�)rZr[r\r]r�r��staticmethodr3r�r*rbrMr�rcrdres@rCr2r2sP��������#T��#T�J
167�H�B�,��,�A�ArTr2)�168__future__rr=�typingrrrrr�r�baser169rrr
�llamarr"r�registerrrjr�rr�r�r2rUrTrC�<module>rOs`��"�	�9�9����6�6������(�)�7,�:�7,�*�7,�t���+�-J�K�.A�l�.A�L�.A�b���.�/�\R��\R�0�\R�~���1�3E�F�J$��o�J$�G�J$�Z���;�<�OA�{�OA�=�OA�d���?�@��#;��A��6���<�=�~A��~A�>�~ArT
Brunobkr/llama.cpp_AlgMor24_github · Team Ai