Maaac/CodeLLaMA-Linux-BugFix
08
1�
2 �nh�3 � �6 � d dl Z d dlmZmZ d dlmZ d dlmZ d dlZd dlZd dl Z d dl4Z5dZ dZ dZ
dZd Zd6Ze j" j% � rd nd Z e j( e j* j- e
� d
�� ed� ej0 e � Z ej0 e e j4 d�� Zej9 � ed� ede d�� Z g Z ed� ee � D ]� Z e d Z e d Z! ee dd
e�� jE e� Z# e jH � 5 ejJ d3i e#�edd ejL ejN d ���Z(ddd� ejS e(d d
� � Z*e jW e e!jY � e*jY � d �� �� e-e
d d!�"� 5 Z. ej^ e e.d#�$� ddd� ed%e
� �� e-ed d!d&�'� 5 Z. ej` e.g d ��(� Z1e1je � e1jg e � ddd� ed)e� �� ed*� e7jh d+� Z5 e8jh d,� Z6e D � cg c] } | d- �� c} Z7e D � cg c] } | d. �� c} Z8e5js e7e8D �cg c] }|g�� c}�/� Z:e6js e7e8�/� Z; ed0� ed1e:� ed2 ejx e;d#�$� � y# 1 sw Y ���xY w# 1 sw Y ��-xY w# 1 sw Y ��xY wc c} w c c} w c c}w )4� N)�
AutoTokenizer�AutoModelForCausalLM)� load_dataset)�tqdmz&../train/output/qlora-codellama-bugfixz9eval.jsonlz./output/eval_results.jsonz./output/eval_results.csvi � �cuda�cpuT)�exist_oku 🔄 Loading model...�auto)� torch_dtype�10device_mapu 📂 Loading evaluation data...�json�train)�11data_files�splitu ⚙️ Running inference...�prompt�12completion�pt)�return_tensors�13truncation�14max_lengthF� )�max_new_tokens� do_sample� num_beams� pad_token_id� eos_token_id)�skip_special_tokens)r � reference�15prediction�wzutf-8)�encoding� )�indentu ✅ Saved JSON to � )r" �newline)�16fieldnamesu ✅ Saved CSV to u 📊 Computing BLEU and ROUGE...�bleu�rouger r )� predictions�17referencesu 18📈 Evaluation Results:zBLEU:zROUGE:� )=�torch� transformersr r �datasetsr r r �csv�os�evaluate�19MODEL_PATH� EVAL_FILE� OUTPUT_JSON�20OUTPUT_CSV�
MAX_INPUT_LEN�MAX_NEW_TOKENSr � is_available�DEVICE�makedirs�path�dirname�print�from_pretrained� tokenizer�bfloat16�model�eval� eval_data�results�exampler r �to�inputs�no_grad�generater r �outputs�decoder �append�strip�open�f�dump�21DictWriter�writer� writeheader� writerows�loadr( r) r* r+ �compute�22bleu_score� rouge_score�dumps)�r�refs 00�?/workspace/workspace/Linux-Kernel-Detector/evaluate/evaluate.py�<module>r^ s) �� � <� !� � � 23� � � 6�24� � �*� �
(�25��
����:�:�*�*�,��%�� �� � �B�G�G�O�O�K� (�4� 8� � � � )�M� )� )�*� 5� �,� �,�,������ ��26 �27�28� � �'� (� ��I�W� E� �
�� �#� $��I�� �G�
�X�
�F�� �%�I�
�v�d�t�P]�
^�
a�
a�bh�
i�F� �� � � � 29� �%�.�.� 30��31�)���"�/�/�"�/�/�
32��33� �!�!�'�!�*�$�!�G�J� �N�N�� �_�_�&� �&�&�(�� �%�2 34�+�s�W�-� $��
�D�I�I�g�q��#�$� ��;�-�(� )� �*�c�G�R�8� �A�
�S�^�^�A�*O�
P�F�35����36���W� � � ��*� �&� '� �(� )��x�}�}�V� ����
�
�g� ��(/�0�1�q� ��0� �&-�
.��a� �n�
.�37�
�\�\�k�PZ�>[���u�>[�\�
\�38��m�m� �39�m�K� � �"� #� �g�z� � �h�40��41�42�;�q�1� 2�U43� 44��&$� $�� � �� 1��
.��>[s6 �'/K&�2K3�'8L �" L �5 L�45L46�&K0 �3K=� L