HuggingFaceTB/python-edu-scorer

1---2license: apache-2.03base_model: Snowflake/snowflake-arctic-embed-m4tags:5- generated_from_trainer6metrics:7- precision8- recall9- accuracy10model-index:11- name: python-edu-scorer12  results: []13---14 15 16# Python-Edu Scorer17 18This model is a fine-tuned version of [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) on a dataset of Python files labeled by Llama3 for educational value.19We used this classifier to build the [Python-Edu](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) dataset.20 21### How to use in transformers22To load the Python-Edu classifier, use the following code:23 24```python25from transformers import AutoTokenizer, AutoModelForSequenceClassification26 27tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/python-edu-scorer")28model = AutoModelForSequenceClassification.from_pretrained("HuggingFaceTB/python-edu-scorer")29 30text = "This is a test sentence."31inputs = tokenizer(text, return_tensors="pt", padding="longest", truncation=True)32outputs = model(**inputs)33logits = outputs.logits.squeeze(-1).float().detach().numpy()34score = logits.item()35result = {36    "text": text,37    "score": score,38    "int_score": int(round(max(0, min(score, 5)))),39}40 41print(result)42# {'text': 'This is a test sentence.', 'score': 0.07964489609003067, 'int_score': 0}43```44 45## Intended uses & limitations46 47While the Python-Edu classifier performs well in distinguishing high-quality python code, there are some limitations:48 49- Scope: The model's performance might change for other datasets, in particular for out of distribution samples. It is also focused on educational content relevant to beginners and may not perform as well on content intended for higher education or specialized domains.50- Bias: The model's performance is dependent on the quality and representativeness of the training data and the LLM used for the annotation. Biases in both can affect the classifier's judgments. It might overfit to thoroughly commented code.51- Context: The classifier evaluates individual code files without considering broader context, which might impact its effectiveness in certain scenarios.52 53The training and inference code is available on GitHub 54https://github.com/huggingface/cosmopedia/tree/main/classification55 56## Training procedure57 58The classifier was trained on 450,000 pairs of python code files and their scores from 1 to 5, generated by Llama3. The samples were annotated based on their educational quality with 1 being not educational and 5 being highly educational. 59 60We added a classification head with a single regression output to [Snowflake-arctic-embed](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) and trained the model for 20 epochs with a learning rate of 3e-4. During training, the embedding and encoder layers were frozen to focus on the classification head. 61 62### Training hyperparameters63 64The following hyperparameters were used during training:65- learning_rate: 0.000366- train_batch_size: 25667- eval_batch_size: 12868- seed: 069- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0870- lr_scheduler_type: linear71- num_epochs: 2072 73### Training results74 75```76              precision    recall  f1-score   support77 78           1       0.84      0.46      0.59      836479           2       0.61      0.76      0.68     1960580           3       0.60      0.62      0.61     1618781           4       0.72      0.50      0.59      487282           5       0.38      0.08      0.13       11883 84    accuracy                           0.64     4914685   macro avg       0.63      0.48      0.52     4914686weighted avg       0.66      0.64      0.63     4914687```88 89 90### Framework versions91 92- Transformers 4.40.193- Pytorch 2.2.0+cu12194- Datasets 2.17.195- Tokenizers 0.19.196