KoalaAI/Text-Moderation

1---2tags:3- autotrain4- text-classification5language:6- en7widget:8- text: I love AutoTrain9- text: I absolutely hate those people10- text: I love cake!11- text: >-12    lets build the wall and deport illegals "they walk across the border like13    this is Central park"14- text: EU offers to pay countries 6,000 euros per person to take in migrants15datasets:16- mmathys/openai-moderation-api-evaluation17- KoalaAI/Text-Moderation-v2-small18co2_eq_emissions:19  emissions: 0.0396746811326873820license: openrail21---22 23# Text Moderation24This model is a text classification model based on Deberta-v3 that predicts whether a text contains text that could be considered offensive.25It is split up in the following labels:26 27| Category | Label | Definition |28| -------- | ----- | ---------- |29| sexual   | `S`   | Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness). |30| hate     | `H`   | Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. |31| violence | `V`   | Content that promotes or glorifies violence or celebrates the suffering or humiliation of others. |32| harassment       | `HR`   | Content that may be used to torment or annoy individuals in real life, or make harassment more likely to occur. |33| self-harm        | `SH`   | Content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders. |34| sexual/minors    | `S3`   | Sexual content that includes an individual who is under 18 years old. |35| hate/threatening | `H2`   | Hateful content that also includes violence or serious harm towards the targeted group. |36| violence/graphic | `V2`   | Violent content that depicts death, violence, or serious physical injury in extreme graphic detail. |37| OK | `OK` | Not offensive38 39It's important to remember that this model was only trained on English texts, and may not perform well on non-English inputs.40 41## Ethical Considerations42This is a model that deals with sensitive and potentially harmful language. Users should consider the ethical implications and potential risks of using or deploying this model in their applications or contexts. Some of the ethical issues that may arise are:43 44- The model may reinforce or amplify existing biases or stereotypes in the data or in the society. For example, the model may associate certain words or topics with offensive language based on the frequency or co-occurrence in the data, without considering the meaning or intent behind them. This may result in unfair or inaccurate predictions for some groups or individuals.45 46Users should carefully consider the purpose, context, and impact of using this model, and take appropriate measures to prevent or mitigate any potential harm. Users should also respect the privacy and consent of the data subjects, and adhere to the relevant laws and regulations in their jurisdictions.47 48## License49 50This model is licensed under the CodeML OpenRAIL-M 0.1 license, which is a variant of the BigCode OpenRAIL-M license. This license allows you to freely access, use, modify, and distribute this model and its derivatives, for research, commercial or non-commercial purposes, as long as you comply with the following conditions:51 52- You must include a copy of the license and the original source of the model in any copies or derivatives of the model that you distribute.53- You must not use the model or its derivatives for any unlawful, harmful, abusive, discriminatory, or offensive purposes, or to cause or contribute to any social or environmental harm.54- You must respect the privacy and consent of the data subjects whose data was used to train or evaluate the model, and adhere to the relevant laws and regulations in your jurisdiction.55- You must acknowledge that the model and its derivatives are provided "as is", without any warranties or guarantees of any kind, and that the licensor is not liable for any damages or losses arising from your use of the model or its derivatives.56 57By accessing or using this model, you agree to be bound by the terms of this license. If you do not agree with the terms of this license, you must not access or use this model.58 59## Training Details60- Problem type: Multi-class Classification61- CO2 Emissions (in grams): 0.039762 63## Validation Metrics64 65- Loss: 0.84866- Accuracy: 0.749 (75%)67- Macro F1: 0.32668- Micro F1: 0.74969- Weighted F1: 0.70370- Macro Precision: 0.32171- Micro Precision: 0.74972- Weighted Precision: 0.67173- Macro Recall: 0.34974- Micro Recall: 0.74975- Weighted Recall: 0.74976 77 78## Usage79 80You can use cURL to access this model:81 82```83$ curl -X POST -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" -d '{"inputs": "I love AutoTrain"}' https://api-inference.huggingface.co/models/KoalaAI/Text-Moderation84```85 86Or Python API:87 88```python89from transformers import AutoModelForSequenceClassification, AutoTokenizer90 91# Load the model and tokenizer92model = AutoModelForSequenceClassification.from_pretrained("KoalaAI/Text-Moderation")93tokenizer = AutoTokenizer.from_pretrained("KoalaAI/Text-Moderation")94 95# Run the model on your input96inputs = tokenizer("I love AutoTrain", return_tensors="pt")97outputs = model(**inputs)98 99# Get the predicted logits100logits = outputs.logits101 102# Apply softmax to get probabilities (scores)103probabilities = logits.softmax(dim=-1).squeeze()104 105# Retrieve the labels106id2label = model.config.id2label107labels = [id2label[idx] for idx in range(len(probabilities))]108 109# Combine labels and probabilities, then sort110label_prob_pairs = list(zip(labels, probabilities))111label_prob_pairs.sort(key=lambda item: item[1], reverse=True)  112 113# Print the sorted results114for label, probability in label_prob_pairs:115    print(f"Label: {label} - Probability: {probability:.4f}") 116```117 118The output of the above Python code will look like this:119```120Label: OK - Probability: 0.9840121Label: H - Probability: 0.0043122Label: SH - Probability: 0.0039123Label: V - Probability: 0.0019124Label: S - Probability: 0.0018125Label: HR - Probability: 0.0015126Label: V2 - Probability: 0.0011127Label: S3 - Probability: 0.0010128Label: H2 - Probability: 0.0006129```