WebOrganizer/FormatClassifier-NoURL

1---2library_name: transformers3datasets:4- WebOrganizer/FormatAnnotations-Llama-3.1-8B5- WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP86base_model:7- Alibaba-NLP/gte-base-en-v1.58---9# WebOrganizer/FormatClassifier-NoURL10 11[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)]12 13The FormatClassifier-NoURL organizes web content into 24 categories based on the text contents of web pages (without using URL information).14The model is a [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) with 140M parameters fine-tuned on the following training data:151. [WebOrganizer/FormatAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/FormatAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training)162. [WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training)17 18#### All Domain Classifiers19- [WebOrganizer/FormatClassifier](https://huggingface.co/WebOrganizer/FormatClassifier)20- [WebOrganizer/FormatClassifier-NoURL](https://huggingface.co/WebOrganizer/FormatClassifier-NoURL) *← you are here!*21- [WebOrganizer/TopicClassifier](https://huggingface.co/WebOrganizer/TopicClassifier)22- [WebOrganizer/TopicClassifier-NoURL](https://huggingface.co/WebOrganizer/TopicClassifier-NoURL)23 24## Usage25 26This classifier expects input in the following format:27```28{text}29```30 31Example:32```python33from transformers import AutoTokenizer, AutoModelForSequenceClassification34 35tokenizer = AutoTokenizer.from_pretrained("WebOrganizer/FormatClassifier-NoURL")36model = AutoModelForSequenceClassification.from_pretrained(37    "WebOrganizer/FormatClassifier-NoURL",38    trust_remote_code=True,39    use_memory_efficient_attention=False)40 41web_page = """How to make a good sandwich? [Click here to read article]"""42 43inputs = tokenizer([web_page], return_tensors="pt")44outputs = model(**inputs)45 46probs = outputs.logits.softmax(dim=-1)47print(probs.argmax(dim=-1))48# -> 6 ("Truncated" format, which covers incomplete content)49```50 51You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config):521. Academic Writing532. Content Listing543. Creative Writing554. Customer Support565. Comment Section576. FAQ587. Truncated598. Knowledge Article609. Legal Notices6110. Listicle6211. News Article6312. Nonfiction Writing6413. About (Org.)6514. News (Org.)6615. About (Pers.)6716. Personal Blog6817. Product Page6918. Q&A Forum7019. Spam / Ads7120. Structured Data7221. Documentation7322. Audio Transcript7423. Tutorial7524. User Review76 77The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/formats.yaml).78 79#### Efficient Inference80We recommend that you use the efficient gte-base-en-v1.5 implementation by enabling unpadding and memory efficient attention. This __requires installing `xformers`__ (see more [here](https://huggingface.co/Alibaba-NLP/new-impl#recommendation-enable-unpadding-and-acceleration-with-xformers)) and loading the model like:81```python82AutoModelForSequenceClassification.from_pretrained(83    "WebOrganizer/FormatClassifier-NoURL",84    trust_remote_code=True,85    unpad_inputs=True,86    use_memory_efficient_attention=True,87    torch_dtype=torch.bfloat1688)89```90 91## Citation92```bibtex93@article{wettig2025organize,94  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},95  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},96  journal={arXiv preprint arXiv:2502.10341},97  year={2025}98}99```