mrkwanzaa/functionalizer-100M-github-code-python-seed4

1---2library_name: transformers3pipeline_tag: text-generation4tags:5- functionalizer6- tokenizer7- gpt28---9 10## Functionalizer 100M model11 12This is a 100M example model. Each model is trained on the dataset specified in the name for 50000 epochs with the given seed.13 14The model was trained using the Functionalizer framework, presented in [The Functionalizer: Lossless Functional Decomposition for Subword Tokenization](https://huggingface.co/papers/2609.15991). Training code and detailed performance analysis are available: https://github.com/connor-makowski/functionalizer15 16### Running the model17 18To run the model, use the custom tokenizer fork available here: https://github.com/connor-makowski/tokenizers/tree/functionalizer19 20### Citation21 22If you find this model or the Functionalizer framework useful, please cite:23 24```bibtex25@misc{makowski2026functionalizerlosslessfunctionaldecomposition,26      title={The Functionalizer: Lossless Functional Decomposition for Subword Tokenization}, 27      author={Connor Makowski and Willem Guter},28      year={2026},29      eprint={2609.15991},30      archivePrefix={arXiv},31      primaryClass={cs.CL},32      url={https://arxiv.org/abs/2609.15991}, 33}34```