mrkwanzaa/functionalizer-100M-github-code-python-seed1

1---2tags:3- functionalizer4- tokenizer5- gpt26library_name: transformers7pipeline_tag: text-generation8---9 10This model was presented in the paper [The Functionalizer: Lossless Functional Decomposition for Subword Tokenization](https://huggingface.co/papers/2609.15991).11 12## Functionalizer 100M model13This is a 100M example model. Each model is trained on the dataset specified in the name for 50000 epochs with the given seed.14 15The model was trained using the Functionalizer framework. Training code and detailed performance analysis are available: https://github.com/connor-makowski/functionalizer16 17### Running the model18 19To run the model, use the custom tokenizer fork available here: https://github.com/connor-makowski/tokenizers/tree/functionalizer20 21### Citation22 23If you find this model or the Functionalizer framework useful, please cite:24 25```bibtex26@misc{makowski2026functionalizerlosslessfunctionaldecomposition,27      title={The Functionalizer: Lossless Functional Decomposition for Subword Tokenization}, 28      author={Connor Makowski and Willem Guter},29      year={2026},30      eprint={2609.15991},31      archivePrefix={arXiv},32      primaryClass={cs.CL},33      url={https://arxiv.org/abs/2609.15991}, 34}35```