Modelpublic

deliciouscat/deberta-v3-base-encoder-decoder-v0.3

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes16downloads

Model Card

Encoder-Decoder model with DeBERTa encoder

pre-trained models

—deliciouscat/deberta-v3-base-encoder-decoder-v0.2

-> 297511524(298M) params

Data used

—HuggingFaceFW/fineweb

—AiHub ko-en translation corpus (English part)

—Some papers that I kept

Training hparams

—optimizer: AdamW, lr=3e-5, betas=(0.875, 0.997)

—batch size: 12

-> training on denoising objective (BART), 29523 step

How to use

from transformers import AutoTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
tokenizer = AutoTokenizer.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")

Future work!

—train more scientific data

—fine-tune on keyword extraction task