LeroyDyer/SpydazWeb_Speech_Vision_EncoderDecoder_Multimodal_5b_Project

1---2license: mit3language:4- en5library_name: transformers6tags:7- 'vision '8- speech9- image-text-text10- audio-text-text11- Multi-Modal12---13 14# Creation Process15<img src="https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg" width="300"/>16 17Vmodel = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(18    "google/vit-base-patch16-224-in21k", "LeroyDyer/Mixtral_AI_Tiny"19)20_Encoder_ImageProcessor = Vmodel.encoder21_Decoder_ImageTokenizer = Vmodel.decoder22_VisionEncoderDecoderModel = Vmodel23# Add Pad tokems24LM_MODEL.VisionEncoderDecoder = _VisionEncoderDecoderModel25# Add Sub Components26LM_MODEL.Encoder_ImageProcessor = _Encoder_ImageProcessor27LM_MODEL.Decoder_ImageTokenizer = _Decoder_ImageTokenizer28LM_MODEL29 30 31```32 33# ADD AUDIO34 35```python36 37 38 39print('Add Audio...')40#Add Head41# Combine pre-trained encoder and pre-trained decoder to form a Seq2Seq model42_AudioFeatureExtractor = AutoFeatureExtractor.from_pretrained("openai/whisper-small")43_AudioTokenizer = AutoTokenizer.from_pretrained("openai/whisper-small")44_SpeechEncoderDecoder = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained("openai/whisper-small","openai/whisper-small")45 46# Add Pad tokems47_SpeechEncoderDecoder.config.decoder_start_token_id = _AudioTokenizer.cls_token_id48_SpeechEncoderDecoder.config.pad_token_id = _AudioTokenizer.pad_token_id49LM_MODEL.SpeechEncoderDecoder = _SpeechEncoderDecoder50# Add Sub Components51LM_MODEL.Decoder_AudioTokenizer = _AudioTokenizer52LM_MODEL.Encoder_AudioFeatureExtractor = _AudioFeatureExtractor53LM_MODEL54 55```56 57# SAVE58```python59print('Final stages:...')60print('Add tokenizer...')61LM_MODEL.resize_token_embeddings(len(tokenizer))62LM_MODEL.tokenizer = tokenizer63print('Save model...')64LM_MODEL.to(torch.float16)65LM_MODEL.save_pretrained("Mixtral_AI_MiniModalTron")66print('Save tokenizer...')67tokenizer.save_pretrained("Mixtral_AI_MiniModalTron")68 69```