Team Ai
Modelpublic

LeroyDyer/SpydazWeb_Speech_Vision_EncoderDecoder_Multimodal_5b_Project

sourceHugging Facemitupdated 2y agoView on Hugging Face
1likes17downloads
Model Card

Creation Process

<img src="https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg" width="300"/>

Vmodel = VisionEncoderDecoderModel.fromencoderdecoderpretrained( "google/vit-base-patch16-224-in21k", "LeroyDyer/MixtralAITiny" ) EncoderImageProcessor = Vmodel.encoder DecoderImageTokenizer = Vmodel.decoder VisionEncoderDecoderModel = Vmodel

Add Pad tokems

LMMODEL.VisionEncoderDecoder = VisionEncoderDecoderModel

Add Sub Components

LMMODEL.EncoderImageProcessor = EncoderImageProcessor LMMODEL.DecoderImageTokenizer = DecoderImageTokenizer LM_MODEL


# ADD AUDIO

print('Add Audio...') #Add Head

Combine pre-trained encoder and pre-trained decoder to form a Seq2Seq model

AudioFeatureExtractor = AutoFeatureExtractor.frompretrained("openai/whisper-small") AudioTokenizer = AutoTokenizer.frompretrained("openai/whisper-small") SpeechEncoderDecoder = SpeechEncoderDecoderModel.fromencoderdecoderpretrained("openai/whisper-small","openai/whisper-small")

Add Pad tokems

SpeechEncoderDecoder.config.decoderstarttokenid = AudioTokenizer.clstokenid SpeechEncoderDecoder.config.padtokenid = AudioTokenizer.padtokenid LMMODEL.SpeechEncoderDecoder = _SpeechEncoderDecoder

Add Sub Components

LMMODEL.DecoderAudioTokenizer = AudioTokenizer LMMODEL.EncoderAudioFeatureExtractor = AudioFeatureExtractor LM_MODEL


# SAVE

print('Final stages:...') print('Add tokenizer...') LMMODEL.resizetokenembeddings(len(tokenizer)) LMMODEL.tokenizer = tokenizer print('Save model...') LMMODEL.to(torch.float16) LMMODEL.savepretrained("MixtralAIMiniModalTron") print('Save tokenizer...') tokenizer.savepretrained("MixtralAIMiniModalTron")