opendiffusionai/sd15vae-texttuned

sd 1.5 vae, text trained demo

It has been said that the sd vae is not capable of faithful text rendering, at a fundamental level. I decided to test that and it turns out to not exactly be true.

This 4channel vae can render a 16pt font with almost perfect fidelity.

This is a TECH DEMO ONLY. I have trained it exclusively on text, so normal rendering has drastically suffered as a result.

Training details

This started with the original sd vae. I generated an initial dataset of 14pt-only text images, with https://github.com/ppbrown/ai-training/blob/main/trainer/vae/genglyphdataset.py ( around 4000 of them) I then trained the vae at LR 1e-5 for 28000 steps, batchsize 1, using my standard settings of lpips/vgg weights 0.08, LAP 0.02 and "rawvgg" enabled, with my train_vae.py in that same repo

I then generated a MIXED size font dataset,ranging across 14,16,18, and 20pt total images around 8000 This time I chose batch size 2, LR 2e-5.

14pt text is of course more difficult than 16pt text. 16pt, however, renders almost indistinguishable from original. The specific font used across all images, was Libertinus Serif

I did have a fairly good result at 69000 steps for 14pt. I thought it had hit a floor, but then suddenly at 93000 steps, there was a dramatic breakthrough, at least by the numbers. (and then there was another one at 157000)

!(clean 16pt)[bookimg16.step093000.webp]

If you want to try it out for yourself, you can use the tools in my repo, such as

trainer/cache-utils/createimgcachesdvae.py --vae --previewonly \ --model opendiffusionai/sd15vae-texttrained --data_root someimgdir