MiniGPT-4
On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a 
linear layer, and train just the tiny layer on some datasets of image-text pairs.<p>But the <i>results</i> are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Source AI.<p>The repo's default inference code is kind of bad -- vicuna is loaded in fp16 so it can't fit on any consumer hardware. I created a PR on the repo to load it with int8, so hopefully by tomorrow it'll be runnable by 3090/4090 users.<p>I also developed a toy discord bot (<a href="https://github.com/152334H/MiniGPT-4-discord-bot">https://github.com/152334H/MiniGPT-4-discord-bot</a>) to show the model to some people, but inference is very slow so I doubt I'll be hosting it publicly.