Show HN: Paint.wtf, a drawing game judged by OpenAI's zero-shot image classifier
We&#x27;re really excited about CLIP[1], OpenAI&#x27;s new zero-shot image classification model. We think it&#x27;s going to power a ton of creative new products. But being inspired isn&#x27;t as fun as being inspiring so this weekend our team[2] (and a friend[3]) built a drawing game powered by CLIP.<p>The concept is simple: you draw a prompt, we feed it into CLIP and calculate a score[4] based on how close it thinks your drawing is to the prompt, we keep track of a leaderboard for each prompt. Your goal: draw the best representation of the prompt, as judged by CLIP.<p>*Why it&#x27;s so cool*<p>Two weeks ago, if you wanted to make a game like this you&#x27;d have to collect a huge dataset of images matching each prompt and train a custom model to learn what drawings of each of them looked like. This meant you&#x27;d need to limit it to simple concepts like &quot;apple&quot; or &quot;fish&quot; that you could find a sufficient number of images for. And adding in new prompts ones would be just as hard; you&#x27;d have to collect more data and train another model.<p>With CLIP this is unnecessary. We can feed it <i>any</i> prompt eg &quot;a raccoon driving a tractor&quot; and it performs marvelously without having to be trained on custom data at all. This is because, unlike traditional image classifiers, it uses the information from the text to sort the images into buckets (not just the pixels from the image) and like GPT-3 it has already seen enough text and images to be able to generalize to new combinations of those things.<p>It&#x27;s a bit like vocabulary: if you know what a &quot;drawing&quot; is, you know what a &quot;raccoon&quot; is, you know what &quot;driving&quot; is, and you know what a &quot;tractor&quot; is, you don&#x27;t have to have seen any &quot;drawings of a raccoon driving a tractor&quot; before to be able to identify them.<p>[1] <a href="https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;clip&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;clip&#x2F;</a><p>[2] Roboflow, <a href="https:&#x2F;&#x2F;roboflow.com" rel="nofollow">https:&#x2F;&#x2F;roboflow.com</a><p>[3] Erik from Booste, <a href="https:&#x2F;&#x2F;booste.io" rel="nofollow">https:&#x2F;&#x2F;booste.io</a><p>[4] Full technical writeup in progress; check back later this week