AI writing stories based on Pictures
What happens when you take a curious kid, that loves writing stories, and combine her with technology?
Well, she spends countless nights debugging a program to generate stories based off an image.
She’s inspired by the fantastic (and classic) Andrej Karpathy unreasonable effectiveness of recurrent neural network article and decides to build this thing for funzies.
The TL;DR, is I trained two LSTM’s: one to take an image and describe it. The other to take the description, and generate a story.
You might be wondering…
What the heck is an LSTM!
LSTM stands for long-short-term memory. It’s a type of RNN, that’s more effective in storing memory.
So, an RNN (or recurrent neural network) is a network with a memory: it keeps feeding its inputs back into itself, so it can remember past information.
This is what a “vanilla” or plain RNN looks like:
How they work, is instead of having one input (x), like every other model, RNNs have something called a hidden state. (Indicated by h)… this hidden state is fed into the model with the new input.
So, instead of 1 input, the model has 2.
RNNs are used for sequencial data. (They have hidden states to remember old info, so they can do stuff like understand all words in a sentence) So, stuff like text, video frames, pixels in an image. But, they do have some flaws in remembering information*insert calculous explanation*
Which is why we invented a sundae with sprinkles on the top model (LSTM).
An LSTM is a type of recurrent network cell, and it is a bit more complicated. It has a bunch of gates or mini neural networks (indicated by the sigmoid function). These help understand the inputs into the model since they’re using more complicated but more organized math.
Think of an RNN cell as a student with no binders. They just collect a bunch of papers in their bag, they’re honestly a hot mess.
But an LSTM has everything under control, with different gates (notebooks/binders) for the information to flow through and stay organized. The LSTM also throws away irrelevant information, whereas RNN keeps everything, and ends up getting overwhelmed and forgetting everything.
Since LSTMs have more gates, they also have more weights. In Machine learning, we train machines to learn by updating their weights.
Generally, deep learning models (most layers or “weights”) are state-of-the-art for doing detecting most complicated patterns.
DL is more efficient at diagnosing breast cancer, than say, a simple linear regression model.
LSTMs have more weights and are therefore deeper than RNNs so, in general, they’re more efficient for complex problems.
Using LSTMs, we can understand text (i.e. sequencial data) AND get good at predicting the next character.
For example, x is the input, y is the output:
#input:
x = 'CHAPTE' **Using LSTM: **#predicition:
Y = 'R'
The Char-level detection Model was pretty simple to make, using Keras and Tensorflow.
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTMmodel = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')print(model.summary())
To detect the next character in a given sequence, we just need to train an LSTM. It’s able to understand sequences (sentences + paragraphs), and we can teach it to predict.
Model Summary:
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm_2 (LSTM) (None, 256) 264192 _________________________________________________________________ dropout_2 (Dropout) (None, 256) 0 _________________________________________________________________ dense_2 (Dense) (None, 52) 13364 ================================================================= Total params: 277,556 Trainable params: 277,556 Non-trainable params: 0 _________________________________________________________________ None
I used an LSTM cell, a dropout layer, and a Dense layer. These are pretty standard for LSTMs, especially when you don’t have state-of-the-art GPUs… aka me.
Back to my project.
So, I used two LSTMs for very different tasks, and the end goal was being able to input an image and output a story.
My project can be broken into 2 parts:
- Picture captioner
- Text Generator
The output of the picture captioner is the input of the text generator.
To illustrate:
The image gets captioned, which is like the “prompt” for the other network to write the story. Just like a seed is the “prompt” for something to grow.
You know what they say… a picture’s worth a thousand words…
Now that we have a better idea of the whole model, let’s break it up.
#Captioner
The captioner model takes an image, understands that image, and uses its understanding, to make a relevant caption. (Hey, this could be used for Instagram captions 😎 #lifehacks)
The captioner starts with image classification. Using the VGG19 model (A state of the art image classifying CNN)… we have a neural network that’s as good at understanding images as Gordan Ramsay is at cooking. 🙌
CNNs are really good at identifying patterns.
They consist of a bunch of convolutional layers (which is almost like taking a flashlight over an image, and finding its most prevalent features, then compiling them). A really accurate CNN (like my homie VGG) have tons of these layers.
The network has a bunch of these layers (which detect features) and many max-pooling layers (which simplify the convolutional layers).
The last layer of the VGG network is an image classifying layer, which takes all of the patterns found in the image, and classifies what’s in the image (e.g. cat!).
But we don’t need the classification layer for the captioner, we’re really interested in the content representation, which comes right before this layer.
The content representation layers represents all the features/items in an image, in number form. Woah… I know… totally not in the name!
We train the captioner on images’ content representation( the numerical/feature map of the images from the content representation layer) and their corresponding captions. This is supervised learning.
Given a certain image (in # form), the captioner needs to come up with a caption.
Preparing Data
We clean the image data (put it through the VGG), and clean the text data (remove punctuation, make it lower case, remove numbers and single letter words (i.e. ‘a’)).
Each image, from the dataset I used, had 1–5 captions, so I created a dictionary with each image feature map, and its associated captions. With the captions, I used set() to remove the duplicate words.
An example:
{2252123185_487f21e336 : bunch on people are seated in stadium}{3457129268_451b88v679 : boy with straw hat}
For image, with content representation 2252123185_487f21e336, there are certain keywords associated.
We’ll train an LSTM to get good at predicting captions.
In order to begin predicting captions, the LSTM needs a seed word, we’ll use ‘startseq’, and ‘endseq’ to indicate beginning/end of a caption.
Given the seed, it prompts the next word. LSTMs are kind of like unmotivated teenagers, they need a little nudge to start doing stuff with themselves.
For example, if the caption was “Boy with lollipop in field”. Our LSTM should follow this training sequence:
input(x) output(y)
startseq Boy
Startseq Boy with
Startseq Boy with lollipop
Startseq Boy with lollipop in
Startseq Boy with lollipop in field
Startseq Boy with lollipop in field endseq
I did this (well, my computer did this) thousands of times so the model can get good at predicting captions based on an image input. If you want to play around with this idea of teaching a model to predict the next word/character in a sequence, this is a good resource.
To visualize the captioner mode, it’s an encoder + decoder model. The encoder, encodes images, while the decoder takes the encoding and turns it into words.
TL;DR captioner
- Takes an image, runs it through VGG 19 to get content representation.
- Cleans text data
- Creates a dictionary (or training set), of a given images’ content representation and its caption
- Use supervised learning + our favs, LSTMs to train the model to predict the next word in a given sequence.
- Run the captioner model, to describe any image of my choice.
VOILA:
… not too shabby for a 💻
#Generator
The generator is trained on text (i.e. books & blogs), and it is able to generate similar text to its training data, given a seed, using the one-and-only LSTMs.
So, theoretically, we can train an LSTM on a bunch of fairytale books, it’ll be able to write fairy tales.
Or, we can even train an LSTM on trump tweets to tweet like trump.
I used a few books from the Project Gutenberg dataset. I chose a variety of genres and styles so my LSTM could have a diverse vocabulary. I used the top 10 books from the top 100 English book list and compiled them into their own text file, for the training data.
Once the data’s prepared, I needed to do 3 things:
- Find the # of unique characters
- Create an integer to character dictionary
- Create an character to integer dictionary
A review of python dictionaries
The reason we’re doing this is to pre-process the data so it can be fed into the RNN model. Computers speak numbers, so we need to encode our letters into numbers.
chars = sorted(list(set(raw_text)))
char_to_int = dict((c, i) for i, c in enumerate(chars))
int_to_char = dict((i, c) for i, c in enumerate(chars)
Chars tell my computer what option of characters it has to guess. This generator model uses char-rnn prediction, meaning given a certain character it can predict the next single character (but not the whole word).
char_to_int and int_to_char is my computer’s way of translating its guesses to something I can understand, and it translates my inputs to something it can understand.
They’re dictionaries. So just like I could use a Spanish dictionary to translate English words to Spanish, we can translate numbers to letters.
After creating these, I had to split and prepare my data to train the model (down below)
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
Once I got the model prepared for training, I trained it for 50 epochs and saved the weights for the model with the lowest loss value.
Boom! Now, the model is able to take a seed, and generate text from it.
If you’re interested in going deeper into the code, some really useful resources for me have been: machine learning mastery, analytics vihdya and towards data science article.
This model takes in a seed, so I made the seed variable equal to the image caption created by the captioner model. (Example at the bottom)
Just to recap
Using two LSTMs (captioner, generator), I was able to take a given image, and create a story out of it. Pretty powerful.
Working on this project had tons of learnings, like how to create an LSTM model, how to learn about math [for LSTMs] and how to combine two models.
But, in the end, it was crazy fun, and I got some rather silly stories.
Seed = many people sit on rocks
Generator’s story (Capped at 600 characters ):
[many people sit on rocks] with heads in skye and the sister had head lap, they mummed togeher in, wind tooo. the fat fat boy ran, and the little sab sing ran and run and woos ‘so much wind blow high prety and book was good to rock soo kid ran and book andfad child . people like eat fod oneanother saw travel bird , sit si fun. time wass padrty and good music we have little rock see hiegh way for long time , sky blue, gooder stuff ahead with peep (something don’t happen don’t now we go head) context important yes i. we hv and lool hon lets go and go to mountan because we run now, from sibbit!
Couldn’t have said it better my self because we run now, from sibbit!
But for real, if I had more GPUs and training data, this generator could make the NYTimes 🙃
There are so many ways I could optimize this model, some ideas:
- Add more LSTM layers in either network
- Run through more epochs
- Train the captioner on a larger dataset
- Create more descriptive captions
- Train generator on a larger dataset, with more diverse dialogue
… what’s exciting about machine learning is there’s always so many ways to improve especially when you’re running from sibbit ;)
Thanks for reading! Be sure to clap & share with your network!
To connect with me, my email is igrandic03@gmail.com, twitter, LinkedIn, and you can sign up for my monthly newsletter.
If you like camels, read here. Or maybe you’re a dog person? Here.
But, if food’s more of your thing, I still got you. 🤩