in order to get good word embeddings.
comfortably more efficient way to learn this types of embeddings.
Most of the ideas I'll present in this video are due to Tomas Mikolov, Kai Chen, Greg Corrado, and Jeff Dean.
Let's say you're given this sentence in your training set.
to target errors to create our supervised learning problem.
randomly pick a word to be the context word.
And let's say we chose the word orange.
And what we're going to do is randomly pick another word within some window.
plus minus ten words of the context word and we choose that to be target word.
that's just one word later.
Or you might choose two words before.
Maybe just by chance you choose the word my as the target.
ten word window, or plus minus five or ten word window of that input context word.
plus minus 10 words of the word orange, it could be a lot of different words.
it is that we want to use this learning problem to learn good word embeddings.
So, here are the details of the model.
Let's say that we'll continue to our vocab of 10,000 words.
And some have been on vocab sizes that exceeds a million words.
the word glass or the word my, if we use the example from the previous slide.
juice is the word 4834 in our vocab of 10,000 words.
And so that's the input x that you want to learn to map to that open y.
there's a one hot vector for the context words.
so here EC is equal to capital E times that one hot vector.
feed it to a softmax unit.
So I've been drawing softmax unit as a node in a neural network.
That's not an o, that's a softmax unit.
And then there's a drop in the softmax unit to output y hat.
So to write out this model in detail.
sum from J equals one to all 10,000 words of e to the theta j transposed ec.
of a particular word, t, being the label.
we could include that too if we wish.
And then finally the loss function for softmax will be the usual.
So we use y to represent the target word.
And we use a one-hot representation for y hat and y here.
to 10,000 of yi log yi hat.
we're representing the target y as a one hot vector.
So this would be a one hot vector with just 1 1 and the rest zeros.
And if the target word is juice, then it'd be element 4834 from up here.
That is equal to 1 and the rest will be equal to 0.
unit with probabilities for all 10,000 possible targets words.
network with basically looking up the embedding and then just a soft max unit.
corresponding to all of these embedding vectors, E subscript C.
these parameters, you actually get a pretty good set of embedding vectors.
the left or the right side.
To predict what comes little bit before little bit after the context words.
Now, it turns out there are a couple problems with using this algorithm.
And the primary problem is computational speed.
probability, you need to carry out a sum over all 10,000 words in your vocabulary.
it gets really slow to sum up over this denominator every single time.
it makes even harder to scale to larger vocabularies.
is to use a hierarchical softmax classifier.
instead of trying to categorize something into all 10,000 carries on one go.
it tells you is the target word in the first 5,000 words in the vocabulary?
Or is in the second 5,000 words in the vocabulary?
words of vocab or in the second 2,500 words vocab and so on.
means that each of the retriever nodes of the tree can be just a binding classifier.
else it will capsize in order to make a single classification.
scales like log of the vocab size rather than linear in vocab size.
So this is called a hierarchical softmax classifier.
with equal numbers of words on the left and right sides of each branch.
whereas the less common words like durian can be buried much deeper in the tree.
you might need only a few traversals to get to common words like the and of.
okay that are buried deep in the tree because you don't need to go that deep.
building the tree how you used to build the hierarchical software spire.
the speeding up the softmax classification.
But I won't spend too much more time.
and others, on the first slide.
But I won't spend too much more time on this.
called nectar sampling, which I think is even simpler.
the problem of needing the sum over the entire cap size in the denominator.
So you see more of that in the next video.
one quick Topic I want you to understand is how to sample the context C.
how do you choose the context C?
and, to and so on that appear extremely frequently.
like orange, apple, and also durian that don't appear that often.
ec, for those frequently occurring words.
these less common words like e durian.
something from the common words together with the less common words.
So that's it for the Word2Vec skip-gram model.
paper actually had two versions of this Word2Vec model, the skip gram was one.
and that algorithm also works, it has some advantages and disadvantages.
sum over your entire vocabulary size into the denominator of the soft packs.
bigger fitting set as well and therefore learn much better word embeddings.
Lets go onto the next video.