Question: I am quite new to machine learning and neural nets. I‘ve used the following model for sentiment analysis of short texts. I generally understand how signals are computed, all the way to the output layer. Now what I dont understand is how the inputs are found. When the model classifies a word, how is that word translated to the 512 input units? What features of the word does the model assess and how is that decided?

```
  model = Sequential()
  model.add(Dense(512, input_shape=(max_words,), activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(256, activation='sigmoid'))
  model.add(Dropout(0.5))
  model.add(Dense(2, activation='softmax'))

  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])

```

Answer: 
> 
> When the model classifies a word, how is that word translated to the
> 512 input units?
> 
> 
> 

As you already noticed, before any kind of written information (single words, sentences or whole texts) can be processed by a neural network, it must be *encoded* into a vector representation. This is called an [*embedding*](https://en.wikipedia.org/wiki/Word_embedding) or a *representation* and to find suitable embeddings is subfield of Natural Language Procesessing (NLP) research.

Over the years a number of different representations were published. For single words e.g. [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) in which a neural network has "learned" the embedding based on the semantic similarity of the words. That means words which are similar in context should be close by in the vector space.

The most simple embedding for a sentence would be a [*bag-of-words*](https://en.wikipedia.org/wiki/Bag-of-words_model) embedding. This means we count how many different words we have in our corpus of sentences (e.g. `N`) and we transform each sentence into a vector of length `N` where each index of the vector represents a word and the value at the index the number of occurrences of that word in the sentence.

Of course there are many more sophisticated text embeddings.
There are multiple methods by which you can obtain the vector embedding of a word.

1. Count based methods: PMI, PPMI and SVD
2. Prediction based methods: CBOW and Skip-Gram

The count-based methods create a co-occurrence matrix of words of shape Vocabulary\*Vocabulary where each word is represented by some sort of count of co-occurrence in K neighborhood.

The prediction-based models train on a corpus and create a vector embedding basis on how close the context of two words are.