Question: I have two matrix V\_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V\_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a way to do it using broadcasting?

```
    list=[]
for v in V_r:
    a=np.linalg.norm(vecs-v,axis=1)
    list.append(a)
M=np.vstack(list)

```

Answer: 
```
from keras.preprocessing import Tokenizer
samples = ["grss is green and sun is hot"]
tokenizer = Tokenizer(num_words=1000)
tokenizer.fit_on_texts(samples)
sequences = tokenizer.texts_to_sequences(samples)

```

The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into your neural network. There are other ways like TF/IDF and CountVectorizer in Sklearn for more simpler algorithms. `num_words` takes the most 1000 frequent words and tokenizes them. 

Link : [Keras text processing](https://keras.io/preprocessing/text/)
Danny is 100% correct. Also, different NLP problems require different preprocessing solutions.

Regarding your question:

> 
> how to convert the sequences of words into digitized input
> 
> 
> 

First, Split the string text into unique words.

Then, I recommend ordering the outcome in descending order. Depending on the use case, you will probably want to remove or at least know which words repeat the most.

Finally, create a dictionary that maps the word to an integer in ascending order. For example, the most common expression would be index 0 (or 1 in some cases) and another dictionary to map the integers back to the words.

```
from collections import Counter  

text = "the grass is green and the sun is hot"

# split the text into words
word_counts = Counter(text.split(' '))

# list the words from most common to less common 
sorted_counts = sorted(word_counts, key=word_counts.get, reverse=True)

# creates an integer to word dictionary 
int_to_word = {i: word for i, word in enumerate(sorted_counts)} 

# creates a word to integer dictionary 
word_to_int = {word: i for i, word in int_to_word.items()} 

# convert the words to numbers
for word, i in word_to_int.items():
    text = text.replace(word, f'{i}')

```

Integer to Word:

> 
> {0: 'the', 1: 'is', 2: 'grass', 3: 'green', 4: 'and', 5: 'sun', 6:> 'hot'}
> 
> 
> 

Word to Integer:

> 
> {'the': 0, 'is': 1, 'grass': 2, 'green': 3, 'and': 4, 'sun': 5, 'hot': 6}
> 
> 
> 

Finally,

> 
> text = '0 2 1 3 4 0 5 1 6'
> 
> 
> 

Depending on the problem you're trying to solve, you will likely need additional processing (for example, the numbers are still strings). So there is a lot of work ahead of you. However, at least you now know how to represent words as numbers and decode numbers back to words!