title: When processing IMDB data prepared by myself with a Keras RNN, accuracy never exceeds 0.5
body: <p>A very strange thing is happening. I fetched the IMDB corpus from Kaggle, kept only the 50,000 positive and negative texts, counted word frequencies, sorted words according to their decreasing frequency, replaced in the texts the 10,000 most frequent words by their rank (plus 3 units), inserted a 1 at all sentence begins and replaced all words outside the 10,000 most frequent ones by number 2. In this I followed exactly the instructions given in the documentation of Keras <code>imdb</code> class.</p>
  
  <p>I then ran a RNN with an Embedding layer, a SimpleRNN layer, a Dense layer. The result I get is an accuracy always around 0.5, no matter how hard I try. I then replace my code by <code>imdb.load_data(num_words=10000)</code> and get an accuracy of 0.86 already at the third epoch. How is this possible? Why such an extreme difference? What have I done wrong?</p>
  
  <p>Here is the code I used:</p>
  
  <pre><code>import re, os, time, pickle
  word=re.compile(r'^[A-Za-z]+$')
  spacyline=re.compile(r'^([0-9]+) ([^ ]+) ([^ ]+) ([^ ]+) ([0-9]+) ([A-Za-z]+)')
  
  DICT=dict()
  
  inp=open("imdb_master_lemma.txt")
  for ligne in inp:
      if (ligne[0:9]=="DEBUT DOC"):
          if (ligne[-4:-1]=="neg"):
              classe=-1
          elif (ligne[-4:-1]=="pos"):
              classe=1
      elif (ligne[0:9]=="FIN DOCUM"):
          a=1
      else:
          res=spacyline.findall(ligne)
          if res:
              lemma=str(res[0][3])
              if (word.match(lemma)):
                  if (lemma in DICT.keys()):
                      DICT[lemma] += 1
                  else:
                      DICT[lemma]=1
  inp.close()
  
  SORTED_WORDS=sorted(DICT.keys(), key=lambda x:DICT[x], reverse=True)
  THOUSAND=SORTED_WORDS[:9997]
  ORDRE=dict()
  c=0
  for w in THOUSAND:
      ORDRE[w]=c
      c+=1
  CORPUS=[]
  CLASSES=[]
  
  inp=open("imdb_master_lemma.txt")
  for ligne in inp:
      if (ligne[0:9]=="DEBUT DOC"):
          if (ligne[-4:-1]=="neg"):
              classe=0
          elif (ligne[-4:-1]=="pos"):
              classe=1
          a=[]
      if (ligne[0:9]=="DEBUT PHR"):
          a.append(1)
      elif (ligne[0:9]=="FIN DOCUM"):
          CORPUS.append(a)
          CLASSES.append(classe)
      else:
          res=spacyline.findall(ligne)
          if res:
              lemma=str(res[0][3])
              if lemma in ORDRE:
                  a.append(ORDRE[lemma]+3)
              elif (word.match(lemma)):
                  a.append(2)
  inp.close()
  
  from sklearn.utils import shuffle
  CORPUS, CLASSES=shuffle(CORPUS, CLASSES)
  
  out=open("keras1000.pickle","wb")
  pickle.dump((CORPUS,CLASSES,ORDRE),out)
  out.close()
  </code></pre>
  
  <p>The file <code>imdb_master_lemma.txt</code> contains the IMDB texts processed by Spacy, and I keep only the lemma (which is already in lowercase, so this is more or less what is used in Keras <code>imdb</code> only it should work even better since there are no plurals and verbs are lemmatized). Once the pickle file stored, I recall it and use it as follows:</p>
  
  <pre><code>picklefile=open("keras1000.pickle","rb")
  (CORPUS,CLASSES,ORDRE)=pickle.load(picklefile)
  picklefile.close()
  
  import numpy as np
  def vectorize_sequences(sequences, dimension=10000):
      results = np.zeros((len(sequences), dimension))
      for i, sequence in enumerate(sequences):
          results[i, sequence] = 1.
      return results
  x_train = np.array(vectorize_sequences(CORPUS[:25000]),dtype=object)
  x_test = np.array(vectorize_sequences(CORPUS[25000:]),dtype=object)
  train_labels = np.array(CLASSES[:25000])
  test_labels = np.array(CLASSES[25000:])
  
  from keras import models
  from keras import layers
  from keras.models import Sequential
  from keras.layers import Flatten, Dense, Embedding, SimpleRNN, LSTM, Bidirectional
  from keras.preprocessing import sequence
  
  input_train = sequence.pad_sequences(x_train, maxlen=500)
  input_test = sequence.pad_sequences(x_test, maxlen=500)
  print('input_train shape:', input_train.shape)
  print('input_test shape:', input_test.shape)
  
  model = Sequential()
  model.add(Embedding(10000, 32))
  model.add(SimpleRNN(32))
  model.add(Dense(1, activation='sigmoid'))
  model.compile(optimizer='rmsprop',
                loss='binary_crossentropy',
                metrics=['acc'])
  
  history = model.fit(input_train,
                      train_labels,
                      epochs=10,
                      batch_size=128,
                      validation_split=0.2)
  results = model.evaluate(input_test, test_labels)
  
  print(results)
  </code></pre>
  
  <p>The result is utterly disappointing, an accuracy around 0.5. When I replace the 14 first lines by</p>
  
  <pre><code>from keras.datasets import imdb
  (x_train, train_labels), (x_test, test_labels) = imdb.load_data(num_words=10000)
  </code></pre>
  
  <p>then everything works as described in Chollet's book and I get immediately a very high accuracy.</p>
  
  <p>Can anyone tell me what I am doing wrong?</p>
  
  <p>PS. Here is a small sample of the data, to illustrate the preparation process: The two first sentences of the first IMDB document, as processed by spaCy, are</p>
  
  <pre><code>DEBUT DOCUMENT neg
  DEBUT PHRASE
  0 Once RB once 1 advmod
  1 again RB again 5 advmod
  2 Mr. NNP mr. 3 compound
  3 Costner NNP costner 5 nsubj
  4 has VBZ have 5 aux
  5 dragged VBN drag 5 ROOT
  6 out RP out 5 prt
  7 a DT a 8 det
  8 movie NN movie 5 dobj
  9 for IN for 5 prep
  10 far RB far 11 advmod
  11 longer JJR long 9 pcomp
  12 than IN than 11 prep
  13 necessary JJ necessary 12 amod
  14 . . . 5 punct
  FIN PHRASE
  DEBUT PHRASE
  15 Aside RB aside 16 advmod
  16 from IN from 33 prep
  17 the DT the 21 det
  18 terrific JJ terrific 19 amod
  19 sea NN sea 21 compound
  20 rescue NN rescue 21 compound
  21 sequences NNS sequence 16 pobj
  22 , , , 21 punct
  23 of IN of 26 prep
  24 which WDT which 23 pobj
  25 there EX there 26 expl
  26 are VBP be 21 relcl
  27 very RB very 28 advmod
  28 few JJ few 26 acomp
  29 I PRP -PRON- 33 nsubj
  30 just RB just 33 advmod
  31 did VBD do 33 aux
  32 not RB not 33 neg
  33 care VB care 33 ROOT
  34 about IN about 33 prep
  35 any DT any 34 pobj
  36 of IN of 35 prep
  37 the DT the 38 det
  38 characters NNS character 36 pobj
  39 . . . 33 punct
  FIN PHRASE
  </code></pre>
  
  <p>This becomes:</p>
  
  <pre><code>[1, 258, 155, 5920, 13, 979, 38, 6, 14, 17, 207, 165, 68, 1526, 1, 1044, 33, 3, 1212, 1380, 1396, 382, 7, 58, 34, 4, 51, 150, 37, 19, 12, 338, 39, 91, 7, 3, 46,
  </code></pre>
  
  <p>etc. As you can see, the <code>1</code> shows the sentence begin, <code>258</code> is <code>once</code>, <code>155</code> is <code>again</code>, I have missed <code>mr.</code> because it contains a period (but this can hardly be the reason my system is failing), <code>5920</code> is <code>costner</code> (apparently Kevin Costner's name appears so often, that it is included in the 10,000 most frequent words), <code>13</code> is <code>have</code>, <code>979</code> is <code>drag</code>, <code>38</code> is <code>out</code>, <code>6</code> is the article <code>a</code>, <code>14</code> is the word <code>movie</code>, and so on. These ranks are all very reasonable, I think, so I don't see what may have went wrong.</p>