Answers.Body: <p>A tutorial is not keys to a production grade system.  They are just to get one familiar with the basic concepts.  To move toward a system that works on more than the provided examples, takes quite a bit of research and experience.</p>
  <p>For example:</p>
  <ul>
  <li>Sigmoid vs Softmax.  Sigmoid activation in classification is often confidently wrong.  Softmax tends to be a little bit more stable.</li>
  <li>Two dense layers with 16 neurons are unlikely to be effective on more than a few tokens.  Transformers have been developed for a reason.</li>
  <li>Nothing has been said about fit or training accuracy or any cross validation that was performed.  These concerns give a hint about stability of a model.  Without it I cannot imagine expecting a model to preform reliably.</li>
  </ul>
  <p>This list is certainly not comprehensive, text processing is almost a field unto it's own.  I am not an expert and this is not the forum to debug one's code.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T18:14:38.353Z
Answers.LastActivityDate: 2024-04-25T18:14:38.353Z
Answers.UserDisplayName: foreverska
Body: <p>I have built a classical neural network based on IMDB reviews according to the tutorial in one book about AI. 25 000 positive reviews, 25 000 negative reviews. Positive reviews result to &quot;0&quot;, negative - to &quot;1&quot;.</p>
  <p>Here is the code:</p>
  <pre><code>from tensorflow.keras.datasets import imdb
  (train_data, train_label), (test_data, test_label) = imdb.load_data(num_words=10000)
  import numpy as np
  def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate (sequences):
      for j in sequence:
        results[i,j]=1
    return results
  x_train = vectorize_sequences(train_data)
  x_test = vectorize_sequences(test_data)
  y_train = np.asarray(train_label).astype('float32')
  y_test = np.asarray(test_label).astype('float32')
  from tensorflow import keras
  from tensorflow.keras import layers
  model = keras.Sequential([
      layers.Dense(16, activation=&quot;relu&quot;),
      layers.Dense(16, activation=&quot;relu&quot;),
      layers.Dense(1, activation=&quot;sigmoid&quot;)
  ])
  model.compile(optimizer='rmsprop',
                loss='binary_crossentropy',
                metrics=['accuracy']
  )
  x_val = x_train[:10000]
  partial_x_train = x_train[10000:]
  y_val = y_train[:10000]
  partial_y_train = y_train[10000:]
  history = model.fit(partial_x_train,
                      partial_y_train,
                      epochs=20,
                      batch_size=512,
                      validation_data=(x_val,y_val))
  results = model.evaluate(x_test, y_test)
  </code></pre>
  <p>Now I go to the IMDB website to see user reviews of Fallout. <a href="https://www.imdb.com/title/tt12637874/reviews?ref_=tt_urv" rel="nofollow noreferrer">https://www.imdb.com/title/tt12637874/reviews?ref_=tt_urv</a></p>
  <p>Then I copied the 10/10 review. Replaced the word &quot;Bethesda&quot; with &quot;character&quot;, &quot;Aliexpress&quot; with &quot;online store&quot;, since the words were not in the neural network's vocabulary.</p>
  <pre><code>word_index = imdb.get_word_index()
  string = &quot;&quot;&quot;Fallout franchise had a few shifts in the lore and perception, resulting in even the fans arguing about &quot;what Fallout should be&quot;.
  
  While the series are not as gritty-depressive as Fallout 1-2, and followed a bit closer to the character's vision, the level of actual dedication to the franchise will make it liked by everyone!
  
  The props are amazing, the lore is very fitting, the characters are matching the universe. In-game brands and characters are exactly in place. It is no Master Chief from online store, but a great adaptation which does not retell the story of any of the games directly.
  
  The storytelling will both properly introduce the universe to those who had never heard about it and please the fans. It is rare. But what is even more outstanding - even if Fallout games wouldn't exist, the show is highly entertaining.
  
  Nowadays we see more and more game-sourced shows, and it is great Fallout joins Cyberpunk, Arcane and The Last of Us in the list.&quot;&quot;&quot;
  import re
  regex = re.compile(r&quot;[a-z]+&quot;, re.IGNORECASE)
  string_words = regex.findall(string)
  string_words_to_lower = []
  for word in string_words:
    index = word_index[word.lower()]
    if index &lt; 10000:
      string_words_to_lower.append(index)
  model.predict(vectorize_sequences([string_words_to_lower]))
  </code></pre>
  <p>The result is:</p>
  <p><strong>array([[0.00060081]], dtype=float32)</strong></p>
  <p>Yeeeeey! The neural network works, right?</p>
  <p>Now I take the second positive review from the IMDB page and change the string variable to it:</p>
  <pre><code>string = &quot;&quot;&quot;I was afraid this show will be weak or average, but it's simply awesome.
  
  Everything is very detailed, every item is like from Fallout, as if someone who knows the game created a movie - it's truly shocking that this is possible in 2024.
  
  The humor, the story, the mood, the music - everything is there!!!
  
  I was afraid that Lucy will be &quot;strong woman&quot; like in other modern shows but she is very cool and likeable. Of course Cooper is fantastic and Maximus is also great.
  
  Dear Amazon, dear owner - please cancel Rings of Power immediately and put all your money into the next seasons of Fallout!&quot;&quot;&quot;
  </code></pre>
  <p>I changed &quot;Jeff Bezos&quot; to &quot;owner&quot;.</p>
  <p>The result is:</p>
  <p><strong>array([[0.94335175]], dtype=float32)</strong></p>
  <p>This neural network can't find it's way out of the paper back. Why all those data was processed? What for? This &quot;neural network&quot; is just a random number generator. Why are we always fed with this garbage that does not work in the real world?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T16:12:30.623Z
Comments.Text: why are you so mad to tools that you seem to be not very familiar with? take your time to fully learn them, understand how they work and how to use them, they are algorithms, getting mad with them doesn't make a difference for them...
Comments.UserDisplayName: Alberto
Tags: <neural-networks><tensorflow>
Title: Why are the tutorials and built-in datasets giving us examples that simply do not work?
UserDisplayName: Jaroslav Tavgen
ThreadText: # Why are the tutorials and built-in datasets giving us examples that simply do not work?
  
  (Asked by: username_0 on 2024-04-25)
  
  I have built a classical neural network based on IMDB reviews according to the tutorial in one book about AI. 25 000 positive reviews, 25 000 negative reviews. Positive reviews result to "0", negative - to "1".
  
  Here is the code:
  
  ```
  from tensorflow.keras.datasets import imdb
  (train_data, train_label), (test_data, test_label) = imdb.load_data(num_words=10000)
  import numpy as np
  def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate (sequences):
      for j in sequence:
        results[i,j]=1
    return results
  x_train = vectorize_sequences(train_data)
  x_test = vectorize_sequences(test_data)
  y_train = np.asarray(train_label).astype('float32')
  y_test = np.asarray(test_label).astype('float32')
  from tensorflow import keras
  from tensorflow.keras import layers
  model = keras.Sequential([
      layers.Dense(16, activation="relu"),
      layers.Dense(16, activation="relu"),
      layers.Dense(1, activation="sigmoid")
  ])
  model.compile(optimizer='rmsprop',
                loss='binary_crossentropy',
                metrics=['accuracy']
  )
  x_val = x_train[:10000]
  partial_x_train = x_train[10000:]
  y_val = y_train[:10000]
  partial_y_train = y_train[10000:]
  history = model.fit(partial_x_train,
                      partial_y_train,
                      epochs=20,
                      batch_size=512,
                      validation_data=(x_val,y_val))
  results = model.evaluate(x_test, y_test)
  
  ```
  
  Now I go to the IMDB website to see user reviews of Fallout. <https://www.imdb.com/title/tt12637874/reviews?ref_=tt_urv>
  
  Then I copied the 10/10 review. Replaced the word "Bethesda" with "character", "Aliexpress" with "online store", since the words were not in the neural network's vocabulary.
  
  ```
  word_index = imdb.get_word_index()
  string = """Fallout franchise had a few shifts in the lore and perception, resulting in even the fans arguing about "what Fallout should be".
  
  While the series are not as gritty-depressive as Fallout 1-2, and followed a bit closer to the character's vision, the level of actual dedication to the franchise will make it liked by everyone!
  
  The props are amazing, the lore is very fitting, the characters are matching the universe. In-game brands and characters are exactly in place. It is no Master Chief from online store, but a great adaptation which does not retell the story of any of the games directly.
  
  The storytelling will both properly introduce the universe to those who had never heard about it and please the fans. It is rare. But what is even more outstanding - even if Fallout games wouldn't exist, the show is highly entertaining.
  
  Nowadays we see more and more game-sourced shows, and it is great Fallout joins Cyberpunk, Arcane and The Last of Us in the list."""
  import re
  regex = re.compile(r"[a-z]+", re.IGNORECASE)
  string_words = regex.findall(string)
  string_words_to_lower = []
  for word in string_words:
    index = word_index[word.lower()]
    if index < 10000:
      string_words_to_lower.append(index)
  model.predict(vectorize_sequences([string_words_to_lower]))
  
  ```
  
  The result is:
  
  **array([[0.00060081]], dtype=float32)**
  
  Yeeeeey! The neural network works, right?
  
  Now I take the second positive review from the IMDB page and change the string variable to it:
  
  ```
  string = """I was afraid this show will be weak or average, but it's simply awesome.
  
  Everything is very detailed, every item is like from Fallout, as if someone who knows the game created a movie - it's truly shocking that this is possible in 2024.
  
  The humor, the story, the mood, the music - everything is there!!!
  
  I was afraid that Lucy will be "strong woman" like in other modern shows but she is very cool and likeable. Of course Cooper is fantastic and Maximus is also great.
  
  Dear Amazon, dear owner - please cancel Rings of Power immediately and put all your money into the next seasons of Fallout!"""
  
  ```
  
  I changed "Jeff Bezos" to "owner".
  
  The result is:
  
  **array([[0.94335175]], dtype=float32)**
  
  This neural network can't find it's way out of the paper back. Why all those data was processed? What for? This "neural network" is just a random number generator. Why are we always fed with this garbage that does not work in the real world?
  
  **Tags:** neural-networks, tensorflow
  
  > **Comments:**
  > **username_1**: why are you so mad to tools that you seem to be not very familiar with? take your time to fully learn them, understand how they work and how to use them, they are algorithms, getting mad with them doesn't make a difference for them...  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-25. Score: 2
  
  A tutorial is not keys to a production grade system. They are just to get one familiar with the basic concepts. To move toward a system that works on more than the provided examples, takes quite a bit of research and experience.
  
  For example:
  
  * Sigmoid vs Softmax. Sigmoid activation in classification is often confidently wrong. Softmax tends to be a little bit more stable.
  * Two dense layers with 16 neurons are unlikely to be effective on more than a few tokens. Transformers have been developed for a reason.
  * Nothing has been said about fit or training accuracy or any cross validation that was performed. These concerns give a hint about stability of a model. Without it I cannot imagine expecting a model to preform reliably.
  
  This list is certainly not comprehensive, text processing is almost a field unto it's own. I am not an expert and this is not the forum to debug one's code.
  
  
  ---
  
  