Answers.Body: <p>I've discovered Doc2Vec which does something similar to what I am trying to accomplish. This doesn't <em>exactly</em> answer my question of why the network I was trying to build doesn't work, but at least it shows how indexed outputs can be pulled from a network, with open source to show how it is built.</p>
  
  <p><a href="https://datascience.stackexchange.com/questions/23969/sentence-similarity-prediction">https://datascience.stackexchange.com/questions/23969/sentence-similarity-prediction</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-02T05:28:26.070Z
Answers.LastActivityDate: 2019-01-02T05:28:26.070Z
Answers.UserDisplayName: John
Body: <p>I am trying to use a artifical neural network to produce a single output, which in my mind should be an index into a list of data (or close to it). All of the results I get are 0.9999+ and very close to each other. I don't know if my whole way of thinking here is off, or if I am just missing an approach, or if perhaps my network code is just broken.</p>
  
  <p>I am trying to make use of the simple neural network from Microsoft here:
  <a href="https://social.technet.microsoft.com/wiki/contents/articles/36428.basis-of-neural-networks-in-c.aspx" rel="nofollow noreferrer">https://social.technet.microsoft.com/wiki/contents/articles/36428.basis-of-neural-networks-in-c.aspx</a></p>
  
  <p>I have tried this with a significantly more complex data set, but I've also tried using a very simple data set. </p>
  
  <p>Here is the simple training data I'm trying to use:</p>
  
  <pre><code>eat poo bad
  eat dirt bad
  eat cookies okay
  eat fruit good
  study poo okay
  study dirt okay
  study cookies okay
  study fruit okay
  dispose poo good
  dispose dirt okay
  dispose cookies bad
  dispose fruit bad
  </code></pre>
  
  <p>The basic idea is that the network has two input neurons and a single output neuron. I assigned a unique number to each distinct word such that I can train the network with two inputs (verb and object), and expect a single output (good, bad, or okay).</p>
  
  <p>Example training:</p>
  
  <pre><code>input: 1 (for eat) 10 (for dirt) output: 15 (for bad)
  input: 1 (for eat) 11 (for cookies) output: 16 (for good)
  </code></pre>
  
  <p>I would expect that after training, I would see the output numbers close to 15, 16, etc, but all I get are numbers like 0.999997333313168, etc.</p>
  
  <p>Example run:</p>
  
  <pre><code>input: 1 (for eat) 10 (for dirt), output is 0.999997333313168 (instead of ~15 expected)
  </code></pre>
  
  <p>What do these outputs mean, or what am I missing in how I should be thinking about making a basic classification system (given inputs, get a meaningful output)?</p>
  
  <p>The C# code I am using, if it is helpful:</p>
  
  <pre><code>using NeuralNet.NeuralNet;
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  
  namespace NeuralNet
  {
      internal class TestSmallSampleNetwork
      {
          internal static void Run()
          {
              var data = @"eat poo bad
                          eat dirt bad
                          eat cookies okay
                          eat fruit good
                          sell poo bad
                          sell dirt okay
                          sell cookies okay
                          sell fruit okay
                          study poo okay
                          study dirt okay
                          study cookies okay
                          study fruit okay
                          dispose poo good
                          dispose dirt okay
                          dispose cookies bad
                          dispose fruit bad";
              var simples = data.Split(new[] { "\r\n" }, StringSplitOptions.None ).Select(_ =&gt; new Simple(_)).ToList();
  
              var verbs = simples.Select(_ =&gt; _.Verb).Distinct().Select(_ =&gt; new NetworkValue(_)).ToList();
              var objects = simples.Select(_ =&gt; _.Object).Distinct().Select(_ =&gt; new NetworkValue(_)).ToList();
              var judgments = simples.Select(_ =&gt; _.Good).Distinct().Select(_ =&gt; new NetworkValue(_)).ToList();
              var values = verbs.Concat(objects).Concat(judgments).ToDictionary(_ =&gt; _.Term, _ =&gt; _);
  
              // Create a network with 2 inputs, 2 neurons on a single hidden layer, and 1 neuron output.
              var net = new NeuralNetwork(0.9, new int[] { 2, 2, 1 });
  
              for (int iTrain = 0; iTrain &lt; 1000; iTrain++)
              {
                  for (int iSimple = 0; iSimple &lt; simples.Count; iSimple++)
                  {
                      net.Train(MakeInputs(simples[iSimple], values), MakeOutputs(simples[iSimple], values));
                  }
              }
  
              foreach (var value in values.Values)
              {
                  Console.WriteLine(value);
              }
  
              Console.WriteLine();
  
              // Run samples and get results back from the network
              Run("study", "poo", values, net);
              Run("eat", "poo", values, net);
              Run("dispose", "fruit", values, net);
              Run("sell", "dirt", values, net);
          }
  
          private static void Run(string verb, string obj, Dictionary&lt;string, NetworkValue&gt; values, NeuralNetwork net)
          {
              var result = net.Run(new List&lt;double&gt; {
                  values[verb].Value,
                  values[obj].Value,
              }).Single();
  
              var good = "xxx";
  
              Console.WriteLine($"{verb} {obj} {good} ({result})");
          }
  
          private static List&lt;double&gt; MakeInputs(Simple simple, Dictionary&lt;string, NetworkValue&gt; values)
          {
              return new List&lt;double&gt;() {
                  values[simple.Verb].Value,
                  values[simple.Object].Value
              };
          }
  
          private static List&lt;double&gt; MakeOutputs(Simple simple, Dictionary&lt;string, NetworkValue&gt; values)
          {
              return new List&lt;double&gt; { values[simple.Good].Value };
          }
  
          public class Simple
          {
              public string Verb { get; set; }
              public string Object { get; set; }
              public string Good { get; set; }
  
              public Simple(string line)
              {
                  var words = line.Trim().Split(" ".ToCharArray(), 3, StringSplitOptions.None);
                  Verb = words[0];
                  Object = words[1];
                  Good = words[2];
              }
  
              public override string ToString()
              {
                  return $"{Verb} {Object} {Good}";
              }
          }
  
          public class NetworkValue
          {
              private static int Next = 1;
  
              public string Term { get; set; }
              public double Value { get; set; }
  
              public NetworkValue(string term)
              {
                  Term = term;
                  Value = Next++;
              }
  
              public override string ToString()
              {
                  return $"{Value}. {Term}";
              }
          }
      }
  }
  </code></pre>
  
Tags: <neural-networks>
Title: How do I get a meaningful output value for a simple neural network that can map to a set of data?
UserDisplayName: John
ThreadText: # How do I get a meaningful output value for a simple neural network that can map to a set of data?
  
  (Asked by: username_0 on 2018-12-25)
  
  I am trying to use a artifical neural network to produce a single output, which in my mind should be an index into a list of data (or close to it). All of the results I get are 0.9999+ and very close to each other. I don't know if my whole way of thinking here is off, or if I am just missing an approach, or if perhaps my network code is just broken.
  I am trying to make use of the simple neural network from Microsoft here:
  https://social.technet.microsoft.com/wiki/contents/articles/36428.basis-of-neural-networks-in-c.aspx
  I have tried this with a significantly more complex data set, but I've also tried using a very simple data set. 
  Here is the simple training data I'm trying to use:
  eat poo bad
  eat dirt bad
  eat cookies okay
  eat fruit good
  study poo okay
  study dirt okay
  study cookies okay
  study fruit okay
  dispose poo good
  dispose dirt okay
  dispose cookies bad
  dispose fruit bad
  
  The basic idea is that the network has two input neurons and a single output neuron. I assigned a unique number to each distinct word such that I can train the network with two inputs (verb and object), and expect a single output (good, bad, or okay).
  Example training:
  input: 1 (for eat) 10 (for dirt) output: 15 (for bad)
  input: 1 (for eat) 11 (for cookies) output: 16 (for good)
  
  I would expect that after training, I would see the output numbers close to 15, 16, etc, but all I get are numbers like 0.999997333313168, etc.
  Example run:
  input: 1 (for eat) 10 (for dirt), output is 0.999997333313168 (instead of ~15 expected)
  
  What do these outputs mean, or what am I missing in how I should be thinking about making a basic classification system (given inputs, get a meaningful output)?
  The C# code I am using, if it is helpful:
  using NeuralNet.NeuralNet;
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  
  namespace NeuralNet
  {
      internal class TestSmallSampleNetwork
      {
          internal static void Run()
          {
              var data = @"eat poo bad
                          eat dirt bad
                          eat cookies okay
                          eat fruit good
                          sell poo bad
                          sell dirt okay
                          sell cookies okay
                          sell fruit okay
                          study poo okay
                          study dirt okay
                          study cookies okay
                          study fruit okay
                          dispose poo good
                          dispose dirt okay
                          dispose cookies bad
                          dispose fruit bad";
              var simples = data.Split(new[] { "\r\n" }, StringSplitOptions.None ).Select(_ => new Simple(_)).ToList();
  
              var verbs = simples.Select(_ => _.Verb).Distinct().Select(_ => new NetworkValue(_)).ToList();
              var objects = simples.Select(_ => _.Object).Distinct().Select(_ => new NetworkValue(_)).ToList();
              var judgments = simples.Select(_ => _.Good).Distinct().Select(_ => new NetworkValue(_)).ToList();
              var values = verbs.Concat(objects).Concat(judgments).ToDictionary(_ => _.Term, _ => _);
  
              // Create a network with 2 inputs, 2 neurons on a single hidden layer, and 1 neuron output.
              var net = new NeuralNetwork(0.9, new int[] { 2, 2, 1 });
  
              for (int iTrain = 0; iTrain < 1000; iTrain++)
              {
                  for (int iSimple = 0; iSimple < simples.Count; iSimple++)
                  {
                      net.Train(MakeInputs(simples[iSimple], values), MakeOutputs(simples[iSimple], values));
                  }
              }
  
              foreach (var value in values.Values)
              {
                  Console.WriteLine(value);
              }
  
              Console.WriteLine();
  
              // Run samples and get results back from the network
              Run("study", "poo", values, net);
              Run("eat", "poo", values, net);
              Run("dispose", "fruit", values, net);
              Run("sell", "dirt", values, net);
          }
  
          private static void Run(string verb, string obj, Dictionary values, NeuralNetwork net)
          {
              var result = net.Run(new List {
                  values[verb].Value,
                  values[obj].Value,
              }).Single();
  
              var good = "xxx";
  
              Console.WriteLine($"{verb} {obj} {good} ({result})");
          }
  
          private static List MakeInputs(Simple simple, Dictionary values)
          {
              return new List() {
                  values[simple.Verb].Value,
                  values[simple.Object].Value
              };
          }
  
          private static List MakeOutputs(Simple simple, Dictionary values)
          {
              return new List { values[simple.Good].Value };
          }
  
          public class Simple
          {
              public string Verb { get; set; }
              public string Object { get; set; }
              public string Good { get; set; }
  
              public Simple(string line)
              {
                  var words = line.Trim().Split(" ".ToCharArray(), 3, StringSplitOptions.None);
                  Verb = words[0];
                  Object = words[1];
                  Good = words[2];
              }
  
              public override string ToString()
              {
                  return $"{Verb} {Object} {Good}";
              }
          }
  
          public class NetworkValue
          {
              private static int Next = 1;
  
              public string Term { get; set; }
              public double Value { get; set; }
  
              public NetworkValue(string term)
              {
                  Term = term;
                  Value = Next++;
              }
  
              public override string ToString()
              {
                  return $"{Value}. {Term}";
              }
          }
      }
  }
  
  **Tags:** neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-01-02. Score: 1
  
  I've discovered Doc2Vec which does something similar to what I am trying to accomplish. This doesn't *exactly* answer my question of why the network I was trying to build doesn't work, but at least it shows how indexed outputs can be pulled from a network, with open source to show how it is built.
  
  <https://datascience.stackexchange.com/questions/23969/sentence-similarity-prediction>
  
  
  ---
  
  