Body: <h2>Problem description</h2>
  <p>I'm creating a clock with 4 seven-segment LED displays. In an effort to get more familiar with tensorflow, I figured I should try to drive this clock with use of a Neural Network.</p>
  <p>The input of the network is the Unix time.
  Initially I wanted to make the network output a UINT32 value, which I can then bit-shift into the shift-registers I use for driving the LEDS. Because this proved to be unsuccessful, I removed the last step, and instead went with 32 booleans for every led-segment as output.</p>
  <h3>Current status</h3>
  <p>This last action was unfruitful as well, the best I get my network is a loss of about 0.48, indicating to me that it's best effort is to guess what the output could be.</p>
  <h3>Input</h3>
  <pre><code>print(x_validate[0])
  print(y_validate[0])
  3116360099
  [0 1 1 0 0 1 1 0 1 1 1 1 0 0 1 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0]
  </code></pre>
  <p>The input is the unix time code. I tried normalizing this by dividing by 2^32, but this didn't make any significant difference.
  Validation is an array with either 0 or 1 based on whether the LED needs to be on or off. The first 8 bits represent the first 7-segment display, etc. (The 8th bit is never used here because that's connected to the dot on the display.)</p>
  <h3>Data generation</h3>
  <pre><code># Number of sample datapoints
  SAMPLES = 2000000
  
  # Generate a uniformly distributed set of random numbers in the range from
  # 0 to uin32 max
  x_values = np.random.uniform(
      low=0, high=2**32, size=SAMPLES).astype(np.uint32)
  
  # Shuffle the values to guarantee they're not in order
  np.random.shuffle(x_values)
  
  print(x_values)
  
  # Time helper function 
  def to_utc(x):
    return datetime.utcfromtimestamp(x).replace(tzinfo=timezone.utc).astimezone(pytz.timezone(&quot;Europe/Amsterdam&quot;)).strftime('%H%M')
  to_utc_batch = np.vectorize(to_utc)
  
  y_values = to_utc_batch(x_values)
  print(y_values)
  
  # translate to bitstream
  def lookup(number):
    switch = {
        0: [1,1,1,1,1,1,0,0],
        1: [0,1,1,0,0,0,0,0],
        2: [1,1,0,1,1,0,1,0],
        3: [1,1,1,1,0,0,1,0],
        4: [0,1,1,0,0,1,1,0],
        5: [1,0,1,1,0,1,1,0],
        6: [1,0,1,1,1,1,1,0],
        7: [1,1,1,0,0,0,0,0],
        8: [1,1,1,1,1,1,1,0],
        9: [1,1,1,1,0,1,1,0]
    }
    return switch.get(number)
  
  print(y_values)
  
  def compile_output(value):
    f = []
    for i, c in reversed(list(enumerate(value))):
      f = f + lookup(int(c))
    return f
  
  output_values = []
  
  for y in y_values:
    output_values.append(compile_output(y))
  
  y_values = output_values
  </code></pre>
  <p>After, data is distributed in 3 sets:</p>
  <pre><code>Training: 60%
  Validation: 20%
  Testing: 20%
  </code></pre>
  <h3>Model</h3>
  <pre><code>model = tf.keras.Sequential()
  model.add(keras.layers.Dense(32, activation='relu', input_shape=(1,)))
  model.add(keras.layers.Dense(64, activation='relu'))
  model.add(keras.layers.Dense(64, activation='relu'))
  
  # Final layer
  model.add(keras.layers.Dense(32, activation='sigmoid'))
  
  model.compile( optimizer='adam', loss='binary_crossentropy', metrics=[&quot;accuracy&quot;])
  </code></pre>
  <p>I went with binary_crossentropy and sigmoid as I figured the case is essentially a multi-label setup.</p>
  <p>I have tried the following already, but did not succeed:</p>
  <ul>
  <li>Add more layers</li>
  <li>Make dense layers wider (to max of 512)</li>
  <li>Add a Dropout layer -&gt; so to have it try out more things to find the relation</li>
  <li>Use Softmax activation</li>
  <li>Normalize input data by dividing Unix time by 2^32</li>
  <li>enlarge sample data to 20.000.000</li>
  <li>do anything between 10 and 50 epochs ( I usually quit after 15 epochs, when absolutely no change was observed between the last 5 epochs)</li>
  </ul>
  <h2>Question</h2>
  <ul>
  <li>Why is this model not successful in finding a relation between the data?</li>
  <li>How can I improve this model or the data so it will be able to succeed?</li>
  </ul>
  <h3>Bonus</h3>
  <p>When successful, ideally the output of the model would be a UINT32 number instead of this array of booleans. Any tips on how to get there would be appreciated as well.</p>
  <p><em>Edit:</em>
  Really sorry, left out this particular line:</p>
  <pre><code># Train the model
  history = model.fit(x_train, y_train, epochs=10, batch_size=64,
                      validation_data=(x_validate, y_validate))
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T11:45:21.853Z
Comments.Text: What is your batch size? What is your learning rate? Very generally, those two are good parameters to play around with first, and you don't mention them neither in code nor text (unless I'm very blind). I don't think you need more layers or bigger layers. Why are your inputs so high? If I'm not mistaken, you have 24*60=1440 different options for clock displays, and if not you should be able to do modulo to bring it into this range, as you're only interested in the time, right? Please clarify if I'm misunderstanding your idea.
Comments.UserDisplayName: Mafu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T12:10:17.070Z
Comments.Text: @Mafu, Thank you for your comment! Sorry I forgot to specify the batch size, added it now. Learning rate I didn't specify, I guess it's using the default for Adam? 
  
  Do you mean high numbers as input? That's because the Unix time code is a 32-bit value, and the idea is that the model will be able to deduce the time-setting from any arbitrary time code.
Comments.UserDisplayName: Micha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T23:03:43.913Z
Comments.Text: Ok, I think you should really try varying the learning rate. That's one of the most important tuning factors. According to the [Tensorflow Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam), the default is 0.001, which in many of my applications has been too high. Try going with 1e-4, 1e-5, also 1e-2, see how it changes the learning curve and the results. From that you should be able to deduce useful conclusions.
Comments.UserDisplayName: Mafu
Tags: <machine-learning><tensorflow>
Title: NN to find arbitrary transformation
UserDisplayName: Micha
ThreadText: # NN to find arbitrary transformation
  
  (Asked by: username_0 on 2021-02-02)
  
  ## Problem description
  
  I'm creating a clock with 4 seven-segment LED displays. In an effort to get more familiar with tensorflow, I figured I should try to drive this clock with use of a Neural Network.
  
  The input of the network is the Unix time.
  Initially I wanted to make the network output a UINT32 value, which I can then bit-shift into the shift-registers I use for driving the LEDS. Because this proved to be unsuccessful, I removed the last step, and instead went with 32 booleans for every led-segment as output.
  
  ### Current status
  
  This last action was unfruitful as well, the best I get my network is a loss of about 0.48, indicating to me that it's best effort is to guess what the output could be.
  
  ### Input
  
  ```
  print(x_validate[0])
  print(y_validate[0])
  3116360099
  [0 1 1 0 0 1 1 0 1 1 1 1 0 0 1 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0]
  
  ```
  
  The input is the unix time code. I tried normalizing this by dividing by 2^32, but this didn't make any significant difference.
  Validation is an array with either 0 or 1 based on whether the LED needs to be on or off. The first 8 bits represent the first 7-segment display, etc. (The 8th bit is never used here because that's connected to the dot on the display.)
  
  ### Data generation
  
  ```
  # Number of sample datapoints
  SAMPLES = 2000000
  
  # Generate a uniformly distributed set of random numbers in the range from
  # 0 to uin32 max
  x_values = np.random.uniform(
      low=0, high=2**32, size=SAMPLES).astype(np.uint32)
  
  # Shuffle the values to guarantee they're not in order
  np.random.shuffle(x_values)
  
  print(x_values)
  
  # Time helper function 
  def to_utc(x):
    return datetime.utcfromtimestamp(x).replace(tzinfo=timezone.utc).astimezone(pytz.timezone("Europe/Amsterdam")).strftime('%H%M')
  to_utc_batch = np.vectorize(to_utc)
  
  y_values = to_utc_batch(x_values)
  print(y_values)
  
  # translate to bitstream
  def lookup(number):
    switch = {
        0: [1,1,1,1,1,1,0,0],
        1: [0,1,1,0,0,0,0,0],
        2: [1,1,0,1,1,0,1,0],
        3: [1,1,1,1,0,0,1,0],
        4: [0,1,1,0,0,1,1,0],
        5: [1,0,1,1,0,1,1,0],
        6: [1,0,1,1,1,1,1,0],
        7: [1,1,1,0,0,0,0,0],
        8: [1,1,1,1,1,1,1,0],
        9: [1,1,1,1,0,1,1,0]
    }
    return switch.get(number)
  
  print(y_values)
  
  def compile_output(value):
    f = []
    for i, c in reversed(list(enumerate(value))):
      f = f + lookup(int(c))
    return f
  
  output_values = []
  
  for y in y_values:
    output_values.append(compile_output(y))
  
  y_values = output_values
  
  ```
  
  After, data is distributed in 3 sets:
  
  ```
  Training: 60%
  Validation: 20%
  Testing: 20%
  
  ```
  
  ### Model
  
  ```
  model = tf.keras.Sequential()
  model.add(keras.layers.Dense(32, activation='relu', input_shape=(1,)))
  model.add(keras.layers.Dense(64, activation='relu'))
  model.add(keras.layers.Dense(64, activation='relu'))
  
  # Final layer
  model.add(keras.layers.Dense(32, activation='sigmoid'))
  
  model.compile( optimizer='adam', loss='binary_crossentropy', metrics=["accuracy"])
  
  ```
  
  I went with binary\_crossentropy and sigmoid as I figured the case is essentially a multi-label setup.
  
  I have tried the following already, but did not succeed:
  
  * Add more layers
  * Make dense layers wider (to max of 512)
  * Add a Dropout layer -> so to have it try out more things to find the relation
  * Use Softmax activation
  * Normalize input data by dividing Unix time by 2^32
  * enlarge sample data to 20.000.000
  * do anything between 10 and 50 epochs ( I usually quit after 15 epochs, when absolutely no change was observed between the last 5 epochs)
  
  ## Question
  
  * Why is this model not successful in finding a relation between the data?
  * How can I improve this model or the data so it will be able to succeed?
  
  ### Bonus
  
  When successful, ideally the output of the model would be a UINT32 number instead of this array of booleans. Any tips on how to get there would be appreciated as well.
  
  *Edit:*
  Really sorry, left out this particular line:
  
  ```
  # Train the model
  history = model.fit(x_train, y_train, epochs=10, batch_size=64,
                      validation_data=(x_validate, y_validate))
  
  ```
  
  **Tags:** machine-learning, tensorflow
  
  > **Comments:**
  > **username_1**: What is your batch size? What is your learning rate? Very generally, those two are good parameters to play around with first, and you don't mention them neither in code nor text (unless I'm very blind). I don't think you need more layers or bigger layers. Why are your inputs so high? If I'm not mistaken, you have 24*60=1440 different options for clock displays, and if not you should be able to do modulo to bring it into this range, as you're only interested in the time, right? Please clarify if I'm misunderstanding your idea.  
  > **username_0**: @username_1, Thank you for your comment! Sorry I forgot to specify the batch size, added it now. Learning rate I didn't specify, I guess it's using the default for Adam?   Do you mean high numbers as input? That's because the Unix time code is a 32-bit value, and the idea is that the model will be able to deduce the time-setting from any arbitrary time code.  
  > **username_1**: Ok, I think you should really try varying the learning rate. That's one of the most important tuning factors. According to the [Tensorflow Documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam), the default is 0.001, which in many of my applications has been too high. Try going with 1e-4, 1e-5, also 1e-2, see how it changes the learning curve and the results. From that you should be able to deduce useful conclusions.  
  
  ---
  
  ## 0 answers
  
  