<p>I am currently learning neural networks using data from <a href="https://www.mariofrank.net/touchalytics/index.html" rel="nofollow noreferrer">Touchscreen Input as a Behavioral Biometric</a>. Basically, I am trying to predict &quot;User ID&quot; by training the neural network model shown below.</p>
<pre><code>import time
import os
BATCH_SIZE=32
embedding_size=256
sequence_length=200
BUFFER_SIZE=10000
input_size=41
learning_rate=0.001

inputs_as_tensors=tf.data.Dataset.from_tensor_slices(train_data_features_array)
targets_as_tensors=tf.data.Dataset.from_tensor_slices(train_data_labels_categorical_array)
training_data=tf.data.Dataset.zip((inputs_as_tensors,targets_as_tensors))
#training_data=training_data.batch(sequence_length,drop_remainder=True)
training_dataset=training_data.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
print(training_dataset)

def build_model(vocab_size,batch_size):
    modelf=tf.keras.Sequential([
        tf.keras.layers.Dense(10,activation=&quot;sigmoid&quot;,input_shape=(None,10)),
        
                                tf.keras.layers.Dense(30,activation=&quot;relu&quot;,use_bias=True),  
                                 
                                     tf.keras.layers.Dropout(0.2),                    
                                   tf.keras.layers.Dense(vocab_size)
                          
                               
                                ])
    return modelf

def training_step(inputs,targets,optimizer):
    with tf.GradientTape() as tape:
        predictions=model(inputs)
        loss=tf.reduce_mean(tf.keras.losses.categorical_crossentropy(targets,predictions,from_logits=True))
        
        grads=tape.gradient(loss,model.trainable_variables)
        optimizer.apply_gradients(zip(grads,model.trainable_variables))
        return loss,predictions

model=build_model(input_size,BATCH_SIZE)
i=0
inner_loop=0
checkpoint_dir ='Moses_Model_x'
checkpoint_prefix = os.path.join(checkpoint_dir, &quot;ckpt_{i}&quot;)

while(1):
    start = time.time()
    for x,y in training_dataset:
        loss,predictions=training_step(x,y,tf.keras.optimizers.RMSprop(learning_rate=0.002))
    print ('Epoch {} Loss {:.4f}'.format(i, loss))
    print ('Time taken for iteration {} is {} sec\n'.format(i,time.time() - start))
    model.save_weights(checkpoint_prefix.format(i=i))
    i=i+1
</code></pre>
<p>However, the loss value is actually increasing. Is there anything I have to change or something wrong in my code?</p>
<p><a href="https://i.sstatic.net/9fzMn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9fzMn.png" alt="Loss graph (loss over epoch)" /></a></p>

CC BY-SA 4.0
2020-03-22T20:11:30.610Z
Can you provide a plot? It’s normal for loss to fluctuate from batch to batch. How does your loss behave from epoch to epoch?
SpiderRico
CC BY-SA 4.0
2020-03-23T09:17:18.410Z
@SpiderRico please check the updates
onexpeters
CC BY-SA 4.0
2020-03-23T22:51:18.937Z
Can you please describe exactly what your neural network is supposed to do? Which task are you trying to solve? What does the dataset contain? Which loss are you using? Some of these questions may be answered by looking at the code, but, ideally, on this site, we should be able to answer your question without looking at the code, so provide these details, please!
nbro
<neural-networks><machine-learning><objective-functions>
Why is the loss associated with my neural network increasing?
onexpeters
# Why is the loss associated with my neural network increasing?

(Asked by: username_0 on 2020-03-22)

I am currently learning neural networks using data from Touchscreen Input as a Behavioral Biometric. Basically, I am trying to predict "User ID" by training the neural network model shown below.
import time
import os
BATCH_SIZE=32
embedding_size=256
sequence_length=200
BUFFER_SIZE=10000
input_size=41
learning_rate=0.001

inputs_as_tensors=tf.data.Dataset.from_tensor_slices(train_data_features_array)
targets_as_tensors=tf.data.Dataset.from_tensor_slices(train_data_labels_categorical_array)
training_data=tf.data.Dataset.zip((inputs_as_tensors,targets_as_tensors))
#training_data=training_data.batch(sequence_length,drop_remainder=True)
training_dataset=training_data.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
print(training_dataset)

def build_model(vocab_size,batch_size):
    modelf=tf.keras.Sequential([
        tf.keras.layers.Dense(10,activation="sigmoid",input_shape=(None,10)),
        
                                tf.keras.layers.Dense(30,activation="relu",use_bias=True),  
                                 
                                     tf.keras.layers.Dropout(0.2),                    
                                   tf.keras.layers.Dense(vocab_size)
                          
                               
                                ])
    return modelf

def training_step(inputs,targets,optimizer):
    with tf.GradientTape() as tape:
        predictions=model(inputs)
        loss=tf.reduce_mean(tf.keras.losses.categorical_crossentropy(targets,predictions,from_logits=True))
        
        grads=tape.gradient(loss,model.trainable_variables)
        optimizer.apply_gradients(zip(grads,model.trainable_variables))
        return loss,predictions

model=build_model(input_size,BATCH_SIZE)
i=0
inner_loop=0
checkpoint_dir ='Moses_Model_x'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{i}")

while(1):
    start = time.time()
    for x,y in training_dataset:
        loss,predictions=training_step(x,y,tf.keras.optimizers.RMSprop(learning_rate=0.002))
    print ('Epoch {} Loss {:.4f}'.format(i, loss))
    print ('Time taken for iteration {} is {} sec\n'.format(i,time.time() - start))
    model.save_weights(checkpoint_prefix.format(i=i))
    i=i+1

However, the loss value is actually increasing. Is there anything I have to change or something wrong in my code?

**Tags:** neural-networks, machine-learning, objective-functions

> **Comments:**
> **username_1**: Can you provide a plot? It’s normal for loss to fluctuate from batch to batch. How does your loss behave from epoch to epoch?  
> **username_0**: @username_1 please check the updates  
> **username_2**: Can you please describe exactly what your neural network is supposed to do? Which task are you trying to solve? What does the dataset contain? Which loss are you using? Some of these questions may be answered by looking at the code, but, ideally, on this site, we should be able to answer your question without looking at the code, so provide these details, please!  

---

## 0 answers

