Solved by ptrblck in post #2 
              
              
                Try to overfit a small dataset, e.g. just 10 samples, by playing around with the hyperparameters. 
Once your model is able to do so scale up the use case again.