You might want to add activation functions into classifier_layer. If that doesn’t help, try to overfit a small data sample (e.g. just 10 samples) by playing around with some hyperparameters. Once this is done you could try to scale up the use case again.