use a scheduler with a validation metric

Setting up an optimizer lr reducer

scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.5,
                              patience=int(params['patience']),
                              verbose=True)
# Making sure that the learning rate scheduler keeps track
def testfunc():
    """
    .................
    """
    scheduler.step(test_average_loss)

Try this for lr decay.