Hello,
I have been training recently a model using different RNNs (simple RNN, LSTMs, GRU), so far the best one was a GRU with one layer dimension and 256 hidden dimensions giving me 87.5% OA and 76.5% MIOU so here are my questions :
1- I have been using a learning rate equal to 0.1 that changes on a certain epoch ( I used 200 epoch for training and I changed on the 140th epoch by learning _rate_new = learning rate_old * decay rate ( 0.1 * 0.7) I only did this once on the 140th epoch not repeatedly, so my question is there a better method for updating the learning rate using a built-in function or another way.
2- for the optimizer weights, I used a tensor of the inverse squared of the classes frequencies in my training dataset ( I read it in another forum that it performs well), Is there a better way for doing this?
and thank you , I just want to push the envelope one last bit and get that sweat 90% OA without overfitting.