I removed Relu (dont have a clear Idea as to why, maybe something related to positive and negative outputs) and somehow the overall loss is decreasing now,  but the Accuracy for the classifier (Fscore) decreased a little bit then became constant.