Thank You for your reply I am working on a multi-task learning problem and this is a model for one of the task.The input to the model is a 3D tensor . I was using the InstanceNorm Layer since i read  in one of the blogs that softmax might make the gradient magnitude small, so its better to use some normalization before doing softmax. I can add more layers to the embedding layer if that would make the gradients increase?