Although I am not sure if this is the reason but your learning rate = 0.01 which seems a bit high. Whenever I have used a learning rate of 0.01 with Adam optimizer I normally get NaN very early in the training. But I was using different architecture and Tensorflow framework. Just try reducing your learning rate and see if it helps.