I once had a similar problem where my predictions would just freak out. I lowered the learning rate and boom done, spent too much time debugging that…
I suggest that you try to overfit to a subset of your dataset (like 100 images) with a lower learning rate(0.001). Report back