Which experiments did you already try? Your idea of a sanity check is reasonable and I cannot find an obvious bug by skimming through your code.
Did you already changed the learning rate and checked, if the loss behavior changes?
Also, are the gradients blowing up?