I cannot spot anything obviously wrong, but would start by checking all shapes, as your current code is using some permutations and reshapes.
Try to add comments with the dimensions to all lines and make sure that the layers are getting the tensors in the expected shapes.