Turns out that a low gradient flowing through the network, in this case, indicates that the image features do not contain interesting information. Upon further checking, I realize that my xavier initialization of the model forgets to exclude the pre-trained resnet, which obviously destroys all the image features. With the inputs to the Transformer being random values, the model learns to ignore the input altogether.