Some side notes:

Did you preprocess your input images? It seems the tensorflow implementation 5 does a zero-centering.
Did you initialize your new Linear layers? Some Xavier init and a small constant is used for the weights and bias, respectively. (It seems the default is xavier uniform, when no args are passed.)