Hi kk!
I built a simple (just for practice) network in tensorflow – two
fully-connected layers with biases.  I successfully rewrote it in
pytorch and got statistically the same results.
I could not find matching built-in weight-initialization methods.
(The weights and/or biases differed one way or another.)  So I
initialized my weights and biases by hand.
Tensorflow was choosing my batches (stochastic gradient
descent), while in pytorch I was choosing my own.  I didn’t
bother trying to make the batch selection match exactly.
So specific numbers generated by tensorflow and pytorch
differed randomly, but, as I said above, they were statistically
the same.
(In my case, it took me a little while to realize that the seemingly
same weight-initialization methods has slightly different recipes
for initializing the biases.)
Yours is a much more complicated network than my toy model,
but I do believe that you should be able to get them to give you
the same results.
Good luck.
K. Frank