Each call into the pseudo-random number generator will increase its state.
Your model creation is most likely calling into a the pseudo-random number generator at some point, which will make all following (random) calls different.
A simple code snippet is here:
torch.manual_seed(2809)

lin1 = nn.Linear(100, 100)
# Comment out lin2
lin2 = nn.Linear(100, 100) # another call to the PRNG
for _ in range(10):
    print(torch.randn(1))

Each run yields deterministic results for itself, but if you comment lin2, the PRNG will not be called for the model instantiation, and thus the following torch.randn calls will still yield deterministic results, but different values compared to the first run.
Have a look at the Wikipedia article about PRNG 1 for more information.
If you need deterministic results for your workflow, I would recommend to initialize both models once and store their state_dicts.
In the test scripts create the model, load their state_dicts, and set the manual seeds afterwards to reset the PRNG for the training.
Note, since the forward methods of both models might also contain calls to the PRNG, the next iteration might again not yield the same batch.
If you see this behavior, you would have to make sure to use exactly the same call sequence and discard one model, if its output is not needed for the run.