The determinism also depends on how many times did you call the random number generator. In you examples, there are two models, which are functionally the same. However, their initialization is different, because the number of random values generated after the seeding is different.
Here is what you are doing for the first model:

Set the seed
Create linear layers with random weights 4 times (I am assuming num_layers = 4)
Initialize the weights of the first layer

The number of times you call the random number generator before the kaiming_normal is 4xNumber of elements in linear layers in the encoder.
Now here is what you do in the second model:

Set the seed
Create a single linear layer with random weights
Initialize the weights of that layer

The number of times you call the random number generator before the kaiming_normal is 1xNumber of elements in linear layers in the encoder. That would create different random numbers. If you want your initialization to be completely identical, you will have to either 1) call the linear constructor in the second case the same number of times as in the first case (including the arguments) or 2) Set the seed right before the kaiming_normal_.
P.S. If you want the question to get answered faster, you might need to change the tag. This question has nothing to do with quantization, so its visibility is limited to a wrong group