In your desired architecture the inputs of the layers are a combination of speaker code and feature previous layer outputs or the feature vector input. I guess you want to condition on the speaker code.
You could pass the speaker code as additional input, e.g. by concatenating it to the regular layer input. Another very simple method is to use addition to mix in the speaker code. From your diagram I cannot see what kind of network you want, e.g. if the red layers are fully-connected layers or conv layers. The first thing you could do is to write down all the tensor dimensionalities and then think about bringing them into a ‘compatible form’, e.g. if the dimensionality does not match you could try use a linear layer to adapt it.