This is a tricky situation. However, if you have a large set of speakers, then on-average you have an “expired” embedding. So the network will likely regularize for this effect implicitly. I’m not convinced that you’d need to do anything explicitly.