Solved by JuanFMontesinos in post #2 
              
              
                One simple way is using FiLM (Feature-wise layer modulation) 
 
 
Sooo if you are using a U-Net this has already been done for audio 
Here you have code.https://github.com/gabolsgabs/cunet 
This way you can simply condition inner layers of U-Net with a one-hot vector (or any tensor if you have some …