Thanks for the answer,
I misunderstood some concept but after reading again it’s more clear 
You’re solution is working great, I’m just doing the “Adaptative”  layer on my own after a classic instance norm.
Thanks again