Solved by albanD in post #4 
              
              
                So here you have two parameters in your module: 

original weights of the module
mask_params that are used to compute the mask

I would modify the module to have all the right Parameters and recompute weight for each forward. 
# Example for a Linear (handle bias the same way if you want them)
mod = …