Solved by ptrblck in post #5 
              
              
                By default the models would by on the CPU, so you would still make sure to push them independently. 
I think your approach of using the functional API for these fixed parameters looks good and I probably wouldn’t try to use modules and reload the parameter. In case you still want to do so, you could…