I found the problem, and as you sad it was an “accidental” access to the cuda context, done when I copied the parameters between the model of the main process (in gpu) to the models of the workers processes (in cpu). This was done through the state_dict method, without mapping the tensors in the state_dict to the CPU before calling load_state_dict in the worker models.
Thanks again for the help!