The issue with your code is that you are removing the nn.DataParallel wrapper by calling:
model = model.module  # For multi-gpu

so remove this line of code.
After removing it, you would also have to change how the optimizers are created, as you need to either access the internal layers via model.module.fc_x.parameters() or create the optimizers before wrapping the model into nn.DataParallel.