The problem was that the original model was saved with DataParallel, so instead of bert.encoder.layer11, it is module.bert.encoder.layer11. I have re-saved the module without DataParallel wrapper. The problem is solved now.