Should you not call add_layer on the loaded model as well. I think the layers is empty. Can you try maybe something like
loaded_module = CustomModule()
loaded_module.add_layer(CustomSubModule(10, 10, T.rand(10, 10), T.rand(10, ),
                                    T.tensor([5]), T.tensor([4])))
loaded_module.load_state_dict(T.load('/tmp/test_model.pt'))