The issue is that there is a dangling Sequential module. When seeing an empty Sequential, the logic is  (erroneously) inserting the observer in there, which causes it to have the error you are seeing. This is fixed in https://github.com/pytorch/pytorch/pull/28384 24 – if you rebase to master, it will fix it. Alternatively, you can modify the model to drop the shortcuts.
If you are still seeing the error after rebasing, please, let us know