I’m not familiar with the mentioned library, but could you post the complete error message you are seeing, if you are not detaching?
Currently, the “skip” layers wouldn’t be trained and I’m not sure, if this fits your use case.