Using the backbone of a pretrained model seems to be orthogonal to the usage of amp and you could call the model directly in your custom model wrapper.
However, if you want to only use the backbone, replace other layers either with nn.Identity layers or check the source code of the model and see, if you could simply call the backbone directly.
Some models are implemented in this way, others are more complicated and would need to a bit more attention.