If you are using custom modules, you could use the @autocast decorator on the forward method and disable it. On the other hand, if you are using built-in modules, you could wrap them in a custom class and disable autocast in the same way.
Alternatively you could also split the nn.Sequential container and just decorate the parts with autocast, which should use it.