The parameters will still be stored on FP32 and the memory saving might come from the activations, which could be stored in FP16 (if the operation is save to be used in FP16).
The amp example 3 would be a good starter and shows how to use autocast.