In Fast.ai course they use their own library which is built on top of pytorch. They have a model loading function where u can set if u want precompute. I dont have access to internals of these library fns. However they can be explored further. Their forum is very popular & I will try to get answers to the issues U raised. If u are using transformations especially the random ones they also would raise issues.
Here is my speculation:
The nn prior to to the fc layer are frozen and batchnorm and dropout not applied. Even certain random transformations cant be used. This reduces the problem to precomputing the activations prior to fc layer for each input image once and using the same over the epochs.
This of course is not ideal bur the speed at which u can validate is phenomenal. 3 epochs on google colab with gpu take 1-2 min or so.  To handle transformations they may be making multiple copies of the activations one for each transformation and choose one at a time. This is only my speculation.