It depends on the implementation of the corresponding dataset and you could check it in the source code (as done for MNIST using my link).
E.g. the often used ImageFolder dataset uses lazy loading, to save memory, while the CIFAR datasets also load the data and target into the RAM as seen here 26.
Custom Dataset implementations can be written in either way, since you are defining how the data is loaded.