It just seems to be incredibly slow. I put some print statements in the LSUN dataset and it is working. Its just gonna take a long time. I tried increasing max number of workers and allowing readahead so hopefully I dont have to wait too long.
I guess I just have to open it the slow way once. After that the cache files makes loading the dataset fast.