The slowdown seems to be triggered by the data loading pipeline.
If you remove the DataLoader usage, the time will drop from ~47s to ~10s (TF uses ~17s).
I don’t know what kind of nodes Colab is using, but you might want to play around with the num_workers argument to the DataLoader to use multiprocessing or create the data as a big tensor instead of loading each image separately (as done in TF), since no data augmentation is used.