trm:

but I was curious if pytorch is doing things in float32.


Yes, by default, but you can also specify to use float64 if needed.



 trm:

Or perhaps it calls everything underneath in C and then maps it back to python. That communication layer between C and python might do some sort of compression. This is for the sake of increasing speeding of course


No, there is no compression involved and PyTorch will use the specified dtype.



 trm:

So in actual training performance is not affected (or maybe float64 is overkill)


I haven’t seen a ML/DL model, which “needs” float64 yet so the performance in the sense of model accuracy should not be affected. However, I would be interested to learn about such a model, which really needs 64 bits. The “speed” performance would be affected as you are not only using 2x the memory, but especially on the GPU would see a slowdown.