Perhaps I’ve seen 512 somewhere else, and it is actually 256 threads/block. Basically, it is hardcoded here: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Loops.cuh#L6 43
And example of “vector kernel” block size selection logic (not pretty): https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Reduce.cuh 26