Hello Alex!



 googlebot:




 KFrank:

One could imagine adding a function to pytorch that does what
you want


… And this operation is called scatter_add, it uses atomicAdd on cuda, on cpu it is a loop, but c++ loop != python loop.


Thanks for that – you’re absolutely right.  And I agree that the use
of “atomicAdd” (assuming it means what it sounds like) eliminates
the need for explicit sequencing (such as with a for loop).  Multiple
cuda operations can attempt to update a given element of A
“simultaneously,” and they should be able to do so on a first-come,
first-served basis, without any conflicts.


A.scatter_add(0, C.view(2,-1).expand(-1,3), B)



Sadly I can’t try this, as I’m stuck several layers down in the pytorch
archeological strata with version 0.3.0, which lacks this function.


PS about your rounding example, many cuda ops are non-deterministic, so we live with it, trying not to sum billions with one in float32.


Yes, that’s fair.  The purpose of my admittedly-contrived example was
to show that when C contains duplicate indices there is an ordering
issue, at least in principle.
As an aside, for scatter(), where the ordering issue is fully real (not
just differing in floating-point round-off), the documentation warns of
this:

Moreover, as for gather(), the values of index must be between 0 and self.size(dim) - 1 inclusive, and all values in a row along the specified dimension dim must be unique.

(Although – at least in version 0.3.0 – unique indices are not enforced
nor checked for.)
Best.
K. Frank