Depending on the numbers B, N, and the size of the voxel grid here, one option might be to do kernel regression. This is quite computationally intensive during voxelization, but in contrast to binning like histograms do, does interpolation, too.
A blunt way to mimic what a histogram does is to quantize the coordinates into integers starting from 0 and then use them as the indices of a sparse tensor. These are allowed to have duplicate indices. Then tensor.coalesce() will sum all values at a given coordinate.
Of course, there so many ways to do this, but these are the ones I’d try first.
Best regards
Thomas