you can use index_select, and yes it’s differentiable wrt buffer (but not differentiable wrt p0 or p1, because it’s not continuous):
buffer = Variable( torch.FloatTensor( 128, 128 ) , requires_grad=True)
p0 = Variable( torch.FloatTensor( [2.4, 9.4] ) )
p1 = Variable( torch.FloatTensor( [5, 11.5] ) )

buf2 = buffer.index_select(0, p0.long()).index_select(1, p1.long())
buf2.backward(torch.ones(2, 2))