Solved by tom in post #2 
              
              
                You can use narrow to your advantage: 
a = torch.randn(5, 2048, 5)
print(((a[:,1:,:] - a[:,:-1,:])-(a.narrow(1, 1, a.size(1)-1)-a.narrow(1, 0, a.size(1)-1))).abs().max().item())
%timeit a[:,1:,:] - a[:,:-1,:]
%timeit a.narrow(1, 1, a.size(1)-1)-a.narrow(1, 0, a.size(1)-1)

gets me 
0.0
14.3 µs ± 190…