Trick is to use numpy itself in torch without hurting the backpropgration.
For x as a 2D tensor this works for me:
import numpy  as np

row_idx, col_idx = np.triu_indices(x.shape[1])
row_idx = torch.LongTensor(row_idx).cuda()
col_idx = torch.LongTensor(col_idx).cuda()
x = x[row_idx, col_idx]

For 3D tensor (assuming first dimension is batch):
row_idx, col_idx = np.triu_indices(x.shape[2])
row_idx = torch.LongTensor(row_idx).cuda()
col_idx = torch.LongTensor(col_idx).cuda()
x = x[:, row_idx, col_idx]

Note that the whole process is still differentiable in pytorch.