After digging more into it I have come to the conclusion that the comparison of scalar tensors is taking a huge time. Any way how to make this faster?
        return ctx.beta - torch.FloatTensor([ \
                ctx.alpha*np.sum([(1 if w[(x, y)] > w[j] else -1) for j in FrankWolfe_MinCutSum_Canonical.neighbors(ctx, (x, y))]) \
                for x in range(ctx.height) \
                for y in range(ctx.width)]).reshape(ctx.width, ctx.height).to(device)