As far as I know, the call to contiguous copies the data and returns a new one, so this might be the explanation for your OOM error. Could you tell how large x is in your case?
Also, I think you might have to call contiguous after permute, since this operation is returning a non-contiguous tensor. Could you try that and run your code again?