Solved by solvingPuzzles in post #2 
              
              
                The approach I ended up using
I ended up using permute to make C the rightmost dimension before LayerNorm, and then permuting again to go back to the original shape. 
Let’s do a simpler example with 3 dimensions instead of 4: 
import torch
from torch import nn

def get_input_tensor(dims):
    t = to…