This is reasonably normal for a network with lots of narrow/index operations, though I’ve never seen the ratio quite that high. Custom kernels might help but torch.cat, which you’re probably using heavily, is already a very nice custom kernel.