I found this issue about backward for narrow operations. https://github.com/ezyang/pytorch/issues/255 224
Narrow is a very cheap operation in forwards, because it doesn’t have to do much, but it is expensive in backwards, since you must allocate a full size tensor and zero pad it out. This is a good opportunity for the optimizer.