I still have this problem on the CPU (although not on the GPU). I have compiled the latest version of PyTorch from source (commit #542c273) with MKL-DNN. torch.norm (L2) seems to be about 3-5 times slower than a square-root of a sum of squares.
Here is my code to reproduce this. On my system, this code takes 383.1 seconds with line #1, and 75.2 seconds with line #2.
import torch
import time

loss = 0

A = torch.rand(10, 100, 1000)
A.requires_grad = True

start = time.time()

for i in range(0, 20000):

	B = torch.norm(A, dim=0)  #1

	# B = torch.sqrt(torch.sum(A * A, dim = 0))  #2

	loss += torch.sum(B)

loss.backward()
end = time.time()

print(end - start)