I am probably misunderstanding something but:
In the docs of functional.normalize (https://pytorch.org/docs/stable/nn.html#torch.nn.functional.normalize 541) we can read:
Performs Lp normalization of inputs over specified dimension.
Does v=v / max(‖v‖p,ϵ)
for each subtensor v over dimension dim of input. Each subtensor is flattened into a vector.
So if I do the following
import torch
import torch.nn.functional as F
x = torch.randn((4, 3, 32, 32))
x = F.normalize(x, dim=0, p=2)

I would expect that each subtensor along dim 0 (for instance x[0]) will have a L2 norm equal to 1.
However, this isn’t the case
torch.sqrt(torch.sum(x[0]**2)) # != 1

(I use pytorch 0.4.1 with CUDA 9.2)