Thank you for answering.
I update the code. After few search also.
The paper what I am trying to rewrite the code use the pointwise first instead of depthwise first.
I also check that the mobilnet v2 also use this approach in some part actually.
Anyway The block is not the problem about network updating look like.
So I am digging the weight initialization. parts. I am not sure I am missing or something.
Thank you.
self.pointwise = nn.Sequential(
            nn.Conv2d(nin, pw, 1, 1, 0, 1, 1, bias=bias),
            nn.BatchNorm2d(pw),
            nn.ReLU(inplace=True),
            nn.Conv2d(pw, pw, kernel_size=3,stride=1, padding=1, groups=pw, bias=bias),
            nn.BatchNorm2d(pw),
            nn.ReLU(inplace=True))