I see! Your arch looks okay to me as per paper;  Try lowering the LR maybe?

As I understood, in Pytorch we need to set  weight_decay  parameter in our optimizer to achieve the same result, but I’m far from being confident here.

Yep!