Different frameworks make different design choices engineering wise.
For example pytorch works with dynamic graphs that allow you to be more flexible in what you forward pass can do wrt branching for example. And use a pure imperative declaration of the model. This has the downside that your python “forward” method is called every time you perform a forward pass in your network. So every time you make a forward pass, you have to pay this cost on top of the cost of running the network itself.
On the other hand, this allows you to implement easy python logic and get proper python errors when running your model.
The point that johnolafenwa was making is that if your model if big enough, the cost of executing the python code becomes negligible compared to the network execution itself (which is the case for “large” networks).
And there are many other implementation details that can make the performance vary for different use cases.
So the conclusion you can draw from your experiment is that training this specific network on mnist with this code (that looks good) is slower with pytorch than with the mxnet implementation you linked.
It is expected that all the frameworks will be better for some workloads than others.
From my understanding, the general conclusion is that:

For large nets, what is important is the backend: you should use cuda and cudnn for best performance.
A static graph should be faster (implementation details can change that), and more especially for small workloads. See how pytorch is planning to address that in here 10