I am following this tutorial on CUSTOM C++ AND CUDA EXTENSIONS.
I pretty much took the code available in the page and ran in a Google Colab environment 2.
Expectation
From the tutorial page we have the following comparisons
Screen Shot 2021-01-07 at 12.07.10 PM1472×634 45.1 KB
Screen Shot 2021-01-07 at 12.07.25 PM1478×794 67.9 KB
Reality
But, when running on the notebook, the backward pass gets worse. Is there any explanation on why this would happen?
====== CPU Default
Forward: 216.003 us | Backward 237.442 us
====== CPU Cpp Custom Extension
Forward: 171.469 us | Backward 437.926 us

====== GPU Default
Forward: 309.739 us | Backward 523.746 us
====== GPU Cpp Custom Extension
Forward: 250.254 us | Backward 879.118 us