Yes but in your diagram above, you can see that the onchip memory gives 900GB/s.
And since many operations we have these days are memory limited. The slowdown you will get can be significant.
That being said, it might depend a lot as well depending on your workload.
I would be interested to know what is the result with your workload.