ptrblck:

E.g. in this post I’ve posted some stats about the used model and you can see that the activations use:
6452341200 / 138357544 ~= 47

times more memory than the parameters.


Wow that’s astonishing. I heard it was more but I had no idea it was this much more.