First of all - thanks for the reply! 
Are you asking how I managed to pass such shape to InceptionResnetV2 due to it not being “rgb”-ish? It’s a modified version, since I’m working with single color channel type of data.
I’m training the weights from scratch so I only had to do a simple modification of the InceptionResnetV2 code to achieve it.
Passing it in a batch as you described does cause an OOM immediately
I’m trying to understand - while the graph is being built, isn’t the fact that all of the weights are shared help in the reduction/reuse of memory ?
maybe there is something explicit I need to do in order to “help” autograd understand that it can reuse existing buffers?