Well if ur memory consumption is so high that it irritates u then I suggest u downsample ur data with convolutional layers (if the problem is not a regression model problem) and try to update most of ur code variables inplace