Hi,
If your images are of size [19, 22], then I would expect the intermediary state to be very small indeed. Especially for a batch size of 100 (if the sample in your first post is accurate).
For example, the output of the conv1 is <2MB (100 * 50 * 19 * 22).
How much memory does your GPU has?