That is because when you are using smaller values for filters, smaller kernels are created i.e, the complexity to compute a single image increases because the kernel being small has to move through more number of pixels. The entire complexity of computing an dataset is multiplied based on the factor i.e, for a single image.
In simple words, small filters increases the accuracy but complexity increases at the same time thereby increase in memory layer after layer.
Large filters decreases accuracy but complexity and memory usage is reduced.
Therefore, smaller filters = More memory!