A quick glance at the code does not reveal anything.
Given how your replay buffer is implemented, it is expected to grow for a bit until it is full.
Could you be clearer about what you tried and the effect?
Did you mean that when you don’t train, you don’t see this behavior?