I believe I’ve just solved this.
It turns out that a specific function was allocating memory that Python wasn’t able to retrieve back and so the accumulation was the cause of this increase. So to solve this I spawn a new process to run the function and terminate it once I get what I want from it.
For anyone that might come across this issue:

reader = mir.MultiResolutionImageReader()
image = reader.open(self.names[index])
def f(x, y, w, h, l, q):
    img = image.getUCharPatch(x, y, w, h, l)
    q.put(img)
from multiprocessing import Process, Queue
q = Queue()
p = Process(target=f, args=x, y, w, h, l,  q))
p.start()
img = q.get()
p.join()
p.terminate()