I’m in a project that requires me to update the training data regularly, that’s why I need to reset the iterator each time I update the dataset. I tried to use tf.placeholder as feedict to update the training set, but the memory usage increase as training progress (not so serious as long as it’s till affordable) but sometimes at some training step the code stop when it reinitialize the iterator (weird that at training step before that it still work fine despite increase memory usage), not only that the memory usage increase much faster than before. On my local machine, I don’t see this problem, but it arise once I send my code to a remote gpu cluster to train. I try to use tcmalloc but not sure if the system load it correctly, as there is no change, it still get OOM after a while