title: Tensorflow training killed by system
body: <p>I am some trouble figuring out why my training is killed by the os during training. </p>
  
  <p>The program can run normally for about 50% of the total training cycles. Then around the last few cycles, the entire program will be killed.</p>
  
  <p>Does tensorflow consume more memory as it trains? Or does it use a constant amount of memory during training?</p>
  
  <p>Note: I am running on a cpu only version of Tensorflow on a virtual machine.</p>