title: Should batch size matter at inference
body: <p>I am training a model</p>
  
  <ul>
  <li>5-layer very narrow CNN,</li>
  <li>followed by a 5-layer highway,</li>
  <li>then fully connected and</li>
  <li>softmax over 7 classes.</li>
  </ul>
  
  <p>As there are 7 equally distributed classes, the random bet lassification accuracy would be 14 %  (1/7th is roughly 14 %).
  Actual accuracy is 40 %. So the net somewhat learns..</p>
  
  <p>Now the weird thing is it learns only with a batch size of 2. Batch sizes of 16, 32 or 64 don't learn at all. </p>
  
  <p>Now the even weirder thing: If I take the checkpoint of the trained net (accuracy 40 %, trained at batch size 2) and start it with a batch size of 32 I should keep on getting my 40 % at least for the first couple of steps, right? I do when I restart at barch size 2. But with the bs 32 initial accuracy is, guess what, 14 %.</p>
  
  <p>Any idea why the batch size would ruin inference? I fear I might be having a shape error somewhere but I cannot find anything. </p>
  
  <p>Thx for your thoughts</p>