title: Running session using tensorflow c++ api is significantly slower than using python
body: <p>I am trying to run SqueezeDet using tensorflow c++ api (CPU only). I have freezed tensorflow graph and loaded it from C++. While in terms of detection quality everything is fine, performance is much slower than in python. What can be the reason of that?</p>
  
  <p>Simplified, my code looks like this:</p>
  
  <pre><code>  int main (int argc, const char * argv[])
    {
      // Initializing graph 
      tensorflow::GraphDef graph_def;
      // Folder in which graph data is located
      string graph_file_name = "Model/graph.pb";
      // Loading graph 
      tensorflow::Status graph_loaded_status =  ReadBinaryProto(tensorflow::Env::Default(), graph_file_name, &amp;graph_def);
      if (!graph_loaded_status.ok())
      {
        cout &lt;&lt; graph_loaded_status.ToString() &lt;&lt; endl;
        return 1;
      }
      unique_ptr&lt;tensorflow::Session&gt; session_sqdet(tensorflow::NewSession(tensorflow::SessionOptions()));
      tensorflow::Status session_create_status = session_sqdet-&gt;Create(graph_def);
      if (!session_create_status.ok())
      {
        cout &lt;&lt; "Session create status: fail." &lt;&lt; endl;
        return 1;
      }
      while ()
      {
        /* create &amp; preprocess batch */
  
        session.Run({{ "image_input", input_tensor}, {"keep_prob", prob_tensor}}, {"probability/score", "bbox/trimming/bbox"}, {}, &amp;final_output);
  
        /* do some postprocessing */
      }
    }
  </code></pre>
  
  <p>What I have tried:</p>
  
  <p>1) Using optimization flags - all are on, no warnings.</p>
  
  <p>2) Using batching: performance increased, but the gap between python and C++ is still significant (running session takes 1s vs 2.4s with batch_size = 20).</p>
  
  <p>Any help would be highly appreciated.</p>