title: psutil vs dd: monitoring disk I/O
body: <p>I'm writing y.a.t. (yet-another-tool :)) for monitoring disk usage on Linux.
  I'm using <em>python 3.3.2</em> and <em>psutil 3.3.0</em>.</p>
  
  <p>The process I'm monitoring does something really basic: I use the <em>dd</em> tool and I vary the <strong>block size</strong> (128, 512, 1024, 4096)</p>
  
  <pre><code>#!/bin/bash
  dd if=./bigfile.txt of=./copy.img bs=4096
  </code></pre>
  
  <p>bigfile.txt:</p>
  
  <pre><code>$ stat bigfile.txt 
       File: ‘bigfile.txt’
       Size: 87851423     Blocks: 171600     IO Block: 4096   regular file
  </code></pre>
  
  <p>And the snippet of the monitor is as follows:</p>
  
  <pre><code>def poll(interval, proc):
      d_before = proc.io_counters()
      time.sleep(interval)
      tst = time.time()
      d_after = proc.io_counters()
  
      usage = OrderedDict.fromkeys(d_after.__dict__.keys())
      for k, v in usage.items():
          usage[k] = d_after.__dict__[k] - d_before.__dict__[k]
  
      return tst, usage
  </code></pre>
  
  <p>At each run, I clear the cache (as suggested many times on stackoverflow):</p>
  
  <pre><code>rm copy.img &amp;&amp; sudo sh -c "echo 3 &gt; /proc/sys/vm/drop_caches"
  </code></pre>
  
  <p><strong>My question is: why aren't the numbers matching?</strong></p>
  
  <p><strong>bs=128</strong>: </p>
  
  <p>dd:</p>
  
  <pre><code>686339+1 records in
  686339+1 records out
  87851423 bytes (88 MB) copied, 1.21664 s, 72.2 MB/s
  </code></pre>
  
  <p>monitor.py:</p>
  
  <pre><code>1450778750.104943 OrderedDict([('read_count', 686352), ('write_count', 686343), ('read_bytes', 87920640), ('write_bytes', 87855104)])
  </code></pre>
  
  <p><strong>bs=4096</strong></p>
  
  <p>dd:</p>
  
  <pre><code>21448+1 records in
  21448+1 records out
  87851423 bytes (88 MB) copied, 0.223911 s, 392 MB/s
  </code></pre>
  
  <p>monitor.py:</p>
  
  <pre><code>1450779294.5541275 OrderedDict([('read_count', 21468), ('write_count', 21452), ('read_bytes', 88252416), ('write_bytes', 87855104)])
  </code></pre>
  
  <p>The difference is still there with all the values of <em>bs</em>.</p>
  
  <p>Is it a matter of certains read/write not being counted? Does psutil performs some extra work? For example, with <em>bs=4096</em>, why in psutil 400993 more bytes (for read) and 3681 (for write) are reported?</p>
  
  <p>Am I missing something big?</p>
  
  <p>Thanks a lot.</p>
  
  <p><strong>EDIT</strong>: as an update, it doesn't matter the granularity of timings in the measurement, i.e., the time.sleep(interval) call. I tried with different values, and summing up the total number of reads and writes reported by psutil. The difference remains.</p>
  
  <p><strong>EDIT2</strong>: typo in snippet code</p>