Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

A few days ago I noticed a disk I/O wait and disk activity drop (which was great). Then I also notice that my cache was full(*) and fragmented. Then I flushed cache. After that, disk latency and disk activity jumped to the previous level (which was bad).

IOtop shows that [jbd2/sda2-8] and [flush-8:00] are always on top of disk usage. This is a Dell R210, hardware RAID 1 (H200) with a lot of free memory (16 GB in total, of which about 8 GB are buffer/cache).

(*) The cache is APC opcode cache for PHP, which reduces disk access for PHP script execution. Cache was full and fragmented because it included files from development instance. When I noticed that, I filtered them out.

The question is: why disk I/O increases when theorically it should decrease? Below are some graphs from munin. Cache was full from Feb 6 to 8.

Change after I commented out apc.mmap_file_mask as told by @cyberx86

And after a few days http://serverfault.com/a/362152/88934

share|improve this question

2 Answers 2

up vote 8 down vote accepted

If you use file-backed memory mapping (e.g. apc.mmap_file_mask=/tmp/apc.XXXXXX) you might see elevated I/O.

Try setting apc.mmap_file_mask to use shared memory (e.g. /apc.shm.XXXXXX) or to /dev/zero (anonymous mmapped memory). Keeping the setting undefined defaults it to using anonymous mmapped memory.

Usually, mmapped files is a great thing:

  • Compared to storing something fully in memory, mmapped files usually require less memory
  • Compared to saving something to a file, mmapped files require less disk I/O (since writes can be aggregated together).

However, compared to storing something purely in memory, they do incur added I/O - considerably so when the file is continuously changing. The downside of not using mmapped files is a lack of persistence - your cache will not survive a restart, since it is stored only in memory.

One may suggest therefore, that while the cache was filling up and stabilizing, it was undergoing the most change, which had to be constantly written to disk; once the cache was full, the ttl for each object slowed the rate that data in the cache was being turned over, decreasing the change and reducing disk writes.

share|improve this answer

After a few days, now I want to come back with some graphs. The change improves much that situation. It reduces everything, except the IO service time (I think it's because there is no longer trivial small PHP file read which is cheap).

The server load (it was quite low already, so I had not discovered the change).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["                  <li><a href=\"//area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"//careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2015 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2015.3.26.2411\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Server Fault works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </noscr"], "copyright__copyright__copyright": ["ck with some graphs. The change improves much that situation. It reduces everything, except the IO service time (I think it's because there is no longer trivial small PHP file read which is cheap).\n\nThe server load (it was quite low already, so I had not discovered the change).\n\nshare|improve this answer\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}