2

Since a recent linux kernel upgrade (3.2.78 on Debian Wheezy) on both my NFS clients and NFS server, I'm seeing really high increase (10 to 20 folds) in the number of nfs_lookup requests from the clients to the server. This is making a huge performance degradation to the application running on my NFS clients (apache+php-fpm).

What can be the cause of such a high increase of nfs_lookup?

Output of mountstats --nfs on one of the clients after about 1h of uptime is as follows:

Stats for ruchbah.nfs://membres mounted on /srv/membres:
  NFS mount options: rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.74.13,minorversion=0,local_lock=none
  NFS server capabilities: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3
  NFS security flavor: 1  pseudoflavor: 0

Cache events:
  data cache invalidated 1231 times
  attribute cache invalidated 23693734 times
  inodes synced 1790 times

VFS calls:
  VFS requested 46296214 inode revalidations
  VFS requested 35900771 dentry revalidations

  VFS called nfs_readdir() 3678 times
  VFS called nfs_lookup() 35200320 times
  VFS called nfs_permission() 11201 times
  VFS called nfs_file_open() 22547 times
  VFS called nfs_file_flush() 1400 times
  VFS called nfs_lock() 7611 times
  VFS called nfs_fsync() 65321 times
  VFS called nfs_file_release() 3731 times

VM calls:
  VFS called nfs_readpage() 856 times
  VFS called nfs_readpages() 6945 times
  VFS called nfs_writepage() 10789 times
  VFS called nfs_writepages() 945872 times

Generic NFS counters:
  File size changing operations:
    truncating SETATTRs: 65272  extending WRITEs: 0
  291 silly renames
  short reads: 11201  short writes: 17
  NFSERR_DELAYs from server: 0

NFS byte counts:
  applications read 162084951 bytes via read(2)
  applications wrote 19636930 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 152501261 bytes via NFS READ
  client wrote 20783074 bytes via NFS WRITE

The number of VFS calls (nfs_lookup in particular) is very high. The cache attribute invalidation is also abnormally high. The clients mostly uses the NFS mount to read data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["<li class=\"-item\"><a href=\"https://www.facebook.com/officialstackoverflow/\" class=\"-link\">Facebook</a></li>\r\n                    <li class=\"-item\"><a href=\"https://twitter.com/stackoverflow\" class=\"-link\">Twitter</a></li>\r\n                    <li class=\"-item\"><a href=\"https://linkedin.com/company/stack-overflow\" class=\"-link\">LinkedIn</a></li>\r\n                </ul>\r\n\r\n                <p class=\"mt-auto mb24\">\r\nsite design / logo &#169; 2019 Stack Exchange Inc; user contributions licensed under <a href=\"https://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a>\r\n                            with <a href=\"https://stackoverflow.blog/2009/06/25/attribution-required/\" rel=\"license\">attribution required</a>.\r\n                    <span id=\"svnrev\">rev&nbsp;2019.7.23.34400</span>\r\n                </p>\r\n            </div>\r\n        </div>\r\n\r\n            </footer>\r\n            <script>StackExchange.ready(function () { StackExchange.responsiveness.addSwitcher(); })</script>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Server Fault works bes"], "copyright__copyright__copyright": ["_DIRECT read(2)\n  applications wrote 0 bytes via O_DIRECT write(2)\n  client read 152501261 bytes via NFS READ\n  client wrote 20783074 bytes via NFS WRITE\n\nThe number of VFS calls (nfs_lookup in particular) is very high. The cache attribute invalidation is also abnormally high. The clients mostly uses the NFS mount to read data.\n\nYour Answer\n\nBy clicking \u201cPost Your Answer\u201d, you agree to our terms of service, privacy policy and cookie policy\n\nBrowse other questions tagged or ask your own question."]}