Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am currently running a debian server that is exporting a large JFS filesystem (22TB) over NFS (nfs-kernel-server.) When attempting to write to the NFS share, the performance is very poor. The 22TB disk is sitting on a NAS mounted using iSCSI.

  • It will bust for a moment near expected line speed, and then sit idle for several seconds. Very little traffic measured in the low kb/sec.
  • The wait peeks on write.
  • When reading from the NFS mount, the system operates at expected speeds (11MB/sec).
  • The issue does not occur when using SFTP, rsync, or local coping (non-nfs).
  • The issue persists between stable and testing releases.
  • On the same machine I have a 14TB ext4 filesystem using the exact same export configuration that does not share the issue. This share is not in regular use and thus not consuming resources.

NFS Server:

cat /etc/exports
/data2      10.1.20.86(rw,no_subtree_check,async,all_squash)

cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq

cat /etc/default/nfs-kernel-server 
RPCNFSDCOUNT=8
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS=--manage-gids
NEED_SVCGSSD=
RPCSVCGSSDOPTS=

NFS Client:

cat /etc/fstab
10.1.20.100:/data2  /root/incoming  nfs     rw,noatime,soft,intr,noacl 0 2

cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq

cat /proc/mounts
10.1.20.100:/data2/ /root/incoming nfs4 rw,noatime,vers=4,rsize=262144,wsize=262144,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.1.20.86,minorversion=0,addr=10.1.20.100 0 0

This problem has me pretty stumped. Any help would be greatly welcomed. Thanks.

share|improve this question
show 1 more comment

3 Answers

I suspect some problem with JumboFrames. Check offloading configuration on both your interfaces using

sudo ethtool -k your_nic

and also try listening the wires using wireshark. You may find some out-of-order packets, dups, ...

share|improve this answer
add comment

My guess is that the number of NFS server threads is too low. Instead of 8, the number should be much higher.

8 threads would probably be enough for shares that contain only small files and are accessed by a very small number of users (e.g. in a home network) or on slow networks (10 Mbit).

Try to determine the retrans value on your NFS server during writing:

nsstat -r

If you get transmission retries, increase the number of server threads.

And I think it would be save to remove the rsize/wsize/tcp settings from your mount options. TCP is the default protocol anyway and with TCP it is not necessary to limit the transfer size.

share|improve this answer
add comment

Maybe it is incompatibility with nfs locking used for writting and jfs. I found some bug in ubuntu: https://bugs.launchpad.net/ubuntu/+source/jfsutils/+bug/754495

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["        <li><a href=\"http://area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"http://careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2014 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2014.4.18.1558\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Server Fault works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </noscr"], "copyright__copyright__copyright": ["ay and with TCP it is not necessary to limit the transfer size.\n\nshare|improve this answer\nadd comment\n\nMaybe it is incompatibility with nfs locking used for writting and jfs. I found some bug in ubuntu: https://bugs.launchpad.net/ubuntu/+source/jfsutils/+bug/754495\n\nshare|improve this answer\nadd comment\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}