SUMMARY: ZFStorage 7120: too slow NFS performance

From: Rob De Langhe <>
Date: Mon Jan 03 2011 - 08:55:27 EST

Some late-2010 answers, and (thanks to several good intentions that
started Jan-1 00:00:01) more early-2011 answers, but nearly concentrated on
network latencies such as on a busy shared LAN.

However, our tests were done over a dedicated point-to-point Ethernet
cable between ZFStorage server and NFS client.

Others pointed me to the "Analytics" functionality in the ZFStorage web
gui, which clearly uses DTrace to capture counters and so. 

However the "Analytics" didn't provide me a means to view "F_SYNC" calls
that might be requested by the NFS clients.

I started to wonder if the "chown -R" duration (to change 28000) files
would maybe still be ok, taking approx 9-12 msecs per NFS_SETATTR call.

Focusing again on the slow login (or Firefox startup) from a Ubuntu
NFS-client into a GNOME desktop, I noticed that crucial processes where
waiting *long* time for locks...

Searching further for NFS and locks relations, provided me the solution :

On the NFS-client, mount using the "nolock" option ; according to the man

"lock/nolock : Selects  whether  to use the NLM sideband protocol to lock
files on the server. If neither option is specified (or if lock is
specified), NLM locking is used for this mount  point. When using the
nolock option, applications can lock files, but such  locks provide
exclusion only against other applications running on the same
client...Using  the nolock  option  is  also  required when mounting
exports on NFS servers that do not support the NLM protocol."

Did so, login and launch of GNOME apps and Firefox happens light-speed

thx all for participating in this quest!


On Fri, 31 Dec 2010 13:07:16 +0100, Rob De Langhe
<> wrote:
> all,
> continuing our quest for better NFS performance out of a ZFStorage 7120
> array:
> Testing from a Linux client (CentOS 4.5) with a dedicated 1Gbit link to
> the ZFStorage array;
> the ZFSstorage provides an NFS-share of a (duh) ZFS filesystem which is
> configured on a pool of double parity RAID and *without* (ZIL) logging.
> - the performance is near network-speed when accessing a huge NFS share
> with reads only (10GB file in 115secs, or 91 MBytes/sec = 728 Mbit/sec),
> when writing a 10 GB chunk (180 secs, or 57 MBytes/sec = 456 Mbit/sec)
> it
> - when I run "du -sk" from the client of 24GB NFS-mounted directory
> containing 28000 files, it takes only 5 secs, so very good, but that's
> course not doing any real data transfer (just reading directories)
> - when I really read data with "tar cf - $nfsdir | wc" from the client
> these 24GB NFS-mounted data, it takes about 391 secs or 61 MBytes/sec (=
> 488 Mbit/sec, not too bad)
> - doing exactly the same operation on the ZFStorage itself, it takes
> 112 secs or 214 MBytes/sec (with single-parity, that's reduced to 25
> - the performance is simply bad when doing many updates like a simple
> "chown -R NNN" in the NFS mount that contains approx 28000 files : 5-6
> minutes to modify all these 28000 files
> - when I do exactly that same "chown -R NNN" command on the ZFStorage
> itself, the command is completed within half a second (ok, sounds like a
> pure-cache operation)
> - also, when mounting a directory from the ZFStorage as home-dir for one
> of the users on the Linux client, the login takes 1-2 minutes before
> and launching Firefox on the client takes 4-5 minutes before done !
> Needless to say that doing the same with a local home-dir on the Linux
> client, everything gets done in a few seconds
> I tried all the same with Solaris(-8) clients, or CentOS 5.5 clients,
> all relevant alternative NFS-mount options on the clients (such as
> rsize=32768, wsize=32767, async, noatime, proto=udp) but no
> possible.
> Reconfiguring the pool on the ZFStorage towards a single-parity "small
> stripe" gave little or no improvement of the performance of the "chown
> test.
> I read some pages on the net discussing a potential FSYNC that would be
> requested by the NFS client after each attribute change (like the
> but can't find any indication of FSYNC in "snoop" on the ZFStorage or in
> "truss" on Solaris-clients or "strace" on Linux-clients that proofs
> Also, that would not explain why logging in and reading plus updating
> several small files takes such a long time.
> When we did the same in another environment with a Solaris NFS-server
> providing home dirs, no such delays with logins or Firefox startup
> so another configuration must be possible on this ZFStorage to achieve
> similar performance.
> thx in advance for any 2010 feedbacks !
> Rob
> _______________________________________________
> sunmanagers mailing list
sunmanagers mailing list
Received on Mon Jan 3 08:55:47 2011

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:44:17 EST