Sorry for the late summary, many thanks to: - Zeev Fisher for the continuous help - Kevin Buterbaugh - Abhilash - Khoi Dinh Most of the the above managers suggested that there are some sort of a heavy IO read over the NFS, Kevin suggested to increase ncsize and ufs_ninode parameters which would have probably fixed the problem. Zeev asked the following outout: netstat -na -P tcp|grep TIME_WAIT|wc -l netstat -na -P tcp|grep FIN_WAIT_2 |wc -l ndd /dev/tcp tcp_conn_hash|head ndd /dev/tcp tcp_conn_hash|wc -l netstat -k|egrep 'norcvbuf|nocanput' and realized: You have TCP drops ( tcpListenDrop value is nonzero so sometimes , incoming requests are refused ) You have probably the default values of tcp_conn_hash and tcp_close_wait_interval Here is what we did which fixed the problem to some extend that we do not hear anything from dba's or users: We found out that the are a huge number of files are being exported via NFS, it appeared that NFS has some sort of limit on the number of inodes being exported, most of the files (about 20GB) were .nfs* files left from previous reboots/crashes etc,. Once we deleted the files, almost immediately the system was back to its "normal" level. Thanks to all. Vahid. ---- Vahid Moghaddasi sunman@ureach.com wrote ---- > To: " Sun-Managers" <sunmanagers@sunmanagers.org> > From: Vahid Moghaddasi <sunman@ureach.com> > Reply-To: <sunman@ureach.com> > Subject: mutex contention > Date: Wed, 31 Jul 2002 13:37:32 -0400 > > > Hi everyone, > > We have a "Serious mutex contention" as se > toolkit puts it. The system is a 4500 4x4 > and exporting about 90GB of disk space as an > nfs server to 4 other systems. Heavy > compilation is done on the system all day. > CPU Status from top shows 0% idle. Here is a > first few lines of lockstat sleep 5: > Count indv cuml rcnt spin > Lock > Caller > ------------------------------------------------------------------------------- > 210070 42% 42% 1.00 102 > vx_bc_bfree_lk > vx_bc_brelse+0x7c > 168414 33% 75% 1.00 90 > vx_bc_bfree_lk > vx_bc_getblk+0x88 > 44338 9% 84% 1.00 51 > 0x662b4b08 > vx_bmap+0x2ec > 29362 6% 90% 1.00 61 > 0x662b4b08 > vx_bmap+0x10 > 22342 4% 94% 1.00 113 > vx_bc_bfree_lk > vx_bc_getblk+0x2ec > 19893 4% 98% 1.00 74 > 0x662b4b08 > vx_bmap+0x1ec > 9168 2% 100% 1.00 61 > 0x662b4b08 > vx_do_bmap_typed+0x220 > 23 0% 100% 1.00 161 > vx_sched_lk > vx_worklist_thread+0x38 > 11 0% 100% 1.00 20 > vx_worklist_lk > vx_worklist_process+0xb4 > > outout of mpstat: > > CPU minf mjf xcal intr ithr csw icsw migr > smtx srw syscl usr sys wt idl > 0 12 0 561 4308 375 10868 3941 49 > 23084 0 969 42 58 0 0 > 1 1 1 259 4119 120 10849 3973 53 > 22867 0 2006 34 66 0 0 > 4 1 2 161 4600 289 11265 4305 57 > 22473 0 1252 42 58 0 0 > 5 26 1 850 4729 255 11494 4296 50 > 22453 0 845 40 60 0 0 > > > Here is some kernel parameters which might > help!!! > > ncstats: 3372263 > ncsize: 69992 > ufs_ninode: 69992 > maxusers: 1024 > max_nprocs: 16394 > maxuprc: 16389 > ndquot: 26634 > nbuf: 800 > > vmstat -s cache_ratio output: > ... > 9920068 total name lookups (cache hits 35%) > ... > > Any help would be appreciated and as always > I will summarize. > VAhid. ---- Vahid Moghaddasi sunman@ureach.com wrote ---- > To: " Sun-Managers" <sunmanagers@sunmanagers.org> > From: Vahid Moghaddasi <sunman@ureach.com> > Reply-To: <sunman@ureach.com> > Subject: mutex contention > Date: Wed, 31 Jul 2002 13:37:32 -0400 > > > Hi everyone, > > We have a "Serious mutex contention" as se > toolkit puts it. The system is a 4500 4x4 > and exporting about 90GB of disk space as an > nfs server to 4 other systems. Heavy > compilation is done on the system all day. > CPU Status from top shows 0% idle. Here is a > first few lines of lockstat sleep 5: > Count indv cuml rcnt spin > Lock > Caller > ------------------------------------------------------------------------------- > 210070 42% 42% 1.00 102 > vx_bc_bfree_lk > vx_bc_brelse+0x7c > 168414 33% 75% 1.00 90 > vx_bc_bfree_lk > vx_bc_getblk+0x88 > 44338 9% 84% 1.00 51 > 0x662b4b08 > vx_bmap+0x2ec > 29362 6% 90% 1.00 61 > 0x662b4b08 > vx_bmap+0x10 > 22342 4% 94% 1.00 113 > vx_bc_bfree_lk > vx_bc_getblk+0x2ec > 19893 4% 98% 1.00 74 > 0x662b4b08 > vx_bmap+0x1ec > 9168 2% 100% 1.00 61 > 0x662b4b08 > vx_do_bmap_typed+0x220 > 23 0% 100% 1.00 161 > vx_sched_lk > vx_worklist_thread+0x38 > 11 0% 100% 1.00 20 > vx_worklist_lk > vx_worklist_process+0xb4 > > outout of mpstat: > > CPU minf mjf xcal intr ithr csw icsw migr > smtx srw syscl usr sys wt idl > 0 12 0 561 4308 375 10868 3941 49 > 23084 0 969 42 58 0 0 > 1 1 1 259 4119 120 10849 3973 53 > 22867 0 2006 34 66 0 0 > 4 1 2 161 4600 289 11265 4305 57 > 22473 0 1252 42 58 0 0 > 5 26 1 850 4729 255 11494 4296 50 > 22453 0 845 40 60 0 0 > > > Here is some kernel parameters which might > help!!! > > ncstats: 3372263 > ncsize: 69992 > ufs_ninode: 69992 > maxusers: 1024 > max_nprocs: 16394 > maxuprc: 16389 > ndquot: 26634 > nbuf: 800 > > vmstat -s cache_ratio output: > ... > 9920068 total name lookups (cache hits 35%) > ... > > Any help would be appreciated and as always > I will summarize. > VAhid. > > ________________________________________________ > Get your own "800" number > Voicemail, fax, email, and a lot more > http://www.ureach.com/reg/tag > _______________________________________________ > sunmanagers mailing list > sunmanagers@sunmanagers.org > http://www.sunmanagers.org/mailman/listinfo/sunmanagers > > ________________________________________________ Get your own "800" number Voicemail, fax, email, and a lot more http://www.ureach.com/reg/tag _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagersReceived on Wed Aug 7 22:34:19 2002
This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:42:51 EST