Summary ithr in mpstat increasing

Hello Sun Managers,

Thanks everyone for quick answer, really appreciate your help.
And I was lucky to talk to someone very knowledgeable at SUN Kernel support.
  
In most cases having abnormal number of interrupts or CPU usage
Indicates an offensive process, however our case was related to 
high network traffic usage. We are doing some heavy work on network and 
network interface send its interrupts to one specific CPU which causes 
the number of interrupts on one CPU goes higher than others. This wont 
cause any problem unless CPU power is less than Network speed.
In this case high number of interrupts pins out other processes 
that are running on that CPU.

I found following website very useful:
http://blogs.sun.com/sunay/entry/the_solaris_networking_the_magic

and SUN support mentioned about increasing number of FIFO on network
interface.

I am going to copy all answers the I received from SUN Managers.
Once again thanks every one.


****************************************************************************
******

You can do this in a multi step process.

1. First select the process you think is causing the high ithr - 
   it will be a process using pthreads, or java, or some lib that allows it
to thread 
   (or it might be a custom thread lib, hence the bad behavior)

2. Select a group of 2 or more processors and create a processor group using
'pset' - 
this processor group will be used to isolate execution.
 If you have several processors (T1 or T2 archs) you should create one
process group 
 for each suspect process to save time.  Having a smaller number of
processors helps 
 increase the frequency of interrupts and context switches.

3.  Also using the pset command, bind your suspect processes to a process
group.  
    Only one process group per process so you can easily see which one is
causing the problem.

4. Run mpstat using the -a option to monitor by processor set.

Hope this helps.

JayJay Florendo


****************************************************************************
******

Hello,

It looks like a thread contention on CPU 2. I think prstat has the
possibility to sort PID based on CPU. 
Just look the PID with the max number of threads that use CPU 2. It should
be your guilty PID.

Kind regards,
Eric Deblon

****************************************************************************
********


Hi,

hopefully this is a Sol10 box.. in that case dtrace might help you find the
process producing these interrupts...

http://prefetch.net/articles/dtracecookbook.html

You can get the dtrace toolkit from

http://www.brendangregg.com/dtrace.html#DTraceToolkit

Cheers

Thom





-----Original Message-----
From: Bigadmin [mailto:bigadmin@unixplanet.biz] 
Sent: Tuesday, December 23, 2008 9:15 PM
To: sunmanagers@sunmanagers.org
Cc: bigadmin@unixplanet.biz
Subject: ithr in mpstat increasing

Hi

mpstat output shows Number of intr and ithr are increasing and it 
Reaches to the point that System disable the CPU.

I don't think it is a hard ware problem and looking for a way to find the
exact
Process that cause this problem and then find the application.


CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  101   2  255  1003  645 5575  345  520  575    0  6214   11  24   0  64
  1   61   1  149   609   73 5768  483  555  553    0  4569   10  24   0  66
  2   55   2 1259 14530 14485 1253   86  185 2141    0  3063    6  39   0
56
  3   77   1  197   833  259 5634  469  544  554    0  5050   11  25   0  65

Thanks
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers