SUMMARY: mpstat on T2000

From: JESSE CARROLL <jesse-carroll_at_usa.net>
Date: Tue Aug 07 2007 - 17:01:07 EDT
(original question below)
To date I've received excellent suggestions from William Hathaway, Ric
Anderson and Mark Round. Mark was correct in that the interrupts where from
one of the NIC's. Ric had an excellent suggestion to use 'prstat -P <cpu>',
which showed which processes where running on a particular CPU. This showed
that the OS was rotating the processes around. William Hathaway takes the
prize though, with his suggestion to use 'intrstat'. This definitely was one
of the NIC's. My guess is that the NIC's somehow are tied to a cpu/core for
interrupt handling as 'intrstat' showed that a particular NIC was
consistently
responsible for the high(er) rates of interrupts for a particular CPU.

As  Michael Hase pointed out, the system was fairly idle during the sample
period. However, earlier today the system was experiencing a weird high rate
swapped out light-weight processes, though the system didn't appear to be
under  a significant load. Still looking for that, but its a hit-or-miss
thing.

JC

------ Original Message ------
Received: Tue, 07 Aug 2007 01:06:05 PM EDT
From: "JESSE CARROLL" <jesse-carroll@usa.net>
To: <sunmanagers@sunmanagers.org>
Subject: mpstat on T2000

While looking at a possible performance issue on a T2000 I noticed that cpu
27
was experiencing a much higher rate of interrupts and interrupts on threads
than the other 31 cores. (The mpstat output is below) I looked at a couple of
other T2000's and saw very similar behavior. I can't seem to find a
particular
process consistently running on cpu 27 and all system run different
applications. Is cpu 27 somehow used differently (e.g. floating point or
specific interrupt handling) than the other ones?

mpstat output
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   22   0  995   314  140  143    1    1    1    0   306    9   1   0  90
  1   16   0  760    50    0   95    1    1    1    0   222    7   1   0  92
  2   10   0  620    34    0   65    0    1    1    0   130    4   0   0  95
  3    8   0  575    31    0   59    0    1    1    0   115    3   0   0  97
  4   13   0  687    44    0   86    0    1    1    0   171    5   1   0  95
  5   13   0  654    44    0   85    0    1    1    0   164    5   1   0  95
  6   12   0  630    45    1   84    0    1    1    0   161    4   1   0  95
  7    9   0  595    41    0   80    0    1    1    0   146    3   0   0  97
  8   14   0  667    44    0   85    0    1    1    0   168    5   1   0  95
  9   11   0  618    41    0   80    0    1    1    0   151    3   0   0  96
 10   12   0  616    43    0   83    0    1    1    0   157    4   0   0  95
 11   12   0  612    43    0   84    0    1    1    0   160    5   0   0  95
 12   12   0  609    42    0   83    0    1    1    0   157    3   1   0  96
 13   12   0  651    44    0   86    0    1    1    0   162    4   1   0  95
 14   12   0  634    44    0   86    0    1    1    0   163    5   1   0  95
 15   11   0  606    42    0   83    0    1    1    0   152    4   0   0  96
 16   13   0  661    43    0   83    0    1    1    0   166    4   1   0  95
 17   12   0  623    43    0   83    0    1    1    0   160    5   0   0  95
 18   12   0  602    42    0   82    0    1    1    0   155    4   0   0  95
 19   11   0  582    42    0   82    0    1    1    0   154    4   0   0  95
 20   11   0  652    43    0   84    0    1    1    0   156    3   1   0  96
 21   13   0  650    45    0   87    0    1    1    0   161    5   1   0  95
 22   13   0  629    45    0   87    0    1    1    0   160    5   1   0  95
 23    9   0  575    41    0   81    0    1    1    0   145    2   0   0  97
 24   13   0  647    43    0   84    0    1    1    0   162    4   1   0  95
 25   12   0  611    42    0   83    0    1    1    0   155    4   1   0  95
 26   12   0  599    42    0   82    0    1    1    0   154    5   1   0  95
 27   10   0 1064   578  535   83    0    1    3    0   155    3   1   0  95
 28   11   0  662    50    8   81    0    1    1    0   157    3   1   0  96
 29   12   0  632    43    0   83    0    1    1    0   158    5   1   0  95
 30   12   0  606    48    6   83    0    1    1    0   157    4   1   0  95
 31   12   0  616    43    0   84    0    1    1    0   160    5   1   0  95
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    1   0  207   310  149  112    1    2    0    0   147    4   1   0  95
  1   43   0  376    33    0   71    0    2    1    0    94    1   3   0  96
  2   38   0  111    33    0   67    0    2    0    0   285    2   0   0  98
  3   15   0   96    29    0   61    0    2    0    0    38    0   0   0  99
  4    1   0   49    20    0   44    0    1    1    0    15    0   0   0 100
  5    0   0   11    21    0   44    0    1    0    0    11    0   0   0 100
  6    0   0   18    15    2   33    0    1    0    0    58    0   0   0 100
  7    1   0   60    22    0   45    0    2    1    0   189    0   0   0  99
  8   68   0  170    73    0  151    1    2    3    0   503    4   0   0  96
  9   27   0  104    13    0   25    0    1    0    0    64    1   0   0  99
 10   26   0   87    21    0   44    0    1    0    0   264    1   0   0  99
 11    0   0    6    10    0   24    0    1    0    0     0    0   0   0 100
 12   16   0   40    40    0   80    0    2    1    0    17    0   0   0 100
 13    0   0   44    40    0   84    0    2    1    0    18    0   0   0 100
 14   15   0   52    14    0   29    0    2    1    0   115    0   0   0 100
 15   43   0  122    31    0   64    0    1    1    0   301    2   0   0  98
 16   43   0   90    17    0   35    0    2    0    0   301    2   0   0  98
 17    0   0    3    11    0   22    0    2    1    0     7    0   0   0 100
 18    0   0   18    28    0   58    0    1    0    0     2    0   0   0 100
 19   15   0 11252    16    0   33    1    1    3    0   816    2   4   0  94
 20   42   0   56    20    0   69    0    1    2    0   261    0   1   0  99
 21    0   0   61    34    0   67    0    1    0    0    12    0   0   0 100
 22   16   0   11    21    0   40    0    1    0    0    58    0   0   0 100
 23    0   0    5    10    0   20    0    1    1    0     1    0   0   0 100
 24    0   0    9    84    0  168    1    2    2    0   152    8   1   0  92
 25    0   0   13    14    0   30    0    2    0    0    93    0   0   0 100
 26    1   0   53    66    0  139    0    2    4    0    20    1   0   0  99
 27    0   0  332   497  471   54    0    2    8    0     2    0   1   0  99
 28   42   0   85    18    2   30    0    1    0    0   265    2   1   0  98
 29    0   0    1     4    0    8    0    1    0    0    13    0   0   0 100
 30    1   0    1    10    5    8    0    1    0    0    56    0   0   0 100
 31    0   0   36    80    0  153    4    1    2    0   206   64   1   0  36
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
Received on Tue Aug 7 17:01:32 2007

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:44:06 EST