SUMMARY: ClearCase VOB server memory requirements

From: Janis Lykakis <Janis.Lykakis_at_ehv.pdsl.philips.com>
Date: Mon Nov 11 2002 - 05:59:05 EST
Hi There,

My questions was:

Currently, our CC VOB server is showing some performance trouble.

It's a E3000, two CPU's (248 Mhz) 2GB memory and Solaris 8.

Recommendations state that the amount of memory should be:
VOB database size/2

Our database size is 14GB ergo, 7GB main mem would be needed.

BUT, what we don't understand is, that if we don't have enough
memory, than why is the SR (ScanRate) parameter as shown by
vmstat -S not greater than zero:

procs     memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m2 m3   in   sy   cs us sy id
 0 0 0 2917976 1114016 0  0 724 4  4  0  0  0  0  0  0 1575 1123  108 18 21 61
 0 0 0 2702088 908768 0   0 501 5  5  0  0  0  0  0  0 1951 18250 3184 27 29 44
 0 0 0 2700384 907776 0   0 696 0  0  0  0  0  0  0  0 2248 20124 2622 37 38 26
 0 0 0 2705832 910832 0   0 698 2  2  0  0  0  0  0  0 1741 19004 2512 34 29 36
 0 2 0 2718848 918688 0   0 2106 2 2  0  0  0  0  0  0 1929 21766 2643 41 31 28
 0 1 0 2718848 918992 0   0 1541 2 2  0  0  0  0  0  0 1889 17221 2606 32 29 39
0 1 0 2718832 919264 0   0 1077 8 8  0  0  0  0  0  0 1803 22904 2581 47 33 20
 0 0 0 2718848 919368 0   0 576 16 13 0  0  0  0  0  0 2298 20052 3341 42 31 27
 0 0 0 2718424 919240 0   0 1077 8 8  0  0  0  0  0  0 1827 21083 2543 41 34 26
 0 2 0 2717944 918984 0   0 1453 10 8 0  0  0  0  0  0 2013 18530 2953 41 27 32
 0 1 0 2715416 917480 0   0 602 0  0  0  0  0  0  0  0 1540 20359 2190 36 28 36
 0 0 0 2715416 917416 0   0 626 13 10 0  0  0  0  0  0 1681 18923 2518 35 29 37


We do see some other issues on te machine, there seems to be an WIO issue,
but now I'm trying to find out if we really NEED 7GB in our VOB server.

Any help would be greatly appreciated.


I got several replies, thanks you all. But I still don't know how to
prove whether or not we need more memory. We plugged in another mainboard
with 2GB mem and 2 CPU's. Some users reported an improvement, some didn't
and vmstat reports 2.5GB free memory now.

Reply's:
------------------------------------------------------------
Thomas Autry:
Some good information on performance tuning a system as well as capacity 
planning can be found in the following guide:

http://www.sun.com/blueprints/0902/816-7882-10.pdf

It is a design guide for the sun fire series but can be used for planning of any 
system.
------------------------------------------------------------
Matt Harris:
It can't hurt.  I'd say upgrade those old processors, too.  :-)
As far stuff to check, you may want to check out your FS cache
performance.  Depending on what kind of database it is, and what kind of
filesystem it's sitting on, there's probably at least one or two simply
kernel tunables that could increase your performance, as well.  If it's
an internal database, chances are they're using DBM or somesuch (just
making an educated guess here, I know nothing of this specific product),
then FS tunables could really help a lot.  If it's Oracle on a raw
partition, then that could explain the scan rate thing... Since I don't
believe that Oracle's raw data devices go through VFS at all... In
reality there's a billion things it could be or that could help, if you
can provide more details as to exactly what the system is doing a lot
of, what it's doing it to, and how it's doing it, I can probably provide
more exacting advice.  :-)
------------------------------------------------------------
Ed:
We run ClearCase here on an E3K, similar to yours, but with 1GB main
memory for a 4.5GB VOB and have no issues with it.  I have never seen the
VOB/2 recommendation from Rational.  What version are you running? 

Part of the rationale may bet that MVFS structures are memory hungry in
newer versions of the software.  Beyond that, sr almost always seems to be
0 in Solaris 8 and 9 due to the introduction of the disk page vs memory
page caches.  SR would often top out when file cache pages took up too
much available memory and had to be paged out to make room for process
memory pages.  This seems to be the real-world result of running high
memory, low-to-mid utilization servers under Solaris 8.  Ask me again when
I get Oracle running on a Solaris 8 box, though. ;)

------------------------------------------------------------
Wanke Matthias:
I'm lost about VOB (what's that?) but i doubt that for a total DB size of
14GB you need 50% in memory; normalay 1%-5%
should be enough, given all the sophisticated algorithms of todays DB
engines; as i see in your output you have nearly 1GB RAM free, so are there
measurements or metrics that support the fact that you actually have a
problem (on the OS side) - where can one see the problem? I don't seem to be
able to see it in the output!?
I don't know about your application but sys-time seems a bit high on a 2
processors machine - do you have software raid on your disk subsystem?

------------------------------------------------------------
McIntosh, Alan:
Wow, does this sound familiar!  We were having the same
problems with our builds due to memory limitations on
our VOB servers.  Don't believe that crap that Rational
is feeding you about "Total VOB database size/2", our 
largest VOB was approaching 37GB with a total of over
80 VOB's at well over 180GB.  That would have meant we
would need around 90GB of memory according to Rational!

I have attached a PowerPoint presentation for the current,
planned, and future recommended environments for our builds.
We actually only increased our memory on the VOB's to 12GB,
rather than the planned 28GB, and added another VOB server
to split the load, which had an additional 12GB.

Our build times went from almost eight(8) hours to around
2-2 1/2 hours after the switch!  By using current equipment
obtained from other departments, we were able to make the
changes for under $50,000 as opposed to other recommendations
that would have cost well over $250,000.  Hope this helps.
------------------------------------------------------------
Kevin Buterbaugh:
    I'm not familiar with ClearCase VOB - does it use shared memory like
other databases (Oracle, Sybase) do?  If so, what do you have your
shminfo_shmmax parameter set to?  Generally speaking for DB servers, the
more memory, the better.

     If, on the other hand, CC VOB doesn't use shared memory, then the
statistics you include seem to indicate that you do not need more memory.
HTH...

------------------------------------------------------------
Karl Vogel:
   Two things come to mind, swap and scan rate.  Try adding the lines below
   to /etc/system and rebooting.  I run with the swapfs_minfree and autoup
   settings and they've never caused me trouble.

*
* Swap
*   System keeps 1/8th of all memory for swap, which is too much for
*   a 4GB system.  Reduce that to 32 Mbytes (4096 8K pages).
set swapfs_minfree=4096

*
* Memory management
*
*   http://www.carumba.com/talk/random/tuning-solaris-checkpoint.txt
*   Tuning Solaris for FireWall-1
*   Rob Thomas robt@cymru.com
*   14 Aug 2000
*
*   On firewalls, it is not at all uncommon to have quite a bit of
*   physical memory.  However, as the amount of physical memory is
*   increased, the amount of time the kernel spends managing that
*   memory also increases.  During periods of high load, this may
*   decrease throughput.
*
*   To decrease the amount of memory fsflush scans during any scan
*   interval, we must modify the kernel variable autoup.  The default
*   is 30.  For firewalls with 128MB of RAM or more, increase this
*   value.  The end result is less time spent managing buffers,
*   and more time spent servicing packets.
set autoup = 120
------------------------------------------------------------
Jeff Woolsey:
> Currently, our CC VOB server is showing some performance trouble.
> 
> It's a E3000, two CPU's (248 Mhz) 2GB memory and Solaris 8.
> 
> Recommendations state that the amount of memory should be:
> VOB database size/2
> 
> Our database size is 14GB ergo, 7GB main mem would be needed.

That seems inordinately huge, from my fading recollection of ClearCase.
IIRC, they're talking about the VOB _database_, not the files in it.
It's sort of like filesystem metadata, except that ClearCase has a lot
of metadata, as filesystems go.

> BUT, what we don't understand is, that if we don't have enough
> memory, than why is the SR (ScanRate) parameter as shown by
> vmstat -S not greater than zero:

The above would seem to apply.  A VOB database is not the whole VOB.
------------------------------------------------------------
Tristan Ball:
Clearcase relies on the disk cache for performance. You _must_ have a high cache 
hit rate for good performance. In particular, think about this as a hypothetical 
example case:

A cache hit takes 1ms, A cache hit takes 100ms.
You have 100 data accesses. At a 100% hit rate, that means 100ms total access 
time. At 99%, thats 199ms access time, at 95% that 595ms, and 80% its 2080ms! 
2080ms vs 100ms is obviously huge. :-)

In reality,a  memory cache hit is generally going be in the order of tens or low 
hundreds of nanoseconds, and a disk hit is going to be 10-50ms, I used 1 and 100 
to make the math easy and clear.

Clearcase accesses the metadata a lot, and if you're getting cache misses, 
performance drops through the floor. You don't have to run out of memory to see 
this, you just have to have too little to cache effeciently.

Rational's recomendation for the memory size is roughly correct, although if 
most of your metadata was legacy information and not often accessed, you could 
probably get away with a vobdb/3 ratio. I try and aim for a vobdb/1 ratio, and 
then wind that back untill the finance guys stop giving me "the look". :-)

I can possibly help with your WIO issue, if there really is one. I'd be looking 
at "iostat -dxcn 1"  output during problem periods if I was you.

Of course, increasing ram will reduce the number of IO's anyway. :-)

For your system, if you've only just started seeing problems, and it was OK 
before, I think you'll find that adding 2G will return your performance to 
previous levels.

I'd also point out that while the cache example above is pertinant, it's not 
always cost effective to get the highest possible hit rate. If there's only one 
access a second, then going from 90%->95% might not even be user noticable!

And as for how I know all this, we went through exactly the same process. It 
really didn't look like the server was loaded. <25% cpu load, 100 ops/sec on a 
disk array that I knew would sustain 600+, <10% network traffic on a gig eth 
link. We went from 2g to 4g on our E450 and the engineers actually cheered.

As we have ongoing discussions here about our rational infrastructure, I'd love 
to hear a little about your setup, number of users, types of clients, where your 
views are stored. We're currently looking at upgrading ourselves... :-)


------------------------------------------------------------
Janis Lykakis.      E-mail: janis.lykakis@ehv.pdsl.philips.com
Philips PDSL
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
Received on Mon Nov 11 06:02:29 2002

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:42:57 EST