For some reasons, I didn't see this summary ever appears, if it did or this
is not the first time you see this summary, please accept my apology. If you
do see this summary before, please let me know by email. Thanks.
Hi everyone,
My original question :
> I am looking for some utilities and information on memory management
>related matters for SunOS 4.0.3 and 4.1.
> Is there any utilities will display the current memory usage, like
>memory fragmentation, allocation per process, freed memory. And when does
>memory actually got freed and give it to some other process. For example, if
>I use malloc to get some 15M of memory, and then freed 5M, does it actually
>free the 5M instantly or just mark it as free. Or will this 5M be freed for
>other processes ? Is there a utility to tell current memory allocation.
>"ps", "sps", "top" .... etc, only give the total resident and size of the
>process, if I want more information, what do I do ?
> What is a good book on SunOS Internals (if that exists) ?
Thanks to all of you, I have got a total of 16 replies for my query, however,
about half of them just ask me to send a summary to them if I did get some
interesting answers.
Here is the summary :
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: shipley@remarque.berkeley.edu
>>For example, if
>>I use malloc to get some 15M of memory, and then freed 5M, does it actually
>>free the 5M instantly or just mark it as free. Or will this 5M be freed for
>>other processes ?
>
>Depends what version of malloc you are using. Most do not sbrk() down.
>
>>Is there a utility to tell current memory allocation.
>>"ps", "sps", "top" .... etc, only give the total resident and size of the
>>process, if I want more information, what do I do ?
>
>yes. "ps l" should do it.
>
>
>> What is a good book on SunOS Internals (if that exists) ?
>
>Look at the Usenix papers, (I also have a few I picked up
>a few years ago somewhere around here). Basicly look up
>on BSD and read kernal source. (I was about to write a kvmlib
>for 4.4bsd after a week of reading)
>
>
> -Pete
>
However, as I have already said, "ps l" only tells me the total resident
and size of the process. I am looking for something that will actually
tell me how many holes do I have, how large per page, etc.
I'll see if I can get a hold on those USENIX papers.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: keener@upenn5.hep.upenn.edu (Paul T. Keener)
>
>I don't know about getting the kind of detailed memory information you are
>asking for, but I have something which I have found useful. I hacked xload
>so that it would graph the amount of swap space currently used. It just
>takes the "swap" line of "pstat -T" instead of the load average. I have the
>source to pstat, so I just looking in the kernal for the info, rather then
>running pstat. If you would like the program, or diffs if you have the
>xload source, let me know and I will send them to you.
>
>> What is a good book on SunOS Internals (if that exists) ?
>
>The source, of course. :-)
>
> -paul
>
No, I am not interested in the usage of swap as a whole. I want
something that will give per process info on memory usage, like how big
the holes are if any.
No kidding, but the source cost $20,000.00 ND per cpu. I'll rather to
have another 4/75gx instead :-).
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: stern@East.Sun.COM (Hal Stern - Consultant)
>
>malloc() and free() dynamically manage your data segment. normally, when
>you malloc() something, malloc() looks through its list of "open" holes in
>the data segment and tries to give you something out of that space. if
>it can't, or there's no holes left to give, it moves your "break" point
>by adding another page to the data segment using brk(). if you then
>free this memory, the block is marked as a "hole" but the break point
>isn't moved -- the hole could be anywhere in the data segment, not
>necessarily at the top of it.
>
>however, in sunos, you can call brk() to move the break point *down*; this
>is useful in some cases (one of the guys at princeton's astrophysics
>depoartment wrote a new version of malloc() that actually takes pages
>out of your address space when you do a free(), if it can).
>
>the SZ field in ps shows you the total size of your stack (usually small)
>and data segments -- so this reflects how many pages you've really got tied
>up.
>
>--hal stern
>
This is exactly what I want to know, thanks Hal, it would be nice to
have a utilities to show the different holes size on a per process
basis though. As a matter of fact, I find out that in SunOS 4.1, it
did provide a "struct mallinfo mallinfo()" for some hints on the space
usage. Do a "man malloc". However, some of the systems we are working on
still running 4.0.3. May be they have something similar, I would like to
find out if any one of you know. Thanks in advance.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: rlg@ida.org (Randy garrett)
>
>If you get any good info, I'd also be interested. I pursued this for a
>while several months ago and didn't have much luck. The most
>informative things for me
>were vmstat and ps -alx. Another utility, hatstat, is similar to vmstat
>but shows PMEG allocations. I heard a rumor there was a memstat utility
>but we don't have such a thing. Anyway, good luck ...
>
>As for the SunOS internals, the best book seems to be:
>Desing and Implementation of the 4.3BSD UNIX Operating System by
>Leffler, McKusick, Karels, & Quarterman.
>
>Even though that's BSD, the principles seem to be pretty much the same.
>There are also two papers, both called something like Sun Virtual Memory
>System. One's by Gingell and the other is by McKusick I believe. These
>are a little helpful. I have the papers so I can look them up if you
>get really interested, but the book explains much more.
>
>Randy Garrett
>
Yes, most of the people suggest the same book as the best :
Design and Implementation of the 4.3BSD UNIX Operating System by
Leffler, McKusick, Karels, & Quarterman (or the DAEMON book).
It is a very good book on UNIX internals, I already have one, even the
SunOS Internals course offered by Sun Microsystems Inc., recommend that
as a reference and follow it quite closely. However, when it comes to VM
and memory management, it follows quite closely to VAX architecture. I am
interested in the Sun's SparcStation architecture, like what kind of MMU
they use, how do they work, and the various details in a similar way as
presented in the DAEMON book. May be there are some SUN papers exist ?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: auspex!guy@uunet.UU.NET (Guy Harris)
>
>The question isn't entirely meaningful.
>
>"malloc()" allocates *virtual* memory, not *physical* memory; not all the
>virtual memory of a process is necessarily backed by physical memory.
>"free()" doesn't give virtual memory back to the kernel. *However*,
>even if it did, it doesn't really "free it for other processes"; if it
>doesn't do anything with the freed-up locations of virtual memory, the
>physical pages that back those locations up can be stolen by other
>processes to back up *their* virtual memory.
>
>The only resource "permanently" consumed by virtual memory is swap
>space; if "free()" were to give virtual memory back to the kernel, it
>*would* end up freeing swap space, but wouldn't free up physical memory
>*per se*.
>
I am not thinking about just physical memory, but VM instead. I don't
want my process keeps on growing and growing but never free anything.
And eventually it's gonna consume all the VM, and I'll get out of memory
error. Of course, I am not talking about freeing physical memory. If my
process keeps allocating memory, it'll eventually run out of swap space.
I am interested to find out how to actually free up the VM I don't need
instead of only marking it as free.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: blknowle@frodo.JDSSC.DCA.MIL (Brad L. Knowles)
>
>Tim,
>
> It sounds like you have a very good handle on what you need to know, you
>just don't have the books immediately available to you. Although it is not a
>book on SunOS per se, I like "The Design and Implementation of the 4.3BSD Unix
>Operating System" by Leffler, McKusick, Karels and Quarterman. It has told me
>a lot about the 4.3BSD Unix OS, most of which should be applicable to SunOS.
>
> Now about how SunOS frees memory. Assuming SunOS does it the way any other
>Unix I have ever encountered, then it will keep only a small "working set" of
>pages in memory. The rest of what you have allocated will be paged out to the
>swap space (and paged in on demand). As soon as you free memory, the part of
>your working set that was paged in that has now been freed will be returned to
>the OS immediately for re-use. The stuff on the swap space will be marked as
>free and over-written whenever it is needed. The way the memory is returned to
>the OS for re-use is of some importance, though. The OS uses a "two-hand
>clock" algorithm to determine what pages can be swapped out, and which ones to
>keep in RAM. All of the memory will be kept on a circular (and doubly linked)
>list. The algorithm will go through the memory being used, and mark each page
>as potentially swap-able. After a predetermined amount of time has gone by (or
>number of pages has been seen), the algorithm will go through memory again.
>Any page that has been used since the previous pass will be kept in memory, and
>any page that has not been referenced since the previous pass will be swapped
>out. This "circular linked list and two-hand clock algorithm" is an integral
>"theme" in 4.3BSD, as you will see it crop up in almost any place where cache's
>are kept. Anyway, as soon as the memory is freed, it is marked as usable,
>which typically means that it will be immediately re-used, but this is system
>load dependant. Of course, the book I mentioned goes into far more detail than
>I can, and does so far more clearly.
>
> On pages 163-165 of the book mentioned above, a *VERY* complete
>bibliography of how memory management is done in Unix is presented. If there
>is an important paper or article that discusses memory management in Unix that
>was written before 1989, it will be listed here. If you *REALLY* want a copy
>of this bibliography, I can type it in for you, but you really would be better
>off buying the book for yourself -- there is so much more that it can give you
>than just the memory management stuff.
>
> Anyway, I hope this helps.
Yes, I've already have that book. Thanks for your description, but I
think that is the job for the pagedaemon. I am interested in the vm
allocation from the process' point of view (or frame of reference :-).
Yes, the physical memory page will be paged out when not referenced for
a period of time, but I am interested in the actual de-allocation of the
VM previously obtained by malloc and the utility to display them.
But, definitely, the DAEMON book is very good. Thanks.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>From: blknowle@frodo.JDSSC.DCA.MIL (Brad L. Knowles)
>
>Tim,
>
> By the way, you might want to check out perfmeter (1). It probably won't
>do everything you want, but it will give you a handle on paging, swapping, cpu
>performance, etc.... You may also want to check out rstatd, and the other
>statistics monitoring/reporting commands, etc..., provided in SunOS.
>
> In all probability, in order to get *EXACTLY* what you want in terms of
>system performance monitoring, you will have to write your own monitoring
>code. Good luck!
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Thanks to all of you guys for providing the information. Even though
sometimes it wasn't exactly what I want, BUT, it definitely provide me
some insight or hints for further pursuing of additional information.
Thanks again.
Tim Chan
================================ Address ===============================
Tim Chan, System Engineer, Teknekron Communications Systems
(415)-649-3645 2121, Allston Way, Berkeley, CA94704
uucp : uunet!tcs!tim Internet: tim@tcs.com
Disclaimer : I only work here. :-)
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:06:12 CDT