This is going to be a long summary.
Firstly, let me apologize for the miscommunication - I had my
email client misconfigured and when someone pointed this out, was
able to get my ISP to redirect the mail.
If anyone would like to continue discussing these issues in private
email, I would welcome the discussion.
*** Brief Summary ***
For those with short attention spans, I was asking about the ideal
way to set up a new Sun machine's filesystems.
A brief summary might be:
- the judge is out on performance. On the same disk, it's probably
negligible
- it depends - what are you using it for? server, user workstation, what?
- classically, when disks were small and few, multiple partitions were
used. Today, this is no longer the case, so using few or single
partitions is acceptable.
- /var holds a lot of log stuff that can overfill the disk, so it may
be useful to have on its own partition.
- consider the issues of backups and o/s upgrades when you lay out the FS
- read Adrian Cockroft's book on Sun Performance
- /usr is very static so probably does not need its own partition
- remember than in single user mode you need to have /usr come up on its
own.
- consider new (to me) things like disk striping, Solstice, etc
- consider NFS mounting /usr/local/bin for shared binaries
- consider making swap different from /tmp
- consider linking /usr/tmp and /var/tmp to /tmp
- consider syslogging all info to a central logging machine
- Solaris 7 may have special features that changes your previous
assumptions
*** Full Summary ***
The full summary follows. I sincerely appreciate everyone's
(in some cases, multiple!) replies.
For the record, yes, our CTO is a former sysadmin, and since we are
a small company (about 35 people but growing quickly) everyone is
involved with multiple tasks. I am the new sysadmin here and
trying to make improvements without breaking things.
Obviously I have a lot to learn about "sysadmin in the 90s" :)
No, he's not on crack, and he's not an NT guy!
I asked:
> Sort of a philosophy here. When setting up a new Sun, I usually
> allocate space for /, /var, /usr, and a bunch of other filesystems
> called /a, /b, etc which I dole out as needed.
>
> At my new job, our CTO is requesting that we follow his model of setting
> up the filesystems, which is to allocate a small / partition,
> and tons of space in other partitions, but no /usr or /var.
>
> His reasoning is that by making /usr/local a link to say /a/usr.local,
> he can vary the size of the filesystems as needed by rearranging the
> link structure. He also asked why we couldn't just use a single huge
> partition.
>
> My reasoning was that I want /usr and /var since /usr will hold a lot of
> local stuff, and /var will hold most of the log files. I think having
> more partitions will improve i/o, but I can't prove it without testing.
>
> So... I'm wondering how everyone else sets up filesystems, and if maybe
> I should try this new scheme, or stick to my guns and use the tried and
> true method.
I also followed up with a rephrased version:
> Regarding setting up a filesystem layout for a new Solaris machine:
>
> - is it advantageous to give /usr and /var their own filesystems?
> - can the same advantages be achieved by have a single huge
> partition?
> - what are the tradeoffs between a filesystem with say /,/var,/usr
> each having their own partitions, as compared to another machine
> with only a / partition, and /var and /usr being symlinks to subdirs
> of another filesystem?
>
> All scenarios assume that there are other partitions that will hold
> user data such as /u, /home, and so forth.
>
> This question is not merely academic - our CTO and I disagree on how the
> filesystems for a new system should be laid out, and I want to make the
> right choice. I am favoring the "classic" approach, reasoning that it
> will give more flexibility and improve i/o. He feels that the symlink
> method or a single huge partition are preferable.
Replies to both questions follow are below.
Note, I also asked if there was a list for NT admins like this one.
>A lot of people were interested, and the only reply was:
>The folks at www.sunbelt-software.com sponsor an ntadmin list.
>If you read their home page carefully, you should be able to
>find the links to their signup page. The list has a handfull
>of STRONG contributors but is a busy & "chatty" list.
Might I suggest that if the list mentioned is not ideal, that we consider
starting our own?
-- I've gone to this method on servers:/ /usr /var /tmp /home
Disks being the size they are these days I size the partitions generously, except the root partition. I figure root and usr are mostly read-only, while var and tmp are getting a lot of read/write activity. If I have multiple disks I'd look at moving var or tmp to another disk, split up swat, etc. I doubt having little partitions helps performance when they're on the same disk. Splitting the different filesystems onto different disks should help.
On workstations I get lazy and do one partition for the system, and one for users. So I'll have:
/ /home
On Solaris I always have /tmp in a seperate partion and use swapfs. Just because that's the way it installs from the CD.
For Linux/BSD I use a swap partition and don't bother with a seperate tmp except on a server. -- Yeah !!! Your method is a better one... But proving that is ......................
Better stick to u'r own method. -- There are many factors involved in the decision. You may want certain partitions on seperate disk for io performance etc. You may be expecting large log files in /var. For our mail servers /var/spool/mail is a separate partition.
I like to have a smallish / partition, less chance of corruption. A seperate /usr and /var ('cause some run away log file may file /var) and I definitly want a seperate partition for /usr/local where I install 3rd party software. I will usally make /usr bigger that the bare min for some software that doesnt like being installed elsewhere. /home is always a separate partition. -- /var is a good one to limit to itself (with some size) as you don't want it going off the rails and filling up everything else.
As for I/O, it's actually faster to have it all in the same partition, as UFS can arrange inodes and blocks better and it isn't forced to seek off to another partition when you cross links/mounts.
Personally, I do one partition for the OS/local, one for /var, then stripe everything else I have together if there are multiple disks, or just make one big one otherwise.
A side note - if I/O is the issue, more spindles and striping/mirroring is the win, not multiple partitions. -- We came across the same situation when getting our first Sun E3500. Our previous unix experience was with ICL / Fujitsu DRS6000's and the philosophy there was to have / /var /usr /home etc etc etc as their owm mount points.
But the Sun method seems to be to tip everything into / , so as the /var & others are no longer mount points. --
He'll rue that the day he can't mount /usr...
> link structure. He also asked why we couldn't just use a single huge > partition.
Common enough, but I prefer the firewall approach.
> My reasoning was that I want /usr and /var since /usr will hold a lot of > local stuff, and /var will hold most of the log files. I think having > more partitions will improve i/o, but I can't prove it without testing.
You put local stuff in /usr? Don't - treat it as read-only. Put local stuff in /usr/local instead. -- The way I set up filesystems now (ignoring home directories and other data partitions) is to use one humongous / partition. On a server, or a machine which will have a lot of optionally software locally installed, I create a separate /opt partition, sized as desired. On servers, I have a separate /var so that runaway logging or mail can't fill the / filesystem. Before I started using one big / I wondered a lot about why not to do it and there is no real reason. It was useful in the good old days when tapes were small so that you could always do a full dump of / to a separate tape to aid in quick recovery. Nowadays with faster machines and bigger tape drives this is not an issue. For me the clincher was to learn that this is what Sun's system people do on their own workstations - if it's good enough for them . . . -- Personally I use a fairly large root with a separate /var but no /usr, /opt. I also have a /pkgs (which holds "depot" and commercial stuff like oracle and sybase). There is a /data which acts as a mount point for various databases.
Why?
Well a really big / means less hassle and remember that / and /usr have to be available for single user mount (i.e. recovery of systems). Why not include /var? Because it is full of log files which have a habit of filling up the filesystem with little or no warning (and often as root which means they go past the 100% full mark).
I don't use /usr/local and all non-Sun software ends up either as /pkgs/Software or /pkgs/depot/Software (if being used under depot). The latter means that users have /pkgs/bin in their path but can see all of the stuff in /pkgs/depot/*/bin.
Hope that helps, happy to answer in more detail if required. -- Unless you have a machine with just one harddrive it's definately = preferable to have a /usr and /var
The biggest consideration for me in having at least a seperate /var is = the fact that i don't want error logs or /var/mail or /var/spool to = suddenly write my root partition full. The first warning you usually = get is when the machine just crashes, unless you implement some kind of = cron job or manually go and check the size of the root partition. And = yes, it is possible for a user to generate a error log file that exceeds = 2gig. :-\
Improving I/O can be a consideration when using a seperate /var or /usr = as you could typically accelerate these file systems using something = like prestoserve. -- The way I set these things up is the create one large partition for / (root), /usr, /var and so one. I also dedicate a disk to this and swap. Any additional filesystems I put on other disks.
The reasoning behind this is simple:
1. By fragmenting a disk into numerous filesystems you waste space. You end up with lots of small fragmented areas.
2. You slow the disk down by fragmenting it, biggest part of an IO operation is you do seeks.
3. You want space under /var for logs files to grow, the adding of patches and packages (/var/sadm).
4. Speed of recovery should the disk fail - you only have one volume to restore, the flip side of this is that backups are speeded up as your only backing up one volume instead of multiple volumes. Also if you share a partition with applications and the filesystem requires a fsck then you increase the time of the fsck thus increasing recovery time (fsck depends on number of files not size of partition).
5. Gives you space if you wanted to do an upgrade.
6. Disks are cheep.
7. Seperation of application and OS (backups, recovery, upgrades etc).
One thing that I've also noticed is that a lot of admins make separate filesystems for / (root) and /usr and then change root default shell from bourne shell. This is bad news for booting in single user mode, if root and usr share the same filesystem then this problem dosen't arise (although I still wouldn't change roots default shell). -- Not again. This topic starts wars going. Not being a fascist I don't have an axe to grind. My answer is: it depends. If you have the disk to spare (4Gb currently but it used to be 2Gb for Sol 2.3) then I make / encompass /, /usr /opt & /var if its a workstation or a lightly loaded server. A server with lots happening I shave off /var. Be flexible, there is no one answer for all circumstances. -- Note that with Solaris, what used to be /usr is now partially regrouped to /opt ...
> At my new job, our CTO is requesting that we follow his model of setting > up the filesystems, which is to allocate a small / partition, > and tons of space in other partitions, but no /usr or /var. > [...] He also asked why we couldn't just use a single huge > partition.
Because one huge mail (affecting /var{/spool,}/{mail,mqueue}) would make the box keel over (OS running out of space in /tmp and /var/tmp), rather than only affecting a couple services, or possibly even only sendmail.
> His reasoning is that by making /usr/local a link to say /a/usr.local, > he can vary the size of the filesystems as needed by rearranging the > link structure.
This is true - been there, done that, reluctantly - but has nothing to do with whether those directories have been set up as symlinks *from square one* or not.
> So... I'm wondering how everyone else sets up filesystems, and if maybe > I should try this new scheme, or stick to my guns and use the tried and > true method.
I start up my brains with every single machine I configure. A mail gateway gets a separate large /var/spool/mqueue, a cluster's fileserver lots of partitions for /home, a loghost large /var/log and/or /var/adm, the computing servers we run BDDs on (type of computation where "it starts to swap/page" means failure) get less swap than the boxes where I keep a bazillion X11 windows in umpteen virtual screens, some NIS servers have a separate *small* /var/yp just so that I can 'dump' this important and frequently changing stuff more easily, etc. etc.. -- Hello, I sort of agree with your CEO and sort of agree with you. So I think a combination is the best from you two is the way to way. /usr should be there and should NOT be host any LOCAL stuff. Setup /, /usr, /var, and give the rest to /opt. /usr/local should be a link, I link it to a nfs filesystem. -- The use of multiple partitions for the operating systems was needed when the size of the disks, could not reasonably contain all of it in one filesystem. Having /usr as a separate filesystem made sense, when diskless clients were the norm. But todays standard of clients with multi-gigabyte disk drives on clients, it doesn't make sense micromanaging the drive.
I use only two partitions, root and swap.
Note: I also mount /usr/local from a server for site-wide executables. Ones that I want particular to a machine, I build and install in /opt/local. -- Unless your multiplicity of partitions are on separate drives, you are unlikely to see any i/o performance gains. I have, for several years, been building client systems with just / and swap partitions. My reasoning being that the free-space is shared between all variable sized files, thus extending the time between system clean-ups. Servers are treated differently, to protect against problems such as corrupted discs or filled spool directories locking the system. ( /var/spool for instance being a separate mount point ). Obviously, if you are creating standalone systems with local non-system software, separate partitions such as /usr/local will be a good idea for maintenace and backup reasons, but again will only give a performance gain if using separate drives. -- I'm an advocacy of default settings.
I have found more evidences to support my "philosophy" :)
I am having a dead DEC Unix box right now. The root and /usr fs's are in AdvFS...well, it doesn't matter what FS.
The thing is because the root disk is dead, I don't know what sort of size it was before. So now if I spend the whole night letting the restore run and boom, out of disk space! Time to re-do restore.
Had I used default, I need to replace the same type of disk, accept default partition layouts... and then my restore would fit and I'd be happy.
People keep track of serial numbers, hardware type, models, what a system has...they seldom keep track of partition layouts.
Playing with partition cylinder/sector and using format is kinda boring after a while. I need to either relax or read docs.sun.com to open my mind and stay marketable.
Interviewer: What did you do at your last job?
You: Uh...we spend time doing the routine stuff plus, I'm going to impress you by telling you we do cool things with format and trying to make the partition slices adhere to certain "magic" ratio...I'll impress you with my math skills...how about the ratio of Pi/2? -- Everyone's opinion on this is somewhat religious, but you asked, so ...
I still believe it is a good thing (tm) from having / be a smallish partition that only holds root and kernel related files and basic binaries required in a debugging or recovery boot. The size of actual files in root should not change significantly over time. The "no change" rule means that /tmp should not be on /.
By the same token, it is a good thing to have /var be its own filesystem. /var holds log files that grow rapidly and you do not want them impinging on either root or functional files.
I will, however, agree with your boss that /usr is a filesystem like any other, including /opt, /a, /b, /c. I see no reason not to symlink it away.
As for performance, I've seen that be dependant on not overusing a single spindle, but if all your partitions are on the same physical disk it should not matter much if there are many or few of them -- just how much you use the whole thing. -- We have done with single big partition for several years and are very satisfied with it. One reason to have a small file system was the backup media but that is not a problem anymore. Also disks are big so the stuff in var gets to be maybe 200M which is nothing compared to 18G. So one partition. -- When I first got into this business, I had 2 IPX's with 350MB drives,which were divided into /, /usr, /export, and swap (I think,it's been 6 years). When I got more systems, I would continue a similar scheme, although I was dealing with drives up to 2GB.
About two years ago, when we were settiing up two E5000's with 2GB internal drives, the Sun tech partitioned / for 1GB and swap for 1GB. We talked about it, and I tend to agree with him that partitioning the root drive doesn't improve anything, and probably uses the space available on the drive more efficiently. Since that time, I have partitioned new systems about like this (assuming a 4GB internal drive):
slice tag Size 0 root 2GB 1 swap .5GB 6 unassigned 1.5GB 7 unassigned 2-5MB
I don't see how I/O is improved by spreading /usr and /var over different areas of the same drive, same controller; maybe I'm missing something. If /usr and /var (etc.) were spread over different drives, then I/O may get better.
I have a Sparc4 with an internal .5GB drive partitioned like this:
slice tag Size 0 root 45MB 1 swap 90MB 6 usr 300MB 7 home 70MB
After I got everything up and running on it, there was a revision to one of the software packages which needed about 65MB free space available, which I had, but I couldn't use. The free space was distributed among the three partitions, but I couldn't get to it.
I wound up physically stealing a standalone 9GB drive and using it as a ufs mounted /home. This, of course, cost me all the 70MB of slice 7 ,which I can't use for anything as it's too d$%^ small.
My suggestion, don't spread the system files over several partitions of the boot drive. Use just one partition for the system files, and make it as large as you can. Put your files which will need a lot of I/O onto other drive(s). -- Put little stuff in /usr / /var or what have you in there.
/usr could be /usr/local/lib and some important stuff that the sytem needs to boot.
When you do a backup of the "system" you capture / and /usr and when time come to do disaster recovery, you will have the system back faster than a humongous /usr because these are typically small by default.
But philosophically, you can treat / /usr /var /tmp as a separate entity...ie, resising in an isolated disk, fully mirrored hopefully...or whatever high availability you employ.
That's can also be treat as "system" partitions...
Data partitions on other disks, RAID...whatever.
Making /usr / /var /tmp bigger encourages abuse...like have a huge space for core dumps...assume you didn't limit your core dump size then a "system" restore would have to be done for those useless core files...assume again, your backup tool isn't flexible enough for you to pick and choose files to restore.
But then when a system is down, is there time to pick and choose, read the manual, or even experiment...I prefer to blast a command at it, let it does it things, and evade nosey people who are just asking for status every second. -- I've been using a single large filesystem (plus a chunk for swap) for my boxes here. It works well, and you don't have to worry about juggling space between filesystems. Aside from swap, I don't think that your OS disk should be doing TOO much I/O, so it shouldn't be a bottleneck. If I do have an application that does a lot of I/O (like our netscape proxy servers' cache), I put those files on separate disks. -- Just a single data point for you - When installing to 2.6, I asked Sun about partitions, and they recommended a single partition. On one system, I did that. / is 2.1GB. Installing the OS was very simple with one partition.
Note, applications goes on another disk, and you may want /var on another disk - in case it fills up - it won't crash the system. -- The "standard" filesystem layout for Solaris is something like this:
/ ( root ) is ( of course ) required. Depending on other filesystems created, it can be fairly small ( < 50 MB ) or larger if you don't do separate filesystems. The /usr and /usr/openwin filesystems are now considered to be optional as separate filesystems from Sun's perspective. For myself, I usually don't bother separating out /usr/openwin, but in most ( not all ) cases I do like a separate /usr partition. The /var filesystem is nice to have separate, since a lot of temporary files live there, and filling up the root filesystem is uncool. /opt is just what it says, space for optional software.
You probably know most of this, or all of it, but here's the kicker that may help you in dealing with your CTO. When doing an upgrade type OS installation, the Solaris install program will usually try to recreate the following filesystems if they exist as separate partitions:
/ /usr /var
In short, in most cases it's not really worth the effort to go too far from Sun's filesystem layouts. If you have some unusual requirements, it can be done. -- Depending on the purpose of the machine, going with a single partition may not be bad. I wouldn't do it on a server, or a critical machine. For a general-use desktop, maybe.. For simplicity's (and your sanity's) sake, keep things simple. Having a separate /var and /usr is, IMHO, a good idea. Symlinking them adds complication, and performance hits.. -- /usr should be just big enough for the OS. You shouldn't put anything local (GNU stuff or X or other) in there IMHO. /var size depends on if this is the loghost for your operation or if it is the mail server. You aren't logging everything on each individual machine are you? How would you ever keep track of that?
My typical layout these days looks like
/ 80Meg /usr 560 Meg (Solaris 7 64bit wants about 500) /var (not a separate partition on the client machines.. it sits in /) /tmp (depends on how much ram I have. 256Meg on 128M ram machines.. up to about 1 gig on machines with lots of RAM)
/disk1 (that is where the rest of it goes.. local scratch areas for when people are running programs that produce lots of data that I don't want dumped to the main file server)
/usr/local /opt/whatever (mount points back to application server) -- I've found that with huge partitions it becomes more difficult to balance backups. Also since a large partition can take much longer to backup than a smaller partition, I just find them easier to deal with.
> My reasoning was that I want /usr and /var since /usr will hold a > lot of local stuff, and /var will hold most of the log files. I > think having more partitions will improve i/o, but I can't prove it > without testing.
Assuming your /, /var, and /usr are large enough, you could also upgrade the OS without repartitioning the disk. This means that you wouldn't have to restore the user partition after an upgrade ... which could be a problem if the partition was full and you had to decrease the user partition to get enough room for the OS. If they aren't large enough, then you would be better off with a single large partition. I especially like a separate /var since that isolates log storage space from users ... since we don't enforce quotas.
> So... I'm wondering how everyone else sets up file systems, and if > maybe I should try this new scheme, or stick to my guns and use the > tried and true method.
Because of our old backup hardware (which we're replacing) and SunOS machines (which we're retiring) we generally set up 2GB partitions everywhere. With new larger and faster disks and tapes, we'll probably go to a 4 or 6GB partition size.
-- I make the root as large as possible - pkginstall programs insist on using /opt/ as their location and this by default is on the root partition. So my boot drive is /tmp and / and that's it. Everything else is elsewhere.
I used to use the Sun defaults and got really tired of having to almost immediately start moving things to other drives becuase the root was getting filled up.
Filesystem kbytes used avail capacity Mounted on /proc 0 0 0 0% /proc /dev/dsk/c0t3d0s0 832838 467603 306937 61% / fd 0 0 0 0% /dev/fd swap 618772 1024 617748 1% /tmp /dev/dsk/c0t0d0s0 1952573 1036709 720607 59% /dr1 /dev/dsk/c0t2d0s3 1952573 213187 1544129 13% /dr3 /dev/dsk/c0t4d0s3 8345201 2695864 4814817 36% /dr4 /dev/dsk/c0t5d0s3 8342893 270374 7238230 4% /dr5 /dev/dsk/c2t6d0s6 16726786 7452690 7601418 50% /dr6 /dev/dsk/c2t4d0s6 16726786 10744745 4309363 72% /dr7 /dev/dsk/c0t1d0s7 579302 344650 176722 67% /export/home --
The structure I have used successfully for system installations is as follows:
/ (which includes [root] and usr) - 1 to 2GB (this partition should stay pretty static unless you plan on installing a bunch of answerbooks into usr, then you might want to go with 2GB, plus you now have room for future upgrades to Solaris. I can't tell you the number of times I have had to ufsdump/ufsrestore a system to re-partition a drive so that I could perform an upgrade because the initial setup of the system skimped on space for the / and /usr partitions.) swap - depends on amount of memory (but usually 2X RAM) /var - 500MB (or more if a mail server) /var/sadm is where Solaris saves all patched info /opt - rest of the drive ... any other filesystems you might need possibly on separate disk drives
I make a symbolic link from /usr/local to /opt/local for installation of GNU software and other stuff...
Having a separate /usr is from the days when disk drives where small and expensive. Now disk is huge and cheap. I wouldn't start out with anything less than a 9.1GB UWSCSI these days. In fact I am looking at 18.2GB drives as standard... -- Ah, a question of religious proportions :-)
Once upon at time, when disks were small and volume management was a misnomer, best industry practices dictated that root should be roughly 25% of available disk space, and a separate partition should be set up for /usr and /var...mainly because on a well-used system, the size of content on both those filesystems tend to grow exponentially. In this scenario, if the directories /usr and/or var were in the root filesystem, root would fill up, and lock up the box. Nowadays, since disks are huge, and volume management places a layer of management between inodes and directories, making one filesystem across the whole shooting match has become a common practice on systems with >2GB total physical disk space. All but the die-hards are doing it, or those *not* using volume management. If protecting your OS is a priority, or you feel your OS is at risk from activity taking place on the machine (i.e. development), it may be a good idea to separate /usr/home, /home, or /export/home, as the case may be, from the root filesystem. I read a good paper on this in one of the Linux rags lately, and if I run across it again I'll forward it to you. As far as speed goes, I'm looking forward to your summary to see if anyone *has* tested that. NT mailing list similar to sun-managers? I wish there were, but I can't find the means to believe that NT admins/gurus would be as self-moderating as the Sun gods who subscribe to this list. Again, I look forward to your summary. -- I agree with your new person that people should NOT be putting local stuff in /usr. there should be one controlled central copy that everybody (up to some reasonable number of clients) mounts; otherwise, every system needs backups and every system is a unique mess.
On the other hand, /var SHOULD be local and should be large, because that is where you get non-tempfs temp space. Of course you could make a large root partition and have /var be on the same partition as root, but one benefit of having separate /var is that not much should then change in the root partition (no temp files, no mail spool files, no printer spool files, etc.)
More partitions essentially always reduces I/O performance, because of guaranteed long seeks to move between partitions. Nevertheless, it may be desirable for other reasons such as the one mentioned above (separating changing data from static data). -- FYI - This explaination revolves around Solaris 2.x but fundamentally holds true for ANY UNIX system.
UNIX's ability to mount drives/partitions anywhere in the directory tree is one of the strengths of its filesystem.
To build a response to your CTO, you need to understand the usage of the main directories:
/ - holds the kernel and it's configuration and obviously the main structure to the whole filesystem. The /etc is usually in the / partition. It contains configuration files for OS which are accessed whenever a sub-system starts up. For example, CRON, SENDMAIL, ALIASES, etc. The /sbin directory contains the necessary binaries to boot the system. The /proc directory contains vital data on all processes.
/var - holds log files and spool directories for sub-systems as well as a tmp directory for users. These sub-systems are always writing inside this directory tree (well, depending on the load of the machine).
/usr - Contains the binaries and libraries that make up the OS. It is obvious that this directory is always being access for data when processes are started. For example, a new SENDMAIL process is started whenever a message comes in to be handled. /usr/local has been a long tradition to use as a depot for user contributions to the system - for example - GNU software: gcc, gzip, etc
/opt - Good place to put additional software besides /usr/local. Some software forces you to put it there. It's a personal preference in many cases.
With all that in mind, it would be great to spread all those partitions, including the SWAP partition, to easy the load on the drive and provide better performance. If sendmail starts up and has to read from the /usr partition to load, read the configuration files from /etc and write out logs to /var, wouldn't it be nicer if it was accessing 3 different drives than the same disk and have to wait for the head to bounce around the platters? Sure most disks these days have about a 9ms access time but multiply 9ms by several hundred if not thousands of operations and you have a big delay.
Whether you do this or not depends on the load of the system. A single huge partition would be fine in "some" cases. But that can cause another problem. Any process could fill up the file system and potentially crash the system. That's the other reason for splitting up the file system across partitions and/or disks - because you don't want any sub-systems or users filling up the whole filesystem instead of just one partition. Don't forget, /usr/tmp is world writable! If the system has one hugh partition and the sendmail sub-system has a problem where it can receive e-mail but cannot process it (either forwarding it to another system or transfering the message to a user mail box which SHOUL D be on another disk besides the OS disk) then it will fill the /var/spool/mqueue directory. This could keep going until the whole file system is filled and the system will crash because it cannot write more data to the filesystem.
YOU DEFINITELY DON'T WANT TO FILL UP THE / PARTITION. If the system cannot write to /proc, then the kernel panics when it tries to create a new process and therefor write new data into the /proc. The /usr partition is not too dangerous to fill up but it does depend on what software you have living in it. The /var partition is very dangerous to fill up because sub-systems write to log files there.
When creating a system you really should sit down and deside what it's purpose is. If you are not sure then give each partition twice the space as required. If space is limited then give enough room for partiitions that will hold additional software (typically /usr and /opt) and a little extra to /var so it can hold a decent amount in log files.
Symbolic links are great... but it can make a system confusing. What I think your CTO is forgetting is that a disk can be mounted at ANY point into the directory tree... /usr/local can be a new disk. If the space allocated to /usr fills up with stuff in /usr/local, you can add a disk and mount it at /usr/local (of course transfering the data first).
Then again, if you don't have time to fix things the right way, symbolic links are the great way to get you going fast... but keep it simple and replace the symbolic link with a new partition when you get free time. -- I usually just make one large / partition. My reasoning behind this is that my understanding of the purpose of partitions is (1) to avoid a system crash because of a full partition ans (2) lower search times for files because there are less inodes to read in say a smaller /var. My feeling today is that disks are soooo BIG that it will be a long time and many upgrades before I could fill a 4Gb or even 9Gb disk that I don't have to worry about that. Also disk speed and system speed in general is so much faster that access time hasn't been a problem. This is, of course, just my opinion. :) -- With the advent of 2.6, I've switch my old model of many partitions to the following:
/ - very large /var (servers only)
(/usr/local is over the net, and we auto-home *everything* else.)
Works very well for us in a bout a 100-station network and many servers, including a 64-CPU E10K. -- Pardon me for sounding rude.... Is your CTO high on crack or something? > Sort of a philosophy here. When setting up a new Sun, I usually > allocate space for /, /var, /usr, and a bunch of other filesystems > called /a, /b, etc which I dole out as needed.
Good that is the way it should be done. Each file system should be able to hold 32767 is it files/direcotries total. performance will be poor once you start hitting that upper limit. (not that is not recursive it is in each layer 1 direcotry will count as one. That is as I understand it.)
> At my new job, our CTO is requesting that we follow his model of setting > up the filesystems, which is to allocate a small / partition,
BAD, I iusually give / at least 100 meg. Disks are cheap and you do not want to run root out of space. That is really bad. Even more so on HPUX 10.02 and loer. I actually had a HP system delte files and change the perms once we hit 95% utilization on the box for /!
> and tons of space in other partitions, but no /usr or /var.
OK no user where to files like /usr/sbin/* go ? OOPS. /usr/share/man???
I susually try to go 500meg + for /usr. /var is another beast as you said the log files tend to go there... OK 500meg+ is a minimal and make it it own mounted partition. There are also some other files that need to be in /var and /usr because UNIX likes them there. We also have the issue of /usr/local/bin where most/all of the GNU files like to go. SO you really need a /usr and a /var.
now for extra space like a users home direcotry make those on the remaining files. I goofed on a setup of one system and had to sym link /usr/local/bin to another drive, but that is not major it is a box I have at home... here is the layout. lesystem kbytes used avail capacity Mounted on /dev/dsk/c0t3d0s0 96534 16000 70881 19% / /dev/dsk/c0t3d0s6 1151920 193925 900399 18% /usr /proc 0 0 0 0% /proc fd 0 0 0 0% /dev/fd /dev/dsk/c0t3d0s3 121718 52376 57171 48% /var /dev/dsk/c0t2d0s7 189858 59155 111718 35% /devl /dev/dsk/c0t3d0s5 962870 33737 871361 4% /opt /dev/dsk/c0t1d0s7 388998 147286 202813 43% /stage /dev/dsk/c0t5d0s1 962425 530659 335524 62% /u1 /dev/dsk/c0t5d0s3 962425 102864 763319 12% /u2 /dev/dsk/c0t5d0s4 962425 121212 744971 14% /u3 /dev/dsk/c0t5d0s5 1052281 376656 570397 40% /u4 /dev/dsk/c0t3d0s7 962870 211631 693467 24% /u5 /dev/dsk/c0t3d0s1 481027 204400 228525 48% /usr/openwin swap 266844 12 266832 1% /tmp $
Another thing that is good to note is that you should not have /tmp and swap sharing the same space as I have above. As time comes and I upgrade the system those changes will be made.
> His reasoning is that by making /usr/local a link to say /a/usr.local, > he can vary the size of the filesystems as needed by rearranging the
it can work but it gets messy if you have to keep moving files around, and pay attention to the links. > link structure. He also asked why we couldn't just use a single huge > partition.
performance would really blow and you you run root out of space... That is a bad thing. Is the guy a NT/Windowz person. NT likes to OWN all the DISK space it sees. It likes to save GIVE ME MINE!!! Where Unix only takes owenership of the disks it is told to take ownership of.
> My reasoning was that I want /usr and /var since /usr will hold a lot of > local stuff, and /var will hold most of the log files. I think having > more partitions will improve i/o, but I can't prove it without testing.
That is what I am lookingat now so it will be good to see your summary. I am going through the Sun Performance and Tunig book Second Edition at this time to see what I can find. ( I am doing this for a enterprise 250 at the office 18gig 36 if we do not mirror.) -- My first question is why is somebody titled as "CTO" worried about disk structures on boxes?
Secondly, Sun has historically always supported multiple 'hard' partitions on a box, not links. Links get confusing and could affect backups when it came time to restore. /var can get fairly huge if your system is active with a lot of users, so you want it on a separate partitions so that if it fills up, your system won't crash. If Solaris can't write to "/" then it will crash I believe, so if /var is part of /, then if /var fills, so does /, and boom!
Your /a /b concept is a nice one, but we simply create /, /usr, /var and /opt originally. If the system is being used a lot and has the potential or opportunity (I like to word it that way) to expand later, we install Solstice DiskSuite up front on it and leave the remaining disks unformatted and unconfigured. This way we can pick and choose slices and partitions to make up metadevices later instead of having hard-coded directory sizes already limiting us. Either way it's the same concept I suppose as what you have.
I'm still really confused as to why a high-level executive CTO is telling you how to configure your partitions... -- A major issue I consider is resiliance in case of a system crash. I have never successfully been able to fix a disk with a bad superblock. So ...
Having /var on / partition means there are active log files on this partition. If a system crash occurs the larger the number of open active files the greater the chance that the filesystem will be damaged. Personally I'd rather recover a system from a working / directory than from the cdrom.
As for making /var and /usr symlinks to other partitions, sounds OK. Just remember they wont be mounted when you boot single user. Disk space is fairly cheap these days - so I tend to go with separate partitions for / /var /opt /usr and /usr/local. On machines where /var/tmp or /usr/tmp is heavly used by applications I make a tmp partition and symlink /var/tmp and /usr/tmp to it. -- > - is it advantageous to give /usr and /var their own filesystems?
Yes. The /var filesystem has files that grow, so you don't want /usr to be affected by a full filesystem.
> - can the same advantages be achieved by have a single huge > partition?
No. That just means it takes longer to fill /var, but it may still occur.
> - what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem? >
See above.
If you can even manage to put /var on a separate disk than /usr, you will get minor performance gains, since both of these filesystems are constantly being accessed. -- I *strongly* dislike the "big bucket" model as it makes the system vulnerable to problems.
I was an IT Manager at Oracle for 6 years and managed a few dozens of Unix machines (all flavors) and never, EVER implemented that approach.
Depending on system use and OS (assuming Sun) I would keep / under manageable values (typicaly 100MB).
I use this rules of thumb:
/ 100MB mirroed with DiskSuite /opt variable, depend on software loaded /var variable, but usually between lean 100MB to "fat" 2GB for Openview systems Tipically mirroed with DS /usr 1GB (handles all OS software plus the whole GNU and 3rd-party system tools) mirrored with DS
/local System depeendent apps /local/users /local/applications /local/oracle /local/...
I also avoid ANY user or application data in /usr. As I conceive my systems (and with success until now), /, /usr and /var are system dependent. The application and user data should be easily decoupled from the system and migrated to another. It also saves time when performing upgrades as there is no need to backup any system areas. Just do not touch application disks and filesystems and you're set.
I hope thsi helps you. I agree completly with you choice of battling this one. Some battle need to be fought if they will generate good admin practices. -- I like using seperate partitions because: 1. /var fragments alot so it will slow down anything else on that part. 2. / and /usr should be as small as possible since they are mainly static (at least in their size) and they will allow you to recover the machine quickly from tape if something bad happens. If you lose everyhing you just boot from CD, restore /, /usr, and /var, then boot up and begin restoring /home, /export, /opt, /usr/local, ... Also, /tmp should be either it's own partition or be tmpfs, you don't want / filling up when someone runs vi! -- Yes. /var hosts the log files. If this partition should fill up with log files gone haywire, then only that partition is a problem, and does not take down other items, and possibly the entire system. This is also where the spools sit. Same reasons there.
/usr has a lot of app software on it. Although it probably will not change too much except for adding more software, you have the advantage of mounting nosuid if security is a concern.
In both cases, if one of the partitions gets hosed, it may not take the whole system with it, and allow a somewhat easier restoration of it.
> - can the same advantages be achieved by have a single huge > partition?
See above. Answer should obviously be "no", as indicated in reasoning above. I do have a couple machines here that are set up as one huge partition.... I hope I never have to fix them.
> - what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
If there is only a / filesystem, the /var and /usr are already on that filesystem. Or am I not understanding the question?
My setups usually consist of /, /var, /usr and /users as a minimum. What I refer to as /users is often called /home on other systems. These user directories can get filled up quite fast.
/tmp is actually part of the swap space on my systems. I have set up systems where /tmp was a symling to /var/tmp so that all temp work is in one area (and /tmp is not in swap). -- The old school of thought, one which I still adhere to, was to use different partitions for /, /usr, /var, and so forth.
There are several reasons that I've heard cited for this thinking:
1. Sometimes processes run away on you and log a bunch of information while doing so. This logging typically goes to /var, and can result in a full filesystem if left unchecked (sleepy SysAdmin not monitoring system, long weekend, whatever). If /var is its own partition, this is not a big deal; kill the process and do some cleanup. If, however, /var is part of the / filesystem, and / fills up, machine performance can be severely impacted, to the point that the machine does not respond, will not boot, etc.
As disks have gotten bigger, this is not so much of an issue; if your /var partition fills up these days, you likely don't deserve to be called a SysAdmin anyway. ;-)
2. Ease of backups/restores. If one is using a backup utility such as ufsdump, one can backup different partitions on a more granular basis, and have an easier time of it when it comes to restoring a particular file or directory. Even restoration of an entire filesystem can be simpler with multiple partitions.
3. Ease of OS upgrades. When one chooses to upgrade an OS, it's usually best if all OS files are in their own partition(s), separate from other user data files. You might even want to have a separate /usr/local in this instance, just because /usr/local is for GNU stuff and the like, not OS files per se.
-- I split my server into multiple file systems using /var and /opt for one simple reason. Variable file sizes. If the log files in /var/log or /var/adm, or the mail files in /var/spool/mqueue or /var/spool/mailbox get huge and fill up the partition, it won't affect the other file systems or software that might be using space on those file systems. I use an /opt partition for our Netscape servers for the same reason...web access logs. If a web servers logs get huge and fill its partition, other file systems wont be affected and the system keeps running.
If you used one / partition for all, there is the possiblity for either logs, mail or web servers or the like, to fill the entire file system effectively hosing your system. -- Personally, I set up small user oriented systems with a / partition and a swap partition and nothing else. Additional disks are mounted off /export as /export/disk2, /export/disk3, etc.
On medium sized systems, I will definitely split /var off onto its own partition, since more users or higher server activity may mean larger log files and larger/more temporary files. If /var fills up, at least files written elsewhere on the system can be written to successfuly.
On larger sized systems, I will split /usr off as well. However, this will still depend on the machine and what it will be used for. A couple reasons for doing this might be for added security... such as nosuid/read-only fs options, etc. Since bigger systems like this have more disk space, or in general, more disks, this shouldn't be a big deal.
> - can the same advantages be achieved by have a single huge > partition?
/usr isn't as important to split off as /var is. The /var tree is the tree that runs the biggest risk of running your disk space out. If your system is primarily used for email and it has a lot of users on it, then have a /var partition and a /var/mail partition. The idea is to break off parts of your directory trees that can fill up your disk space and prevent other critical applications from being able to write to the disk (such as a DB type product).
On all single user oriented systems (like desktops or servers running only a single server application and nothing else), then I would agree to putting everything on a single partition.
> - what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
Don't know about this one. If you were really worried about performance and needed to split /var off, then why not put it on a second disk? > All scenarios assume that there are other partitions that will hold > user data such as /u, /home, and so forth.
The user directories are good candidate to go on their own partition. This prevents a user from filling up the disk space (if you don't use quotas) and breaking things. Of course, putting the user accounts on a seperate disk should dramatically increase performance if you have a lot of users who do a lot all the time :-) > This question is not merely academic - our CTO and I disagree on how the > filesystems for a new system should be laid out, and I want to make the > right choice. I am favoring the "classic" approach, reasoning that it > will give more flexibility and improve i/o. He feels that the symlink > method or a single huge partition are preferable.
I am interested in what type of responses you get about the I/O Performance vs Partition Count problem. However, I usually partition based on seperation and protection and not necessarily on performance. In general, anything that requires a lot of reading or writing (as in, a large percentage of all reads and writes for the drive that goes to a particular directory tree), should be partitioned off by themselves.
Anyways, good luck... and I look forward to your summary. -- Well, my boss and I were of the same mindset as you (everybody gets their own filesystem), but after reading a book on Sun performance tuning by Adrian Cockroft, we're moderating that somewhat. Aparently, Adiran suggests doing the multiple partition approach on server-class machines, and for things like individual user's workstations using one large partitioin. Primarily this is because that normally one user won't do enough to worry about flooding the /var directory, and when it comes time to upgrade you won't have to worry about partitions being too small.
Of course, this is also on a system where mail, home and application directories were NFS-mounted from a remote server, too....
> This question is not merely academic - our CTO and I disagree on how the > filesystems for a new system should be laid out, and I want to make the > right choice. I am favoring the "classic" approach, reasoning that it > will give more flexibility and improve i/o. He feels that the symlink > method or a single huge partition are preferable.
I'm willing to give the Cockroft method a whirl, because we've got quite a few workstations on the desks of various users, and I've also had problems trying to upgrade servers that had multiple partitions on it (/usr being too small, etc.), but since our users' home directories are mounted from a big disk array, we fit quite nicely into the setup that he describes, and this is an option for us.
YMMV, and all that. -- A huge file systems is more flexible if you don't plan to add disks later. I would avoid symlinks, if possible, because they do, slightly, slow things down.
That said, if you have a system that will take occasional patches, it would be nice if /var had space to use for that purpose. Likewise if lots of spooling will happen.
I don't see much benefit in segragating /usr from / unless you plan to have a lot of changes to /usr you wish isolated from / because you want / to be VERY stable.
If you plan to add lots of software or have user-writable areas, it's good to put them in separate partitions like /opt and /var, respectively.
By all means, minimize the risk of /'s becoming full. -- First, you should tell the CTO that if he has time to worry about filesystem layouts, he should be payed what a System Administrator is paid, not what a CTO is paid. Also, is he going to maintain this layout once it's done?!?
OK, with that out of the way : ) we can move on:
1) One big / is bad because if the filesystem with the kernel on it (you guessed it, / ) fills up, the system will stop and filesystem corruption is a distinct possibility.
2) A multiple symlink approach is bad because it requires at least 2 extra I/O's (1 to read the symlink, another to get the actual inode the link point to at a MINIMUM ). Systems are ordinarily I/O bound anyway - why make it worse?
The "classic" way is classic for a reason - it works, it's understood, and when this chump moves on, others will be able to understand it.
-- my two cents worth: on workstations in the past I found I was always running into space problems with separate partitions for /var /usr / etc.. becuase of patches and other considerations which causes quite a problem with partition resizing, especially if you have a lot of machines your dealing with. I now use only 3 partitions / ,swap (with tmp on it) and /opt. this gives me more flexibility and no noticeble perf drop. On servers I give /tmp its own partition as well, as I do not want to use up my swap space. -- My 'thoughts'
1.) keep the local stuff (ie., /usr/local, /opt/local) seperate from vendor stuffies..
2.) Make / a seperate partition...Systems don't like it when / is full!!!
3.) Make /var a seperate partition...Logs files, mail queues, plot queues do nothing but grow!!
4.) Make a /var/tmp (ln to /usr/tmp) a partion for users to do their work in..
-- I'm sure you know by now that setting up a computer is as individual as the= person setting it up. It is nearly impossible to give a standard answer= to your question. Without knowing how much disk, ram, etc. you have it's= hard to answer your question. I have set devices up from everything on= one disk in one partition to each of the major parts /, /usr, /var, /opt,= etc. on a different disk and different file systems. Even some with ufs= file systems and some with vxfs file systems. And then there are disk= packs and arrays.... I think this is why you received no answers. :)= Not that we don't want to answer.
-- I prefer the two partition method. Mainly / and swap. I really hate when /var is close to being full and /usr is sitting with tons of space open and doing nothing. Then you have to make a symbolic link anyways, or add more disk and try to figure out where in the /var space to mount it. Also, I did a SCO job once where I created the right partitions but labeled one wrong. I could have re-installed the whole OS to fix it up since it was a scratch install, but just did a sym link, and forgot about it. The only one who really knows its a symlink is me, and the apps runs, which is what really matters.
Let your boss have his way, then if things don't work, its not your head on the platter, and you may learn something new.
As you mentioned, the question is really academic. I thing I see on your question that as a consultant always bugs me. And that is where you say you want to make the right choice. Its easy, if it works, it can't be wrong. If it ain't wrong, it must be right, and therefore ok to do, otherwise it would be wrong, and would not work. Anything that is wrong, will not work, if it gets the job done, no harm, no foul. To many people are hung up on right and wrong, my way of thinking is works, doesn't work.
-- >- is it advantageous to give /usr and /var their own filesystems?
Depends. Outgoing mail is stored in /var/spool/mqueue. Separating /usr and /var will prevent the /usr filesystem from filling up with mail.
>- can the same advantages be achieved by have a single huge > partition?
yes but see above. It all depends on what your doing, etc.
>- what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
If one filesystem fills up, they all fill up. If separated, you can slap in another drive tar up the full space, mount the new drive over that directory, replace files, etc.
>This question is not merely academic - our CTO and I disagree on how the >filesystems for a new system should be laid out, and I want to make the >right choice. I am favoring the "classic" approach, reasoning that it >will give more flexibility and improve i/o. He feels that the symlink >method or a single huge partition are preferable.
It's a religious thing. I personally would never have everthing in one large partition.
FWIW, here's the output of a df on a couple of machines I control.
-- jf
(backup proxy server - solaris 2.6 x86) $ df -aek Filesystem kbytes used avail capacity Mounted on /dev/dsk/c2t0d0s0 966319 46377 918332 5% / /dev/dsk/c2t0d0s6 966319 371888 592821 39% /usr /proc 0 0 0 0% /proc fd 0 0 0 0% /dev/fd /dev/dsk/c2t0d0s7 960494 241898 716996 26% /home /dev/dsk/c2t0d0s5 966319 4520 960189 1% /opt /dev/dsk/c2t0d0s4 1986078 1775552 203906 90% /usr/local /dev/dsk/c2t0d0s1 2510214 4419 2493244 1% /usr/local/apache/var
swap 145492 192 145300 1% /tmp -hosts 0 0 0 0% /net -xfn 0 0 0 0% /xfn ns1:vold(pid251) 0 0 0 0% /vol
(main proxy server, web and mail. handles mail for 21 schools as well as being a web proxy server, hence a separate /var section.) Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t0d0s0 480620 41499 391059 10% / /dev/dsk/c0t0d0s6 673157 226959 385614 38% /usr /proc 0 0 0 0% /proc fd 0 0 0 0% /dev/fd /dev/dsk/c0t0d0s3 961257 277689 625893 31% /var /dev/dsk/c0t0d0s7 331605 153840 144605 52% /export/home /dev/dsk/c0t0d0s5 673157 213354 399219 35% /opt /dev/dsk/c0t0d0s1 576656 204470 314521 40% /usr/openwin /dev/dsk/c1t0d0s4 1091543 807504 229462 78% /usr/local /dev/dsk/c1t0d0s3 2807087 758844 1992102 28% /usr/local/apache/var
swap 251488 24 251464 1% /tmp -hosts 0 0 0 0% /net auto_home 0 0 0 0% /home -xfn 0 0 0 0% /xfn
-- With the advent of bigger hard drives (4Gb is the smallest on Ultra w/s, 9Gb on servers), it doesn't make as much sense to split filesystems as it used to. I still create seperate /var and /export/home partitions, to prevent logs & lusers from filling the root partition, but that's about it.
Not sure I'm clear on your CTO's advice to link to /u or whatever. Maybe a compromise position would be the best solution, politically and technically speaking.
Your CTO a promoted sys admin by any chance?
-- I had similar discussions with other members of our staff. The biggest downside to splitting up the root volume into many partitions for /var, /usr and /opt is that it makes upgrading and patching considerably harder. If you don't size your partitions properly the first time, you might find that you won't be able to upgrade or patch your system. In the past, system admins partitioned up the disk for system performance reasons. With the new zoned drives, this philosophy doesn't provide any appreciable speed. It makes as much sense to leave the root disk as one large partition. My contention, though, is that /var should be split out on its own partition. This will assist in determining where large files are eating up your disk space. Doing this will also help prevent system crashes due to full /. But again, upgrades and/or patches may be more difficult. What I have done when I have put /var onto its own partition, I leave enough space on / to contain the contents of the /var partition. Thereby, I can move /var back onto the root, perform the upgrade, then move it back to its own partition. -- Things in /usr are more or less read-only, so you could mount it that way if you wanted to.
Things in /var needs to be in a writeable area. This space has the greatest likelyhood that it will fill up.
One large partition has the following advantages: Easier to backup (one ufsdump will dump the entire disk) Freespace is distributed to the places that need it.
The disadvantages are: Easier to mess it up such that the system won't come up - root (/) must be in a state that has some freespace and inodes to come up.
Easier to inconvienience everyone by filling the disk.... With different partitions, the space is localized.
Not really sure if nowadays, one method is really superior over the other. I guess it would depend on WHO is using the machine (server based, or 'desktop' based). For a desktop, I might do the many partition approach to minimize what the user can mess up. For the server, given RAID and other stuff, it might be more advantageous to do 1 filesystem. -- /var /usr can not be symlinks to subdirs of another filesystem. If you do not know this, then I guess you try it or read some basic books on system administration. > All scenarios assume that there are other partitions that will hold > user data such as /u, /home, and so forth. > > This question is not merely academic - our CTO and I disagree on how the > filesystems for a new system should be laid out, and I want to make the > right choice. I am favoring the "classic" approach, reasoning that it > will give more flexibility and improve i/o. He feels that the symlink > method or a single huge partition are preferable.
Your CTO is right in making /usr/local a link and you are not right on this. You are right in making /usr /var seperate fs and you CTO was wrong. -- The first thing I would wonder about is why is a CTO interested in micromanaging partition layouts? In any case his approach is considered the best. Your approach was OK several years back when disks were smaller and slower. Then you tended to have seperate /usr and /opt on different disks. Thing are fast enough these days that larger filesystems are easier to manage. I do the following on desktops:
/ 1000MB swap 128MB /var 300MB /export Remainder of disk
Server: / 1000MB swap 256MB obviously swap size varies with applications /var 500MB /usr/local remainder of disk
On servers I use /usr/local only for things specific to that server, such as the binaries for a web server. Home, http docs, databases etc go on other disks, never the boot disk. I also almost never need much swap, I buy enough memory so swapping doesn't occur on a server.
Don't do symlinks!!! Especially for a path that is referenced a lot, such as /usr. A symlink is just one more thing for the kernel to resolve before any real work gets done.
-- Well, your CTO's scheme is ridiculous. If this is a workstation and just have a swap partition and a single partition for / (that's the lot). On a server, there are many advantages in individual partitions, not the least of them being archiving and restoring (e.g. with ufsdump). It takes some experience to get the sizes sensible in the first place. /usr is always getting bigger, for instance, over new OS releases. On our most carefully-crafted machines, we have /, /usr, /usr/local, /usr/local/src, /var, /opt, as well as swap. Of course, /home<whatever> and other funnies would be separate filesystems as well. Out of interest, how does your CTO manage to boot one of his machines single-user, with these links about...? -- Under 2.6+, we just give a seperate /var if the system is a server. /usr is part of / now.
I tread carefully around /var on servers because it can fill without warning.
>- what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
Symlinks? Why? /usr should remain *about* the same size, no matter what. (Assuming you use /usr/local for the new stuff.)
/var on a server should be fine at around 512M or so - If it has much mail usage, make a mounted /var/mail. We also have a sep. cores mounted (/var/crash points to a spot on the huge root partition) becuase a 64Gig E10K can dump some huge core files.
>All scenarios assume that there are other partitions that will hold >user data such as /u, /home, and so forth. > >This question is not merely academic - our CTO and I disagree on how the >filesystems for a new system should be laid out, and I want to make the >right choice. I am favoring the "classic" approach, reasoning that it >will give more flexibility and improve i/o. He feels that the symlink >method or a single huge partition are preferable.
Well, on our E10K domains (similar to all our major servers): # df -k -F ufs Filesystem kbytes used avail capacity Mounted on /dev/vx/dsk/rootvol 6154195 2375594 3717060 39% / /dev/vx/dsk/var 985507 473474 452903 52% /var
I used to follow the old method, but was finally convinced by 2.6 that it was fine to have big partitions.
-- Tradeoff with setting a huge root partition:
If you set up a huge partition with subdirs for the other file systems (/var , /usr, etc), the use of the disk space is balanced to all sharing this partition. So the disk is (in a way) more efficiently used.
However, this can cause the system to crash. The var partition holds the printer spool queues and they fill up very quickly. If it fills up too much, thhis would also exhaust the space on the root partition and the OS would give you a file system full while trying to update log files and the like.
Best, I feel, is to have the root by itself (whenever posible) and the other file systems on their own partitions. That way, if any of the other file systems fill up (/var, /usr, etc) you can always boot because root partition would not be affected. Once booted and logged on, you can troubleshoot the other problems with other file systems.
Also, with separate file partitions, if a partition becomes corrupted (hardware/software) it becomes isolated to the partition and the rest of the disk is recoverable. One huge partition, it goes bad, you lose the whole thing .... -- separate /usr is mostly academic. the idea is that the system will come up partway with / and not the other file systems. The more you stick on /, the "more likely" corruption of the file system will be damaged beyond the ability of the OS to repair it enough to boot.
Separate /var is more a case of how long you intend to go without intervening in the operation of the machine. /var is where log files, etc go, hence it is the directory most likely to fill up on its own. If /var is separate, filling will cause mail, etc to stop working but the machine will run. If /var is part of /, the system will die (sometimes very badly) when it overflows.
One big partition has the advantage of not wasting space by preallocating it where it ends up not being used.
I use one big root on general workstations, and a separate /var on servers, and a separate /usr if it's really important, in which case I also like to use a serial console tied to another machine.
/ alone is easiest to manage. If it does die, I don't mind waiting until a convenient time to fix it. /var means the machine can possibly be recovered remotely if it dies after hours for urgent. /usr means that if I can get console functions, such as via remote serial through another machine, I can get it to partially boot to maybe fix critical problems.
>- can the same advantages be achieved by have a single huge > partition?
possibly. if the critical server can be booted off the net or something, the serial console gives about as much power as you get to recover a dead machine. For most machines, I'd rather have the convenience the rest of the time since most machines can wait for next day response.
>- what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
If you use symlinks, make sure the mount point for /var has the tmp dir so /var/tmp will always exist, or death of that filesystem makes things much harder to fix. (vi complains, among other things)
Other than that, there really isn't much difference.
separate partitions are a lot of work for benefit that rarely matters. It's a real question, but very few sites will see much effect, imho. -- I've had problems doing this before, at least under SunOS (4.1.4). If /usr is a symlink, it won't be listed (directly) in /etc/fstab (or vfstab), and so the kernel won't know to mount it before it tries to start services which are homed in /usr. --
Philosophies and religion are not my favorite subjects, however, for Solaris 2.x/7 systems, I find it very advantageous to use one larger filesystem for the root FS whereon I place just the operating system. This includes usr, var directories. I would never advise anyone to place local software/files into this filesystem (/), but would instead place these into a separate fs, e.g. /export with the intent that this is to be shared with other hosts. This will simplify upgrading the machine later. Unless your host is a mail server or print server where /var needs to be larger, or there is some other unusually large use of /var, I would include it in the root fs.
There is certainly a lot of flexibility to be gained by sizing one large root filesystem vs. carving up the disk into artificial slices. Again, I prefer not to have to deal with many slices which are unnecessary. It is easier to restore one fs rather than three. You are less likely to have to resize one large fs than three/four fs whose functions may change over time. (My experience).
Consider also your backup strategy. The O/S fs contains few files which change frequently, by and large being fairly static. User data (mail, home) change frequently. You may not wish to perform full backups on filesystems that are mostly static at the same frequency as volatile filesystems.
Lastly, get a copy of Adrian Cockrofts Performance and Tuning Guide from SunSoft. He talks about this important issue. -- Hi , I will give some information on fileystems Pl go thro'
Multiple Filesystems: PRO: 1. Rogue user or process may fill up a filesystem to 100%. This is not so bad if in just one filesystem, but Bad Thing in /. 2. Maintenance of a filesystem is easier. 3. Can umount a single filesystem and manipulate it without taking down entire host. 4. Tailor a filesystem to its expected use. For example, size and # inodes can be fine-tuned. 5. Back-up or restore can be faster. 6. Mix and match filesystems 7. In the stone age, had to have more than one filesystem because disks were small and disk-spanning features (striping, LVM) were not available. 8. Can make a filesystem bigger with not much trouble, assuming that there is unallocated space on the disk(s). 9. Although can't just "lvreduce" a LV without losing data, this can be done if you need to "rob Peter to pay Paul" -- that is make one LV smaller and then assign the space to another LV. 10. Multiple filesystems allow tailoring security/file access. 11. Can share a filesystem with other systems. 12. Reduced fragmentation: static filesystems remain unfragmented, for active filesystems, takes less time to defragment them when needed.
CON: 1. (Mine, from prior NFS questions.) If you want to mount all the disks, you have to mount each filesystem separately. You can't just export /. 2. Can't directly lvreduce the size of a LV without trashing all the data thereon. You have to backup, lvreduce, then restore data. 3. [Mine, admittedly kind of lame]. With multiple filesystems, system is more complex. Adding LVM makes it even more complex, albeit flexible and efficient. 4. If LVM is not available (e.g. my stone age Unix mini), then resizing filesystems is a pain, requiring at minimum archiving 2 filesystems, and then using mkfs or similar to reduce one and enlarge the other, then restore all the data.
Single Filesystem: PRO: 1.On single-user Linux workstations (no LVM) I typically have a big / because resizing partitions is a big deal. I might split off certain partitions, but nowhere near as much as I might on a machine with LVM. On multi-user machines / should have no user-writable directories; 2. If you're careless and install important recovery tools like gtar into /, but stupidly let them require shared libraries in /usr (say), you will have difficulties restoring /usr from backup. *grin*
CON: 1. fsck, du, other commands take longer 2. Maximum size = 128 GB ( in HPUX), or some other system-dependent limitation. 3. Not good idea on multi-user systems, for reverse of many of the reasons in "Multiple-PRO" cited earlier.
Not sure where to put these: 1. [General rule] If a disk fails physically, all the data on that disk is lost, whether one or more filesystems. 2. Best to keep a single filesystem on a single disk? 3. Disk striping and/or a lvol that spans multiple disks: Pros: performance gains, lvol may be bigger than the size of a single physical drive. CONS: somewhat greater exposure to hardware failure. [My assumption] If your filesystem is spread over 3 drives, and one drive fails, you have lost the entire filesystem. Mirroring/RAID/whatever answers this problem (?). -- I would stick with the tied and true standards of /, /usr, /var, /export/home
Var grows.. so if you put it under your root filesystem you are asking for a crash down the road. Or a malicious user can crash it by filling your system with syslog messages. /export/home instead of /home allows you to use the automounter standard of /home for home directories. /usr does not really grow unless you are in a dynamic environment that puts a lot in /usr/local. Splitting up the filesystems over multiple disks and using seperate disks for each filesystem will improve performance. I use the following for a standard desktop config.
/ 100 meg /usr 800 meg /usr/local 600 meg /var 200 meg
You can use disksuite to increase filesystem sizes down the road if need by growing a filesystem. (except you cant grow root i think). -- It is recommendable to have multiple partitions on your SUN harddisk if your assigned may grow after some time. so that it will not fill your / partition easily ....
Another advantage is that if anothign happen to your / partition, it may not affect the others.
Normally Unix /Linux sytems for PCs will ask during installtion have only one partition, but then, SUN server will give you a complete menu of 7 partitions to be separated for ...
Single partition will enable easy mangement of the hard disk.
I do hope these facts will help you in explaining to your superior. -- I basically use single / partition for workstations. The reason is historycal, namely that when, about 6 years ago, I started with those SPARCClassics, 250Mb HD, the swap and Sol2.3 enduser cluster filled that up nicely, I didn't want to fragment that little space left over. I have not had problems with this setup and I added only recently an /export, users can put their personal belongings there.
For servers I would consider setting up a /var partition, thats the place where logs fill up, mailbombs arrive and print queues flood, although if you only have the OS on / it does not make a big difference if you fill up that or /var.
In my case (Software R&D center) I can not see any situation where a separate /usr,/opt would have advantages. -- I'm always wary about 1 large partition for root, as I don't like the thought of logs, new software etc filling up my root partition. I think this can easily be prevented by having /var etc on their own partitions.
And as for performance, I consider that these system partitions should be on different disks and possibly controllers anyway, we all know that disk subsystems are the slow link in todays machines.
I think you should go with your past experience. -- Before Solaris7 I always prefered the "classic" approach, because when a failure at one of the filesystems happen, it's very likely to recover without the need to boot of CD or the network. Since Solaris7 there is a logging facility in the filesystem which logs all fs activitys and allows automatic (and fast) filesystem recoveries in case of a failure. I/O is in my eyes not improved with the "classic" approach until you use different harddisks for the filesystems. -- I myself favour multiple partitions. It doesn't make any difference to performance when you are using the same physical device for all the = partitions. The advantages as far I'm concerned are that you can prevent /var taking = over your whole system when something goes wrong and also, /usr and / can be sized pretty accurately so that the remainder of the disk can be used = for something else. It makes backing up and restoring easier and faster and if you know what you are doing, you can get the system up with only /, = /var and /usr.
There isn't really a convincing technical argument either way, but I've = found=20 it useful to be able to shift parts of the operating system among disks without having to boot from a CDROM. -- On servers and machines where filling / would be more than just inconvenient, I add a separate /var to hold spool areas and log files.
The distinction between / and /usr isn't as neccesary as it used to be, as the newer Solaris versions have a rather platform independent / as well. Earlier, /usr was quite a lot more platform independent than /.
I would reserve / and /usr for the 'platform software'. Operating system, veritas, disksuite, etc. No other software should be allowed to go into / and /usr. But this is just my personal feeling. -- I like to have distinct file systems with static and variable contents. Nothing in /usr should ever change (only due to patch installations), if you have /usr/local on its own partition. I think you could even mount /usr read-only. Different backup frequencies for static system files versus frequently changing files can be scheduled. I also have extra partitions for /tmp and /var because this way the system operations are unaffected if /var gets filled up by someone spooling a big print file, for example. You can mount /tmp as tmpfs (memory based filesystem) which speeds up operations like compiling, which create many temporary files, a lot.
The only disadvantage I see is that you have to plan the partition sizes during installation. (And it seems that Sun is moving away from its former suggestion to have distinct / and /usr partitions.) -- /usr/local is often used for applications, hence it's best to at least put /usr/local on a separate partition.
}- can the same advantages be achieved by have a single huge partition?
This is what I tend to do, because you don't run into annoying problems where the system runs out of space in one partition, but there's stacks in another, which requires a re-format and at least a couple of hours work. Bear in mind I've mostly worked with small root disks...
There are performance issues anti- using a large partition, but TBH they're pretty insignificant compared to amount of RAM (and hence file sys cache) and other issues.
}- what are the tradeoffs between a filesystem with say /,/var,/usr }each having their own partitions, as compared to another machine }with only a / partition, and /var and /usr being symlinks to subdirs }of another filesystem?
Well, I've kind of addressed that above.
Separate partitions: marginally quicker; when one FS fills up, you don't have the root partition filling up; enlarging the size of a partition requires less work.
All in one: more efficient use of disk space; less maintenance.
/var and /usr on another disk requires careful setting up, cos that disk may not be mounted when the kernel starts trying to read/write /var/log/etc... at boot time. -- I believe this question has been posted to this list before. Which is why I, and probably others chose not to answer. You can look it up in the archives. -- I am by no means experienced enough to even have an opinion ( 1.5 yrs admin exp ). But I would certainly put /var on its own partition. If not, and the partition fills up, is it log files or a users runaway process that killed you? The job of researching problems gets muddled.
Your CTO has a good point about being able to expand the file size infinitely, and that was important when disk space wasn't so cheap. Today, you should have plenty of disk space for /var and /usr without crowding anything else. Also today, disk space is _way_ cheaper than down time while you try to figure out a problem. -- Servers: separate /, /usr, /var, /var/cfs and /export for virtually everything else. This is the classic situation for admin and I try to spread the IO load over multiple spindles where possible. We don't allow interactive use on our file servers - just NFS and mail delivery - so there's very little disk IO in normal use except to user files and swap. Desktop workstations: I only have /, swap and /var/cfs - I use jumpstart to install these machines so upgradeability is not an issue. We don't store any user files on these machines either. Other user services machines:
It depends, but one of the 2 previous models usually fits. -- It's basic logics. When you format and partition your drive using format, you allocate the first part of the disk (s0) to / normally. THis is a fairly small FS. Meaning that all the needed files to start are in s0. The second slice is swap (s1). Be aware that the allocation start from the outer bound of the hard disk to the inner bound, why ??? Because the outer bound spins faster !!! THat's a good reason to have / and swap in first on the outer bound of the disk!
Then, you should create a s3 or /var partition for local logs files. Again, this partition is used a lot for writting... The next two are normally /opt in s5 and /usr in s6, these two are mostly read-only fs, so that why they come after /, swap and /var... And finally, a s7 with /export/home. You have to understand that the bigger a FS is, the bigger change you have to seek a lot on that fs, even if FFS (fast file system, the ufs under solaris) try to allocate using cylinder group, you'll never be garanteed of this.
Another reason to partition is to make sure that logs file in /var won't affect /, so / won't be full because some application or daemons is logging to much in /var. This prevents some DoS attahcs... I could go on and on, but I think you get the picture. One last thing, partitioning give you flexibility in terms of backup strategy... Which FS can be backup when ? If only one, you have less possibilities...
Bottom line, faster FS with partitions, protection against DoS (file system full). More administration then only one FS, sometime, must re-partition... -- >- is it advantageous to give /usr and /var their own filesystems?
Yes: - /usr is is a static filesystem with little changes over time, but /var (as the name shows) is "variable" and changes quickly becasue of logs, mail and so. Note that /usr/local is a different story. See below. - Because of this difference, you usually design different backup strategies for /var and /usr. /var should be backed up much quicker than /usr. - Having two different filesystems means that you can have better control on your backup sizes. - On heavily-loaded systems, putting these two on even different disks improves performance a lot.
>- can the same advantages be achieved by have a single huge > partition?
No, since as you see, /var and /usr have different characteristics that need different layout approaches.
>- what are the tradeoffs between a filesystem with say /,/var,/usr > each having their own partitions, as compared to another machine > with only a / partition, and /var and /usr being symlinks to subdirs > of another filesystem?
The second model means that you either should backup the whole big filesystem to have backups of /var and /usr (which is not fine as mentioned above), or if you want to back them up separately, you cannot have incremental backups (see ufsdump man page). >> His reasoning is that by making /usr/local a link to say /a/usr.local, >> he can vary the size of the filesystems as needed by rearranging the >> link structure. He also asked why we couldn't just use a single huge >> partition.
He's right. Personally I prefer not to put the "local" stuff in /usr for the same reason I mentioned above: /usr should very rarely change. Since /usr/local is a place that you install new software from time to time, it's a good idea to keep it on another filesystem.
The reason that I emphesize on keeping /usr untouched is that in crash situations (or even upgrade cases) it's always a nightmare to restore /usr back to its before-crash state, and if you miss a piece, you'll have problems running user programs which is what we should avoid. The less you change /usr from the original installation the sooner you can recover. So make is a separate filesystem just with a size needed for the original installation and a small overhead, and put it aside... >> My reasoning was that I want /usr and /var since /usr will hold a lot of >> local stuff, and /var will hold most of the log files. I think having >> more partitions will improve i/o, but I can't prove it without testing. Having more partitions does not necessarily improve I/O. In fact, you gain I/O performance when you put different partitions on different disks. You may gain some benefits (mostly on big partitions) if you know about the nature of the data you're storing and tuning filesystem block size or NBPI (Number of Bytes Per Inode) for different kinds of data, e.g. if you know you're storing sound or video files that are naturally big, it's a good idea to put them on a separate filesystem with a large NBPI even 32k or 64k rather than the default of 2k.
All in all, here's the tradeoff: Having one huge partition that hosts a lot of directories gives you more flexibility in managing the space and not looking around for free places in different filesystems and making a lot of "referencing" symbolic links.
On the other hand, having separate partitions and filesystems gives you more flexibility on distributing the load across disks, finer granuality and better planning in backup, and more freedom in tuning performance based on your local applications.
Hope this helps. -- This is a classic case of "tradeoffs" without any answer being "correct".
If you create fewer filesystems, they can all "share" all free space that is available. This can come in handy at times. The downside to this approach is if/when free disk space goes to zero, the whole system or major portions will stop working.
By creating more filesystems, you can compartmentalize the impact of a single filesystem filling up; typically /var when some rogue process creates HUGE logfiles.
Personally I tend to use 4 filesystems; /, /usr, /var, & /home I would definitely prefer keeping "user" directories on one or more filesystem(s) that are separate from any "system" filesystems.
-- Depends on your requirements:
1) I am currently building a high availability web server (2 systems with failover). I have a single disk (8.4GB) capable of holding EVERYTHING. Made 1 filesystem (/), 1 swap partition and 1 slice for a meta database. This was the simplest.
2) For OLTP systems with a high degree of logging, database parts, etc I split it all over the place. Usually try to have a pair of mirrored disks for the operating system and binaries. Split the data to as many spindles as possible. What you DO NOT want is for the logging to fill up /tmp or /var/tmp and this shuts down the system. I do the same for /home since users have a tendency to do some VERY strange things.
There would be little difference in symbolic links or separate file systems for what the CTO wanted to do. Any directory can be made it's own mount point. There are some applications that HAVE to reside on the named file system where /usr/local and /a/usr/local would not work.
ON THE OTHER HAND - if the CTO is responsible for authorizing your pay and annual reviews it may be worthwhile to live within his system. Present the arguments for each method, but make sure to pick your battles appropriately. Allow this to go by and fight hard for the memory or CPU upgrade down the road. Or even better - a trip to LISA, SANS or something similar.
*** End Summary *** -- Dan Simoes mail:dans@iclick.com iClick web:www.iclick.com 410 Saw Mill River Road LL 135 voice: 914.693.0837 Ardsley, NY 10502 fax:914.693.1055
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:20 CDT