first, many apologies for the late summary; between getting this
machine back up, a spamming incident here requiring me to upgrade seven
different mercury mail servers, our university's requirement that
everyone take their vacation before july 1st or lose it, and a bad case
of bronchitis, i've been a bit busy.
second, thanks to the following for their prompt responses:
Birger Wathne <birger@sdata.no>,
Brian Desmond <brian@chimera.psych.unimelb.edu.au>,
Frank Cusack <fcusack@iconnet.net>,
Tim Carlson <tim@santafe.edu>,
Saurabh Jang <jang@eis.comm.mot.com>,
David Dhunjishaw <dave@colltech.com>,
Karl Vogel <vogelke@c17mis.region2.wpafb.af.mil>,
Paul Vinson <vinson@mulehenge.net>,
wstuart <wade.stuart@intranetsol.com>,
Jeff Wasilko <jeffw@smoe.org>,
Rich Kulawiec <rsk@gsp.org>,
Chris Liljenstolpe <cds@mcmurdo.gov>,
Al Hopper <al@logical-approach.com>,
John Stoffel <jfs@fluent.com>, and
Matthew Potter <Matthew.Potter@globalone.net>.
third, the summary. my question (quoted in its overlong entirety at
the end of this message) was basically "what might be causing netscape
collabra 3.5 to coredump regularly on an ultra enterprise 3000 running
solaris 2.6, and how well suited is the ue3k to the task of serving
news, given that you can't put multiple controllers in for the 13 or so
disks it has internally?"
(to clarify a point from my original question, i didn't think that
the hardware issues that that machine was experiencing were causing the
coredumps; rather, those issues (such as a bad backplane and some bad
disks) caused the machine to be crashed rather irretreivably (and my
backups were bad, too, of course...), and i decided that since i would
have to start over from scratch anyway, i would do it the right way.)
the general consensus was that collabra wasn't the best choice for a
news server, and that there were questions as to whether or not version
3.5 really was compatible with solaris 2.6 (despite netscape's claims
otherwise); recommended instead was INN, or one of the news servers from
highwind software. others pointed out that my disk-full errors were
probably caused by running out of inodes; i know that i had planned to
increase the number of inodes when first setting this machine up, but it
is very possible that i forgot to.
almost everyone agreed that a ue3k was overkill for news, but that
with only one controller, there would be serious issues; with that sort
of configuration, it would be better suited to data-warehousing-style
apps. some suggested switching to external arrays to get around the
no-multiple-controllers issue, and many suggested adding multiple
controllers for the internal drives, despite my saying that i couldn't.
only a few were able to inform me that the ue3k _can_ support multiple
controllers for its internal disks, if given multiple i/o boards; this
contradicts what our sun sales rep said (he indicated that the internal
disks all had to use the same i/o board), but i'd take the word of
people who have installed those multiple controllers successfully over
that of a sales rep any day. (also, one person who apparently knows
something about GW suggested that i replace the ue3k with the
severely-overpowered (multiple gigs of ram, nice backplane, lots of disk
arrays) main server run by another department here; unfortunately, we
can't touch that machine (and wouldn't waste it on news _or_ mail (which
is basically all it does now).)
in the end, after doing some more research, i decided to stay with
the ue3k, and go with highwind software's breeze news server; among the
reasons were cost (around $400, plus an as-yet-unspecified educational
discount), support (we've been running a fully functional "demo"
version for the last couple of weeks, and highwind support was
absolutely phenomenal in getting it up, even though we still haven't
even discussed payment), and smart design: the highwind line of servers
gets around many of the issues that make news very i/o-intensive, thus
eliminating the need for multiple controllers and a raid array (though
they still get some benefit from having the disks raid-ed, so i went
ahead and set up that array anyways). rather than write each article as
a file, they act like most of the databases i've worked with, and create
large "spool object" files (up to 2gig per object for breeze, or
unlimited for the higher-end servers they have), into which the articles
are written as straight data. this gets around the need for many
inodes, since there are very few physical files, and optimizes space
usage, since there are no half-full blocks wasted, as there are with
most news articles in a standard setup. also, the software caches all
writes in memory, and only performs those writes in integer multiples of
the block-size on that machine, thus reducing the number of writes
necessary, and decreasing the load on the controller. finally, the
spool objects are written to in a FIFO style--if a spool is full, the
oldest article in a given spool is simply overwritten once a new article
comes in. there is thus no need for an expiration process, which is
normally one of the most disk-intensive features of news (and the one
when most of our crashes would occur with our old setup), and disk space
is managed much more efficiently: there is no need to set an arbitrarily
low expire time, which wastes disk space, or a fairly generous one,
which leaves open the possibility that a sudden burst of incoming news
will fill the disks up and cause all sorts of problems; on a 25 gig
/var/news partition with 24 gig allocated to spool objects, we've had
almost exactly 24 gig of news stored since the first day the server went
up, and no problems to speak of with the server.
upon further consideration, this looks a bit like evangelizing for
highwind; all i can say is that it is, because their software really is
amazing.
thanks again to everyone who responded,
sweth.
--- Original Question ---
here at gwu, we are serving usenet news with an ultra enterprise 3000
with 256 mb ram, running solaris 2.6 (patched through may) and netscape
collabra 3.5; there are root, /usr, and /opt partitions on one drive, and the
/var/news partition is on a striped metadevice across the other 9 drives. we
have had _lots_ of problems with the hardware on this machine (as of 3 am on
wednesday night, when a sun engineer and myself finished swapping in a new
backplane, the only original hardware left was the case and the power supply),
but we have also had lots of problems with the software and disk config sides;
if anyone has any advice, i'd really appreciate it.
one of our problems is that collabra will dump core on a fairly
regular basis. generally, running the expire process will fix the problem
(though often, the expire claims to fail). for a while, though, the expire
was running out of room on /tmp, so i had to move it to a new directory
inside /var/news; suddenly, though, that stopped working, but running the
expire on /tmp began working instead to get the server back up again (though
every time, the expire would report failure). many times, the coredumps
would be preceeded by errors saying that there was "no space left on device",
even though the metadevice that /var/news was on was only 80% full. (i
presume, since these errors were being reported by the process that logs
incoming newsfeeds, that it was referring to /var/news; even if not, though,
none of the other partitions were more than 40% full.) for a while (about a
week), the coredumps would happen at almost the same time each day (within
one minute), in the middle of a write by one of our newsfeeds. one of my
coworkers thought that that meant that that particular feed was too heavy
and crashed the system, but our upstream provider simply send articles to
us as they receive them, so that there are always feeds open; even if someone
upstream from them was regularly sending, say, a 40G dump all at once, just
because of network congestion i can't imagine that it was the exact same feed
hitting at the exact same time--instead, my assumption was that it was that
_any_ feed was hitting at that particular time, and something else going on at
that time clashed with processing a feed, so everything died. nothing was
running in cron or via at, however, and that particular incarnation of
coredumps-r-us disappeared after a week, so i'm not sure. on top of all of
that, the system also seems very slow, though that was a purely subjective
measurement.
also for reference, i don't think that the ue3000 is well suited to
serving news. it has 10 4.8 gig drives on one scsi controller, with 2
167-mhz processors, which to me means that it should be doing something
involving a fair amount of processing, involving lots of data that gets many
reads and not too many writes; news uses almost no processor, and (simply
because we have a large feed and not many users) does lots of writes and very
few reads. i've been told that putting the drives on multiple controllers
would help some of the problems we have, because news has a lot of i/o
contention if the articles are on the same controller as the history and
active files; we can't put the drives on multiple controllers, however,
because they are all internal and sun didn't put internal scsi controller
upgradeability on their list of things that the ue3k needed, it seems.
so, the questions:
1) should an ue3000 have these sorts of problems with news? many of
my coworkers are amazed that it has the problems it has, because they have
seen news running on slow, cheap linux installs; the only explanation i can
think of is that if the ue3k were to be configured like the cheap boxes (one
or two drives on one controller, rather than ten) it would work fine, albeit
with much less capacity. how far off base am i here?
2) would anyone configure the disks differently? after the last
hardware crash, i had to reinstall everything, so i have recreated device 0
with a smaller root, /opt, and /usr partition, and given it 512 megs of swap
instead of the 128 that it had before (since some of the problems appear to be
from lack of space in /tmp). does that make sense? what about the /var/news
partition: is raid-0 across all of the other disks a good idea, or would the
fact that there is only one controller cause problems? sun's support staff
(whom i called, since i am using online disksuite) say they are not allowed
to make recommendations as to performance of configurations; i was thinking
about scaling back our feed and increasing our expires considerably, and
putting the entire /var/news partition on one of the 4.8 gig disks at first,
and then gradually adding more disks to that partition, to see if there was,
in fact, a lot of contention caused by the raid.
3) even if the ue3k can be made to work with news, i personally don't
think it is well suited to the task. if i get another machine (the ue3k was
purchased long before i got here) to do the job, what should i be looking for?
i was thinking, based on discussions i had seen on usenet a while ago, of a
small-processored box with three or four controllers so that the history files,
active files, and articles each could be modified independantly. also, if we
do replace the ue3k, what _would_ it be good for? the only thing i can think
of that it seems well-designed to do is data warehousing.
tia,
sweth.
-- Sweth Chandramouli IS Coordinator, The George Washington University <sweth@gwu.edu> / (202) 994 - 8521 (V) / (202) 994 - 0458 (F) <a href="http://astaroth.nit.gwu.edu/~sweth/disc.html">*</a>
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:12:43 CDT