SUMMARY: network expansion

From: Upkar Singh Kohli (upkar@wsu-eng.eng.wayne.edu)
Date: Thu Jul 23 1992 - 10:11:42 CDT


I received a number of very good replies on the following
problem even though a couple of responsible people rightfully
mentioned that this was not a sun-specific question. Since
most of the machines I deal with are suns, I failed to note
this earlier. However, since I received requests for a
summary from many people, I have compiled this for the net.
I will take the blame just this one time.

original question:
------------------
What would be the best hardware option for expanding our network...
We have been using mostly "thin" ethernet (with a few machines on
the "thick"). Now as more and more machines are being incorporated
in the network, the network length is exceeding the limits.
 
network features:

- the whole network is using one segment only with no repeaters or
  bridges
- the thin and thick are "directly" connected
- machines on the network include around 45-50 nodes (sun and iris
  workstations, indigos, cray YMPel, celerity, xterms, PCs, DEC,
  HP, sun4/280, etc.
- all user directories are accessible (via 'amd') over the network (nfs)
- much of the work involves pre/post processing (animations), so
  good interactive performance is crucial

summary of replies:
-------------------
From: (thanks to all, hope I did not miss anybody)

david@elroy.Jpl.Nasa.Gov
fetrow@biostat.washington.edu
issi!issi.com!vasey@cs.utexas.edu
birger@vest.sdata.no
eckhard@ikarus.ts.go.dlr.de
Perry_Hutchison.Portland@xerox.com
jjb@pandora.cs.wayne.edu
nancy@aspiring.unh.edu
john@solar.nova.edu
edguer@alpha.CES.CWRU.Edu
baumann@proton.llumc.edu
etnibsd!vsh@uunet.UU.NET
pml@cacs.usl.edu
drd!drd.com!ron@tulsun.Central.Sun.COM
miker@sbcoc.com

The cheapest thing is to deal with the immediate problem,
cable length. The limiting factor is your thin ethernet,
simplest solution is to replace the "direct" connection
between thin and thick with a repeater. That will increase
the length on your thick. Down side is that it doesn't
reduce traffic. Next step up is to place a bridge instead of
a repeater which may cut down traffic on each segment depending
on your traffic patterns.

Next step would be to create multiple networks connected by
a router. The cheapest router would be to buy an SBus ethernet
card for under $1000. It will decrease your performence, but
in actual experience an SS-1 is so fast at packet routing
that any decrease in performce will not be measureable, especially
if it has lots of memory. I don'r recommend turning any VME based
machine into a router, the VME ethernet boards are all too slow
and inefficient and their cycles are more expensive.

Next step up is a dedicated router box such as a Cisco, Proteon,
Wellfleat or one of a half dozen others. This gets more expensive.

For all of the options except the dirt cheap add a repeater you
should seriously look at your network and where the traffic
is going to and from. Simply splitting it in arbitrary places
may hurt you more than help you. Common topologies are
to place all the clients of a server on their own network
and all the servers on a network so that local NFS traffic is
kept local. Other issues are if you have users trying to
feed data to a Cray as fast as possible you don't want a router
between them. If you go with SBus ethernet cards, which can be
an added plus if you get one with an extra SCSI interface, they
are cheap enough so that you could have multiple networks.

----------------------------

 Well, if I had the option and money I'd redo it in twisted pair

 We are running multiple thinnets out of a DELNI (A DEC multiport thinnet
repeater) and this works very well. We can have up to 8 physical thinwire
cables out of a single box which allows us to shorten the individual lengths
considerably. Electrically, it is just a repeater.

 If our traffic were partioned on these lengths and was nearing saturation;
then a bridge or router would be nice....but in our case neither of those
situations has happened yet.

 A lot of companies make these things, but at the time thinnet was hot new
technology and the DEC version was about it. It was expensive but, on the
other hand, it's still there.

----------------------------

The best performance enhancer you can buy -- and have the least administrative
headaches to boot -- is a Kalpana EtherSwitch. An outstanding improvement on
bridging technology! Reduces traffic on each segment, buffers traffic between
segments (improving throughput), allows star configurations, NO administration.

I think they still cost ~$4k + $1k per port (segment) -- a good investment for
your size of network and type of use. Media (thick, thin, twisted) not relevant.
... call to 800+555-1212 ...

----------------------------

If you run only TCP/IP, you could look at a network coprocessor card
for a Sun. If you have multiple protocols (XNS/DECnet/localtalk/whatever)
i think you should look at a Cisco router. By far the most versatile
router you can find.

----------------------------

The ceapest solution would be a second ethernet card for a PC, using
PCroute on it, but I have no experiences with that, and I'm sceptical
about performance.

Next cheapest would be a second ethernet card for a workstation, which
acts as a gateway. This should not affect network performance too much,
but workstation performance. Moreover, this would make sense only if
the network is already divided logically, e.g. if you have two (or more)
workgroups with different resource usages, and if you can track down
some mayor network load producing connections.

A repeater and a bridge are quite the same price. The repeater is faster
than the bridge, but it repeats collisions on every segment, so if you
have heavy load on one segment, the other will slow down, too, and I
expect this to be the case for you quite often with so many machines in
one subnet.

The best thing I would think of is to have a main fileserver with two
or more ethernet cards, so you could keep at least NFS traffic on the
appripriate network.

----------------------------
       
If length and/or node count (loading) are the only problems, a repeater
is the answer. It should not affect performance very much. If traffic
and/or congestion is also a problem (or seems likely to become one in
the near future) you would do well to consider subnetting and a
router. This will presumably slow down access between machines on
different subnets, however the reduced congestion may more than make up
for that.

A repeater or bridge connects multiple segments into a single subnet --
all such segments have the same network number. A repeater replicates
all traffic and collisions originating on any segment to the other
segment(s). A bridge is somewhat smarter: it replicates traffic only
if the sender and receiver are not on the same segment.

A router connects multiple subnets, each of which has its own network
number. (The difference between different "subnets" and different
"networks" has mostly to do with what you advertise to the outside
world.)

The distinction between a bridge and a router can be somewhat fuzzy --
some devices are configurable to perform either or both functions.
Roughly speaking, bridging is done solely on the basis of ethernet
addresses and knows nothing about higher level protocols (like IP),
while routing involves knowledge of the higher levels. Some
higher-level protocols can be bridged but cannot be routed.

A server (or even a PC) with two ethernet cards can certainly serve as
a router, however the best performance is likely to be obtained with a
machine specifically designed and optimized for that purpose. Packet
size should not be a factor as long as the machine being used for
routing supports a maximum packet size at least as large as that
supported by any other machine on the network, but the need to examine
the packet contents will typically impose some performance penalty as
compared to a repeater (which just copies bits). The penalty is likely
to be larger on a general-purpose machine than on a specialized one,
especially if the general-purpose machine has other work to do in
addition to routing.

----------------------------

     Your best bet is to use routers, because they will isolate
the various networks from each other. Bridges pass all packets
from one physical segment to the next. I cannot tell you what
impact using a workstation or server as a router will have on that
host's performance; I'm sure there must be some. You'd need to
figure out how heavily loaded the machine is now, and balance
the extra load against the cost of a dedicated router.

----------------------------

probably cheapest solution is to put a bridge somewhere on the network.
 they tend to be cheaper than routers. we use clearpoint bridges.

----------------------------

I recommend that you migrate everything over to twisted pair 10BaseT centered
around wiring hubs. Based on your description you network would be completely
down if there was one break in the cable. Using wiring hubs limits the chance
that the entire network would go down.

We use Cabletron equipment at Nova. We are VERY happy with both the hardware
and the level of support we receive from Cabletron. Their hubs can manange
three physical types: Token Ring (802.5), FDDI, and Ethernet (802.3).

The ethernet equipment supports AUI ports, 10BaseT ports, and thinwire.
Cabletron's phone number is 1-603-332-9400.

Subnetting is a good idea if you can group systems into logical workgroups
OR if your network traffic load is getting too high. I do not recommend
using a Unix box with a second ethernet port as a router though. As you
said, it will degrade interactive performance and Unix as good at routing
large amounts of data as a dedicated router is.

We use cisco Systems routers. cisco even makes a router module that slides
into the Cabletron hubs. cisco's phone number is: 1-415-326-1941.

----------------------------

mailing list to which you should submit this sort of questions is:
 
    Sun-Nets: networking sun workstations
        sun-nets-request@uunet.uu.net add requests
        sun-nets@uunet.uu.net submissions

There are a number of books on the topic of designing a network [which
is what you are trying to do] and design considerations [such as when
a bridge is better than a router].

Reports on routers and bridges performance and compliance are posted
to comp.protocols.tcp-ip [such as the reports from the Network Device
Test Lab at Harvard - available via FTP from hsdndev.harvard.edu:/pub/ndtl/]

----------------------------

You may want to contact Clearpoint about their pyxis/carina
e-net brouter. This unit has 4 ports plus a WAN port. the 4 port may be
configured for BNC or AUI (all four must be of the same type). We are expecting
to get one for evaluation RSN (problems with our purchasing dept.).
As the on board software is currently set up, you can have 4 segments in
two sub-nets, although they were expecting to have an upgrade to 4 subnets
within a quarter. The units come in 1 of 3 "flavors". A stand-alone unit,
a PC/AT bus parasite (uses the bus for power only), and a VME bus (6U)
parasite.

There is also a high-performace chassis, I believe they call it the
"Little Dipper", that takes interface cards. each interface card has
AUI,MAU and BNC on it. This unit also has WAN abilities.

Also, we have been using SS-2's with second ethernet cards, and noticing
little degradation of the perfomance, but we are not doing animation.

----------------------------

Options:

    Repeater
    Bridge
    Router
    Sun acting as router

Comparisons

    Repeater
        + low price
        + fast
        + plug it in and it works
        - does not filter packets
    Bridge
        + filters packets
        + low-end bridges almost a cheap as repeaters
        + plug it in and it works
        - low-end bridges slow (though acceptible -- we use them)
        - high-end bridges expensive, but fast
    Router
        + filters and transforms packets
        - expensive
        - may require some programming
        * probably inappropriate for your needs
    Sun acting as router
        + may be cost effective, if 2nd enet interface already owned
        - cost of add'l enet interface ~= repeater or cheap bridge (??)
        - must be programmed

We have cheap bridges (Allied Telesys, I think about $1.2K) and a multiport
bridge (Clearpoint) and a repeater (Cabletron). No Routers.

Recommendation: either a Repeater or a Bridge. Get a bridge unless a repeater
is significantly cheaper.

----------------------------

post but I have a couple of suggestions. If the machines are mostly
diskful (i.e. have OS and swap on a local disk) then you should
not have to bridge this network yet. What I would suggest as a first
step is to install a thin ethernet muti-port repeater. You can find
these with between 4 and 16 "ports" on them. What the device does
is connect multiple thin segments (it's a repeater) into one ethernet.
You can in turn connect the repeater to a thick segment via a transceiver.
So your new net would look like

   T==================X===========================T
                      |
                Thinnet repeater
                 | | | | |
                 | | | | |
                 | | | | |

The === line is your thick ether segment. Machines currently attached there
can stay there. The | lines are the thin segments. Each of these can be
a full 185 meter, 30 host segment. The X is a thick transceiver. This
ought to get you to a much more flexible configuration which can grow
easily.

----------------------------

If your interactive performance is OK now and you are only close to
exceeding the limits of the wire a repeater would be the most cost
effective option. We have a Cabletron MR-2000 repeater that works
great. Installed it and forgot about it.

----------------------------

The rule is "bridge if you must, route if you can" ... We like
Wellfleet routers. You could get a real basic one for around three to
seven thousand, I'd guess, and you'll get the best results. You
mention something about "smaller packet sizes moving from one ethernet
to another", but there's no such problem; each packet will be forwarded
from one network to the other without additional fragmentation, since
the MTU (maximum transmission unit) will be the same on both sides,
since they're both Ethernet.

If/when you subnet, you should strongly consider trying to separate
your users home directories onto the two subnets as much as possible.

E.g., make one faculty, and one students, and have the home directories
local to each network. You certainly can mount them across the router,
but you repeat it enough times, and eventually you'll have more network
load than you need to. Or, if most of your home directories are on one
or two servers, put an additional Ethernet interface in the server, and
then it's local to both networks (you could also have the server route
between the two networks, but with some impact on performance.
Instead, run in.routed with the "-q" flag, so it doesn't broadcast the
fact that it can route to the other network, and the dedicated router
(e.g., Wellfleet) will do all the routing).

I strongly recommend UTP over any sort of coax medium.

----------------------------



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:06:45 CDT