SUMMARY: 2 Ethernet from a SS10 on Kalpana possible? (another time)

From: Real Page (Real.Page@Matrox.COM)
Date: Wed Jul 21 1993 - 13:45:20 CDT


Sorry, this summary dit not got back here.
It has been 5 days. I saw it on info.sun-managers,
people got back to me but I suspect that
not everybody got it, so here it is again:

Yes it is possible, but it is not that simple.

The Sun is a SparcStation10 with a FSBE board.
This give me 2 ethernet ports.

The idea of hooking 2 ethernet ports on the same
network is not a gain because of the limitation
of 10Mb/s of that network.

The Kalpana Ethernet switch is a device that
switch ethernet packets from port to port almost
like a bridge depending on the MAC address.

Now, using two ethernet ports with a Kalpana will
give us about 2 x 10Mb/s network as the
switch take care of the traffic.

The important aspect of this configuration is
to have the server thinks it has 2 subnets
while the rest of the sun believes there is
no subnet.

To get it to work I needed to do the following:

1) Turn off IP_FORWARDING in the config of the kernel,
   this was causing a lot a bad routing information
   on the networks

        options IPFORWARDING="-1" # see /sys/netinet/in_proto.c

2) Assign a different Ethernet address to le1
   This is required by the Kalpana. With the
   same MAC address on 2 ports, the switch keep
   updating it's routing table.

   I added the following to rc.boot

        if [ "$1" != "xx0" ]; then
+ ifconfig $1 ether "`shcat /etc/ether\.$1`"
                ifconfig $1 "`shcat /etc/hostname\.$1`" netmask + -trailers up
        fi

   Where I put the Sun assigned ether in /etc/ether.le0 and
   a different one for /etc/ether.le1

3) Create 'virtual' subnet on the SS10 by using
   
   bogus 138.11.48.214 for le0 on 'pseudo' network 138.11.0.0
   bogus1 138.11.129.214 for le1 on 'pseudo' network 138.11.128.00
 
   Note: You need to be able to broadcast 138.11.255.255 so
         the higher subnet is the most important. Also, because
         the rest of the network is on net 138.11, we might
         need to cover the full address range with all the
         interfaces.

4) Add the following to /etc/networks

        138.11 255.255.128.0

5) Modify /etc/networks to create subnet for the SS10

        loopback 127
        matrox-ether 138.11.0
        matrox-ether1 138.11.128

6) Move the following ifconfig before NIS is started
   in /etc/rc.local in order to have the Sun believe
   it's /etc/networks & /etc/netmasks.

        ifconfig -a netmask + broadcast + > /dev/null

7) Prevented in.routed from starting in /etc/rc.local
   You don't want the Sun routing packets, you want
   the Kalpana doing it.

8) Add necessary route with 'route add ...'

9) Change half to the clients address to 132.11.129.???

10) Place those clients on subnet 138.11.0 on a different
    port of the Kalpana than the clients on net 138.11.128.

11) Modify automount maps so that client on pseudo network
    138.11.128.0 mount from 138.11.129.214
    Without this modification, outbound NFS traffic for
    subnet 138.11.128 was going to le1 but the inbound
    traffic was going to le0.

The setup is running without noticable problem for about
15 days now. (There is some strange things like two
ntp connection, but does not cause any problem)

I have no tools to mesure if we effectively get
2 x 10Mb/s but netstat show that the traffic is
split in half between each interface and that the collision
rate is back to normal level (< 1.25%).

        uptime
        7:32pm up 6 days, 6:44, 2 users, load average: 1.28, 0.77, 0.72

           ifconfig -a
        le0: flags=63<UP,BROADCAST,NOTRAILERS,RUNNING>
                inet 138.11.48.214 netmask ffff8000 broadcast 138.11.127.255
        le1: flags=63<UP,BROADCAST,NOTRAILERS,RUNNING>
                       inet 138.11.129.214 netmask ffff8000 broadcast 138.11.255.255

           netstat -i
        Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
        le0 1500 138.11.0.0 bogus 21516762 109 20531289 1 74122 0
        le1 1500 138.11.128.0 bogus1 18692667 8 23570747 0 290294 0
        lo0 1536 loopback localhost 2700868 0 2700868 0 0 0

        netstat -r
        Routing tables
        Destination Gateway Flags Refcnt Use Interface
        localhost localhost UH 6 2651461 lo0
        138.11.128.0 bogus1 U 4 15335298 le1
        matrox-ether bogus U 60 16521136 le0
        matrox-token alf-ether UG 0 0 le0

Thanks to the following peoples and in particular to
Mike Raffety to pointing out the concept of letting
the server and only the server believe that there is subnets.

        From: Ian Cassidy <pru!cassidyi>
        From: Michael Assels <mjassels@cs.concordia.ca>
        From: pbg@cs.brown.edu (Peter Galvin)
        From: hyder@niwot.scd.ucar.EDU (Paul Hyder)
        From: Jeff Bacon <bacon@mtu.edu>
        From: Mike Raffety <miker@il.us.swissbank.com>
        From: stern@sunne.east.sun.com (Hal Stern - NE Area Systems Engineer)

 Original question:
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 
 Message-Id: <9306220555.AA15108@bigbrother.matrox.com>
 Subject: 2 Ethernet from a SS10 on Kalpana possible?
 To: sun-managers@eecs.nwu.edu (Sun Managers)
 Date: Tue, 22 Jun 1993 01:55:45 -0400
 X-Mailer: ELM [version 2.3 PL11]
 Status: OR
 
 Hello,
 
 We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
 adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
 the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
 still use the same network (that is no subnet).
 
>From what I read on the archive, it does not seem to be possible because
 SunOS will only use le0 in output.
 
 Kalpana send me a configuration sheet regarding NC400 coprocessors, but
 this is not our setup, we have only le0 and le1 interface.
 
 They specify that each ethernet adapter should use a different MAC address.
 This is configurable with ifconfig by using
          8:0:20:1a:f5:a3 for le0 and
         48:0:20:1a:f5:a3 for le1
 I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
 this issue.
 
 Kalpana also say that internal routing should be disable on the SS10,
 so I have to turn off IP_FORWARDING.
 
 What about the IP address, can I use the same 138.11.48.214 on both
 ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
 and 138.11.48.215 for le1 instead?
 
 Is using 2 NC400 different than using 2 lex interface in regards to routing?
 
 Does somebody already try this?
 
 
 Replies:
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: Ian Cassidy <pru!cassidyi>
 
 
> Real Page ,
 
                 Definently don't use the same IP address on both ehternet adaptors. This will cause duplicate IP address error message.
 
 Is the bottleneck your encountering on your network due to traffic or the ethernet interface of the SS10 is unable to handle the amount of information being directed towards ?
 
 There are 3 solutions.
 
 1. maybe think about a subnet using the ss10 as the router (unless you already have a router)
 
 2. look at using ATM to give you 50 MB/s to 10K Mb/s or FDDI, 100 Mb/s ethernet.
 
 Ian Cassidy
 Prudential Life and Pensions
 Reading
 U.K.
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: Michael Assels <mjassels@cs.concordia.ca>
 
 You wrote:
 
 => We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
 => adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
 => the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
 => still use the same network (that is no subnet).
 =>
 => >From what I read on the archive, it does not seem to be possible because
 => SunOS will only use le0 in output.
 
 If you specify routes through le1, then that's where the packets will go.
 Note that you'll need to specify routes to your local subnet as static
 host routes, which may be a bit of a nuisance if there are many hosts on
 the subnet.
 
 => Kalpana send me a configuration sheet regarding NC400 coprocessors, but
 => this is not our setup, we have only le0 and le1 interface.
 
 Sorry. Can't help here.
 
 => They specify that each ethernet adapter should use a different MAC address.
 => This is configurable with ifconfig by using
 => 8:0:20:1a:f5:a3 for le0 and
 => 48:0:20:1a:f5:a3 for le1
 => I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
 => this issue.
 
 That's what we do, and there are no problems. A warning: Make sure that
 there is no time when the two interfaces are both up with the same ethernet
 address, or you'll get a very confused switch when it sees the same address
 on two ports. As far as I can tell, this means doing
 
         ifconfig le0 138.11.48.214 netmask 255.255.255.0 broadcast ...
         ...
         ifconfig le1 ether 48:0:20:1a:f5:a3
         ifconfig le1 138.11.48.215 netmask 255.255.255.0 broadcast ...
 
 in that order (i.e., the first thing you do with le1 is change its ethernet
 address). Watch out for
 
     vmunix: duplicate IP address!! sent from ethernet address: 0:0:f4:80:7:83
 
 messages in /var/adm/messages. I believe these come during the very brief
 time when le1 has no specified IP address. In any event they are only a
 nuisance, not a real problem.
 
 => Kalpana also say that internal routing should be disable on the SS10,
 => so I have to turn off IP_FORWARDING.
 
 This won't hurt, but it shouldn't even be necessary if only one subnet is
 connected to the switch. (We have 4 subnets on the switch, so we do turn it
 off.)
 
 => What about the IP address, can I use the same 138.11.48.214 on both
 => ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
 => and 138.11.48.215 for le1 instead?
 
 You'll definitely want two IP addresses. I'm not sure exactly what would
 happen with only one, but at very least you would would get multiple
 responses to arp, which cause some confusion.
 
 Good luck.
 
 Michael
 --
 +-------------------------------+----------------------------------+
 | Michael Assels, Analyst | Everything is what it is and not |
 | Comp. Sci., Concordia Univ. | another thing. |
 | 1455 de Maisonneuve O | - Aristotle (or someone else) |
 | Montreal, Quebec, H3G 1M8 +----------------------------------+
 | Voice: (514) 848-3035 | mjassels@cs.concordia.ca |
 +-------------------------------+----------------------------------+
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: pbg@cs.brown.edu (Peter Galvin)
 
 I have heard from Sun that indeed, you cannot do this because the Sun
 will only use ethernet, even if two are connected. We solved this
 problem by having multiple servers talking to multiple kalpanas, but
 only one connection between any two boxes.
                                               --Peter
 
 ------------------------------------------ --------------------------------
 Peter Baer Galvin (401) 863-7623
 Systems Manager, Brown U. CS Department pbg@cs.brown.edu
 Box 1910 (115 Waterman Street) uunet!brunix!pbg
 Providence, RI 02912 (02906) pbg@browncs.bitnet
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: hyder@niwot.scd.ucar.EDU (Paul Hyder)
 
 The only thing that has been left out in your description is the
 problem relating to the IP addresses on the two interfaces. The two
 interfaces on the Sun MUST think they are on different >networks<. You
 will need to set special subnet masks on this host or contrive a setup
 with a different IP net or subnet address.
 
 This need for different networks on interfaces produces a secondary
 problem: routing. You will need to tell the Sun which route to use for
 specific traffic. Adding static host routes is probably the best path.
 In general remember that traffic for a specific destination will need
 to always use a specific interface. Dynamic routing is likely to be
 impossible.
 
 If what you were expecting is the logical equivalent of 20Mbit Enet
 then you are likely to be less than satisfied. If what you wanted is
 two logically separate 10Mbit Enets then it works just fine.
         Paul Hyder
         Network Engineer
         NCAR
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: Jeff Bacon <bacon@mtu.edu>
 
> We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
> adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
> the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
> still use the same network (that is no subnet).
>
> >From what I read on the archive, it does not seem to be possible because
> SunOS will only use le0 in output.
 
 well, SunOS will only use one interface per subnet in output. it doesn't
 necessarily have to be le0.
 but you are basically correct, and hence the problem.
  
> They specify that each ethernet adapter should use a different MAC address.
> This is configurable with ifconfig by using
> 8:0:20:1a:f5:a3 for le0 and
> 48:0:20:1a:f5:a3 for le1
> I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
> this issue.
 
 yup
 doesn't really matter one way or the other so long as you grab some
 ether addr that isn't in elsewhere on the subnet.
  
> Kalpana also say that internal routing should be disable on the SS10,
> so I have to turn off IP_FORWARDING.
 
 shouldn't make any difference, cause the two ports are on the same subent
 so it won't forward anyway.
  
> What about the IP address, can I use the same 138.11.48.214 on both
> ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
 
 nope.
 
> and 138.11.48.215 for le1 instead?
 
 yup.
 
 
 this whole area - the idea of having two interfaces on the same subnet -
 is, shall we say, an unknown. you're not supposed to do it. you're
 not supposed to be able to do it. Unix doesn't know how to cope with it.
 
 I have some other thoughts on the matter though. Currently working on
 a paper that I'd like to present at this year's Usenix LISA conference
 on a way around the problem.
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: Mike Raffety <miker@il.us.swissbank.com>
 
 You have to tell the Sun (but NOTHING else) that the network is subnetted,
 and have half of your clients on one subnet, and half the other (but DON'T
 tell the clients about the subnetting, so they'll still talk directly to
 each other, without a router).
 
 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
 From: stern@sunne.east.sun.com (Hal Stern - NE Area Systems Engineer)
 
 while you can do this, you won't get the full benefit
 of two ethernets for *output*. on input, you will.
 you need to assign two IP addresses and two names
 to the interfaces, just like using two nc400s. there
 is no difference in the config (le, nc, doesn't matter;
 they've just device names).
 
 make sure that half of your clients use the name on le0
 and half use the name on le1, otherwise all traffic will
 come in on the le0 interface anyway.
 
 --hal
 
Since then I received the following remark from Mike

 Return-Path: <miker@il.us.swissbank.com>
 Date: Mon, 19 Jul 93 11:16:46 CDT
 X-Organization: Swiss Bank Corporation
 To: Real.Page@matrox.com
 Subject: Re: SUMMARY: 2 Ethernet from a SS10 on Kalpana possible?
 
 BTW, subnet addresses of all-zeroes and all-ones are supposed to be
 illegal, according to the RFC. I.e., if you subnet 138.11 with a
 netmask of 255.255.128.0, you get NO usable subnets, because 138.11.0
 has an all-zeroes (a single bit equal to zero) subnet, and 138.11.128
 has an all-ones (a single bit equal to one) subnet. So you choose a
 netmask of 255.255.192.0, which gives you four, minus two for the
 all-zeroes and all-ones, a net of two subnets, with hosts numbered from
 138.11.65-126, and 138.11.129-190.
 
 It so happens Sun doesn't enforce this limit (yet!), but most router
 manufacturers do, so you might want to keep it in mind for the future.
 

-- 
 Real Page                           !  Real.Page@Matrox.COM
 Gerant des Systemes Informatiques  !   (514) 685-7230 #2359
 Systemes Electroniques Matrox       !  (514) 685-7030 Fax
 Dorval, Quebec                     !   



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:08:02 CDT