SUMMARY: Virtual IP and NFS/Network problem

From: John Bradley (john.bradley@sr5.chinalake.navy.mil)
Date: Fri Aug 14 1998 - 17:17:20 CDT


Thanx to:

John D Groenveld <jdg117@elvis.arl.psu.edu>
"V. Q. Hoang" <vqh@dw.lucent.com>

Also:
Bill Crane <bill.crane@equifax.com>
"Michael J. Garcia" <mjgarcia@corp.auspex.com>
Nickolai Zeldovich <kolya@zepa.net>
Gerald Combs - Unicom Communications <gerald@unicom.net>

SUMMARY:
=============
See patch 105786-05, part of suns recommended patches (Bug Id: 4077132)
-
One suggestions was to setup static routes to the interface.
To do this, I add entries to /etc/gateways with the following format:

host <remote address> gateway <local address> metric 0 passive
-
I've added the Bug Id to the bottom of this email.

ORIGINAL:
============
I have multiple (160+) virtual host on a web server called "server (100.100.100.5)".
The problem is when I try to NFS mount another system called "mail", mail thinks
it's the last virtual host defined IP address (100.100.100.172).
Consequently I get "permission denied". The "mail" system is sharing
to "server", and not the other IP address. The hosts files list
"server" as 100.100.100.5. I can't change it to 100.100.100.172 because
we are constantly adding new virtual hosts, and shouldn't have to.

Is there a way to make sure 100.100.100.5 is the interface it talks
out on? Should I be virtual hosting another way?
Any other thoughts?

thanx,
jb

Here is some data
===================

Script to setup Virtual IP addresses:
-------------------------------------------
# This line disables "interface pooling"
ndd -set /dev/ip ip_enable_group_ifs 0
ifconfig le0:10 up
ifconfig le0:10 100.100.100.10 #somefoo.com
ifconfig le0:11 up
...
ifconfig le0:172 up
ifconfig le0:172 100.100.100.172 #otherfoo.org
# This line must be last
# It disables "Interface pooling"
#
ifconfig le0:0 100.100.100.5 down
ifconfig le0:0 100.100.100.5 up

Output of snoop of NFS request:
-------------------------------------------
     mail -> 100.100.100.172 DNS R port=37661
100.100.100.172 -> mail PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 proto=UDP
     mail -> 100.100.100.172 PORTMAP R GETPORT port=995
100.100.100.172 -> mail MOUNT3 C Null
     mail -> 100.100.100.172 MOUNT3 R Null
100.100.100.172 -> mail MOUNT3 C Mount /export/dir
     mail -> 100.100.100.172 MOUNT3 R Mount Permission denied

System Data:
-------------------------------------------
OS: Solaris 2.6
H/W: SPARC 20

##################################################################
SUN's Bug report:

Bug Id: 4077132
 Category: kernel
 Subcategory: tcp-ip
 State: evaluated
 Synopsis: ip_enable_group_ifs=0 doesn't pick default hostname.
 Description:
cu has multiple virtual interfaces designated for le0. This worked fine on
2.5.1. Cu upgraded to 2.6 and now finds that a new feature of 2.6 (interface
groups) has begun selecting interfaces at random. Solaris 2.6, as it is
shipped
is supposed to operater this way to make load handling more efficient.

However, by setting the ndd option ip_enable_group_ifs to 0, it disables the
interface groups functionality and allows 2.6 to operate like a 2.5.1 or
previous OS release.

The problem is that when the interface groups are disabled, the hostname
selected to handle network calls is the last hostname.le0:[0-9] file
found. I tested this on my workstation and found that no matter what
the last hostname.le0:? file was, the last file found would become the
interface the system always used for network calls.

This is a bad thing for customers who use host authentication and have
an ever-changing network...like webservers.

When interface groups are disabled, the loghost (hostname.le0) should be
selected.
 Work around:
Have the hostname.le0 host name be selected when the interface groups are
disabled.

casper.dik@Holland 1997-10-02

[[ DELETED WRKAROUND ]] <--WHY?????????????????????????

This workaround works for EVERY SINGLE CASE I have seen
to date.

Do not delete this workaround. Even if it only works
part of the time, it is still better than nothing.

The workaround, as stated previously is to (prior to deletion):

        1. cp hostname.<interface> (e.g. hostname.le0) to
            hostname.<interface>:99.

            NOTE: You must use 99 because there will never, ever
                   be a file listed in higher than that (unless
                   there are over 1000 virtual interfaces, then
                   999 would need to be used).

        2. verify that hostname.<interface>:99 matches the contents
            of /etc/nodename.

        3. change the contents of hostname.<interface> to some other
            hostname.

Please do not delete this workaround. Even if it doesn't work all
of the time, it is better than nothing.

It didn't work around the problem all of the time. <- see above

If deleting routes doens't work around theproblem that can only indicate
that a routing problem has been introduced in 2.6, I can't quite put a
finger on it yet.

If routing worked properly, the following should work around the problem:

        ifconfig le0:X metric 100

or
        route delete <interface route>

However, since neither workaround works we do have a problem.

steve.hodnett@East 1997-10-23
Startup script workaround can be added to rc2.d

/etc/rc2.d/S70fixroutes
#!/bin/sh
#
echo "Turning off ifgrps"
/usr/sbin/ndd -set /dev/ip ip_enable_group_ifs 0
#
echo "Removing Duplicate Virtual Interfaces Network routes"
/usr/sbin/route delete net <ip#ofle0:1> <network#>
/usr/sbin/route delete net <ip#ofle0:2> <network#>
....
/usr/sbin/route delete net <ip#ofle0:xx> <network#>

casper.dik@Holland 1997-10-28

I deleted *MY* workaround because it didn't work.

The route delete workaround also doesn't work.
peymaneh.mirshafiei@Corp 1997-11-17
        Integrated in releases:
 Duplicate of:
 Patch id:
 See also: 4081315
 Summary:

End of SUN's Bug report.
##################################################################

Our work around...

in /etc/rc2.d/S69inet put:
#disable the round robin...
/usr/sbin/ndd -set /dev/ip ip_enable_group_ifs 0

then
# put the IP you want the traffic to go through last on the list to be ifconfiged up
cp /etc/hostname.le0 /etc/hostname.le0:99
mv /etc/hostname.le0:<last logical device> /etc/hostname.le0

-- 
                                                  wwwww
                                                g( o o )g
----------------------------------------------o00--(_)--00o---
+  John Bradley  NAWC Chinalake, CA                          +
+  UNIX/PC/Mac/Network Administrator - CTA Inc.              +
+  Phone: 760.939.5887 pager: 760.375.0997     .oooO Oooo.   +
+  E-Mail: john.bradley@chinalake.navy.mil     (   ) (   )   +
------------------------------------------------\ (---) /-----
                                                 \_) (_/



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:12:46 CDT