A week or so ago I posted that our Sun workstations and servers running
SunOS version 4.0.3 were hit with some unusual network requests, which
generated some unusual console messages some campus hosts.
The messages were not recorded in the system logs, this made it a bit
difficult to pinpoint the exact time that each workstation was bothered
by the flurry of the unusual activity. This helped to make it seem
like each machine was being tried in turn, versus all being bothered at
the same time - a flag to check for damage or activity denoting the
presence of a cracker going throught each system in IP order.
In summary, it looks like the new workstation with configuration
problems theory is good, there was a new workstation with problems, and
the problems can be linked to it by time.
There was some concern that the problem might be a cracker, but further
investigation mostly ruled out that theory, so we're keeping an extra
close eye on things.
The only mystery that remained was that a few of the systems that
recieved the messages were having a heck of a time with quotas. The
sa's of thoses sytems simply no longer check quotas ...
We'll all keep our eyes open for suspicious activity & we'll forward
any new developments.
Below is the collection of email that I got on the request of the
sun-managers.
Many many thanks to the complete and quick responses of sun-managers!
##### Forwarded Email #####
From: deltam!dm!mark@uunet.UU.NET (mark galbraith)
Message-Id: <9110170029.AA04078@dm.deltam>
Received: by flyer.deltam (4.1/SMI-4.1)
id AA01550; Wed, 16 Oct 91 17:30:15 PDT
To: uunet!welchlab.welch.jhu.edu!johnj@uunet.UU.NET
In-Reply-To: John A. Johnston's message of Wed, 16 Oct 91 17:43:06 EDT <9110162143.AA01622@welchlab.welch.jhu.edu>
Subject: Console messages not recorded?
Status: RO
The program numbers listed seem to correspond to the numbers listed in
the /etc/rpc file. It appears that either something broke on each
machine, or more likely (IMHO) someone was trying to run some sort of
security breaking program on your machines.
The errors look very suspicious to me in that the host numbers are all
255.255.255.255. This is not normal and usually indicates a
"Broadcast" packet. Why a broadcast packet would be sent to "keyserv"
(program 100029) would be a question I would ask. In so doing, you
may come to the same conclusion I did.
The good news is, your system appears to be secure. At least they
didn't get in that way.
Do a bit more research, and if you find that you have suffered a break
in, or an attempted break in, contact the CERT folks by sending a
message to 'cert@cert.sei.cmu.edu', or calling 412-268-7090. They
operate a 24-hour hotline for reports of security problems.
--Mark Galbraith Voice: +1 510 449 6881--
--Software Engineer UUCP: uunet!deltam!mark--
--Delta Microsystems, Inc. Domain: mark@deltam.com--
______________________________________
From: tgsmith@spdev.East.Sun.COM (Timothy G. Smith - Special Projects)
Message-Id: <9110170030.AA23054@spdev.East.Sun.COM>
To: John A. Johnston <johnj>
Subject: Re: Console messages not recorded?
Status: RO
The messages are from the portmapper.
The reason the messages didn't make it to the log files is thar
portmap use fprintfs to talk to stderr instead of using syslog. This
is arguably a bug. portmap should probably use syslog.
Program 100029 is keyserv. Do you run secure rpc? Do you have
undergrads around? Were all of the machines on the same network?
It kind of sounds like someone was screwing around and *might* have
been into a bit of mischief.
______________________________________
From: stern@sunne.East.Sun.COM (Hal Stern - Consultant)
Message-Id: <9110170253.AA06490@sunne.East.Sun.COM>
To: johnj
Subject: Re: Console messages not recorded?
Status: RO
the problem comes from a machine that was set up with an IP
address of 255.255.255.255. when that machine tries to get its
keyserv daemon running (rpc program 100029), it sends a packet
to (what it thinks is) itself -- but that's a broadcast address,
so everyone gets it. on the machines that get this broadcast,
the portmapper throws away the request because it's not made
from the local machine -- only processes running locally can
make requests of the portmapper.
i'm sure you got one set of these messages for each rpc daemon:
nfs, keyserv, ypserv, lockd, etc.
fix: try
% ping -s x.y.z.255 | fgrep 255.255.255.255
where x.y.z is your network number. this will tell you the
name of the machine that's got the bogus IP address, but it
will also bring your network to its knees while doing so.
--hal
______________________________________
From: AnnKian <yeoak@nusdiscs.bitnet>
To: johnj
Message-Id: <00950430.208339E0.7965@nusdiscs.bitnet>
Subject: RE: Console messages not recorded?
Status: R
Sound like your syslog.conf in /etc is not set to have the messages recorded.
Check whether all the log levels have been included in the syslog.conf
to be directed to /var/adm/messages.
AnnKian
yeoak@nusdiscs.bitnet
______________________________________
From: erueg@cfgauss.uni-math.gwdg.de (Eckhard Rueggeberg)
Message-Id: <9110171219.AA05853@cfgauss.uni-math.gwdg.de>
To: johnj
Subject: Re: Console messages not recorded?
Status: R
If you look at your /etc/syslog.conf, you will probabely
find a line
*.err;kern.debug;auth.notice;user.none /dev/console
or something similar, and a line like
*.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages
Perhaps there is an information class in your case, like auth.info, which is
dierected to the console but not to the messages file.
rpcinfo -p lists
program 100029 to be keyserv, and 100024 as status, and host 255.255.255.255
is the global broadcast, so perhaps a booting workstation in you network
tried something strange.
Eckhard R"uggeberg
Mathematisches Institut der Universit"at G"ottingen
Bunsenstr. 3 - 5
3400 G"ottingen
Germany
erueg@cfgauss.uni-math.gwdg.de
______________________________________
Subject: Re: Console messages not recorded?
In-Reply-To: Your message of Wed, 16 Oct 91 17:43:06 EDT
Reply-To: brent@napa.Telebit.COM
Date: Thu, 17 Oct 91 08:35:55 -0700
From: Brent Chapman <brent@napa.Telebit.COM>
Status: R
#
# Today on our local net we had some occurances of an error message
# being written to the console screen (idle sun console at the login
# prompt) or to the console window (sunview) that were not recorded
# anywhere in the system (messages, dmesg).
#
# Vitals:
# OS version: SunOS 4.0.3 & 4.0.3c
# Arch: 4/490, 4/60, 4/65
#
# The messages can were similar to the following:
#
# > nonlocal attempt to unset (program 100024 version 1) from host
# > 255.255.255.255 port 658
# >
# > nonlocal attempt to unset (program 100029 version 1) from host
# > 255.255.255.255 port 659
# >
# > nonlocal attempt to set (program 100029 version 1) from host
# > 255.255.255.255 port 660
#
# The port numbers increased and the program number varied a bit, some
# were listed as version 2.
#
# There were maybe 50 messages scrolled on the few workstations still
# running SunOS 4.0.3. The messages were not recorded as folks
# vigorously hunted down the source and ended up scrolling things too far
# before a record was made.
#
# The 50 or so lines of the message occured only once on each machine,
# but I cannot detect if they all came at the same time; eg. all the
# machines got them at the same time, or it was a successive type of
# error, with one machine getting the error, then the next etc. - as if
# the machines were "attacked" by a stray roving type of program.
#
# Two other managers on the local net reported the same trouble, with the
# additional note that checking quota's on login was taking a very long
# time after the "storm".
#
# Any ideas why the messages were not recorded, or what the messages may
# be a result of? The other two managers that had the problem have yanked
# their machines from the net, fearing hostilities.
#
# Many thanks.
# -johnj
#
My bet is that the messages are from "portmap", and were simply
written by the program to /dev/console, rather than syslogged as they
should have been.
It looks like somebody is trying to unmap/remap some of your RPC
services. 100029 is "keyserv" (see /etc/rpc for the list of program
numbers and programs). If the port number is sequentially increasing,
then somebody's almost certainly trying to guess what port something
is living on, and hack around it.
I'd consider it hostile until proven otherwise, particularly since the
IP source address on the packets is the broadcast address (which is
never a valid source address...). Since the perpetrator is intending
to use the broadcast address to hear any responses, he must be on your
local net. The command "etherfind -r -v -x -src 255.255.255.255" will
show you any of these bogon packets that come by. The first 6 hex
bytes of the hex display for each packet are the destination ethernet
address. The second set of 6 bytes is the source ethernet address;
from that, you may be able to figure out what machine it's coming from
(look for that address in your "arp -a" output, or your /etc/ethers
file, or your "ethers" YP map).
-Brent
______________________________________
From: vasey@mcc.com (Ron Vasey)
Posted-Date: Thu, 17 Oct 91 11:30:28 CDT
Message-Id: <9110171630.AA03191@glitter.aca.mcc.com>
Received: by glitter.aca.mcc.com (4.1/ACAv4.0i)
id AA03191; Thu, 17 Oct 91 11:30:28 CDT
To: johnj
Subject: Re: Console messages not recorded?
Status: R
The "source host" of your "culprit" is an all-ones broadcast,
which is why everyone sees it and is so difficult to track down.
Even worse: every BSD-based host with IP_forwarding enabled
also "forwards" (repeats) the broadcast, creating quite a storm.
Check to see if any Mac gateways have recently been added to your
network. Kinetics Fastpaths and Cayman Gatorboxes generally use
broadcast packets to talk among themselves (VERY anti-social!!),
which can intrude on other stations not prepared to ignore them.
Unfortunately, the larger implications of networking are not dealt
with in the shrink-wrapped "plug and play" world of Macs & PCs.
But then if they were ... I might not have a job, either ... :^)
++ Ron Vasey@mcc.com MCC:ES_Lab Austin_TX_78759 512+338-3461
______________________________________
From: keves@meaddata.com (Brian Keves - Consultant)
Message-Id: <9110171714.AA00444@casedemo.meaddata.com>
To: johnj
Subject: Re: Console messages not recorded?
Status: R
Those are RPC messages that you are receiving. You could easily have had
a network storm caused by an improperly configured system on the network.
Check your /etc/syslog.conf file and see how you have syslog configured.
That should explain why the messages were not logged. Your syslog.conf
probably tells syslog not to log them.
Brian
-- Brian Keves | Opinions/Ideas presented | Mead Data Central P.O. Box 149 | here are not necessarily | Engineering Productivity Miamisburg, OH 45343-0149 | those of Mead or Mead | keves@meaddata.com (513) 865-1121 x5767 | Data Central | ...!uunet!meaddata!keves ______________________________________ From: gds@la.tis.com (Greg Skinner) Message-Id: <9110172152.AA16520@la.tis.com> To: johnj Subject: Re: Console messages not recorded? Status: RThese messages may just be from user programs that write to /dev/console. In that case they wouldn't go to /etc/dmesg or /usr/adm/messages, since those are kernel messages.
Those errors look like SunRPC program and version numbers to me. 1000024 is the status server, and 100029 is the keyserv server. (You can find out more about these RPC numbers by checking the man pages for rpcinfo.) It looks like someone was broadcasting requests to these servers, and your configuration does not permit a "set" operation. This may or may not be evidence of a breakin. You can find out more by running etherfind or tcpdump to identify the source of these broadcasts.
--gregbo ### End ###
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:06:17 CDT