Thanks for the *immediate* responses from:
Tim Carlson <tim@santafe.edu>
Jonathan Vafai <jjv200@acf2.nyu.edu>
"Dan L. Ostrom" <dostrom@umich.edu>
Michael Cunningham <malice@exit109.com>
Dean Humphrey <dean@cec.wustl.edu>
Casper Dik <casper@holland.sun.com>
John Roman <jrr@cts.wustl.edu>
Line Printer <lps@rahul.net>
Dave Reichard <dave_reichard@ncsu.edu>
Turns out this was indeed an attack, and one that's been attempted on more than
a few people. I've included all replies, as they all have little extra bits of
good information that may not be present in the others. John Roman's response
in particular has an excerpt from Sun which was very useful.
Luckily, it seems that the attack was unsuccessful in my case. I'll be keeping
a really close eye on things for a while to make sure, however...
Original query:
> I had the following show up in my logs last night; seems obvious someone is up
> to no good. My question is, is this a known hack/vulnerability? I don't
> *think* the attempt was successful, although I'll be watching things quite
> closely to make sure...
>
> Anyone know what exactly happened here?
>
> Apr 12 21:41:32 blade rpc.ttdbserverd[1706]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:41:32 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:41:33 blade rpc.ttdbserverd[1707]: iserase(): 78
> Apr 12 21:42:11 blade rpc.ttdbserverd[1707]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:42:11 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:42:12 blade rpc.ttdbserverd[1708]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:42:12 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:42:13 blade rpc.ttdbserverd[1709]: iserase(): 78
> Apr 12 21:42:13 blade rpc.ttdbserverd[1709]: iserase(): 78
> Apr 12 21:44:15 blade rpc.ttdbserverd[1709]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:44:15 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:44:16 blade rpc.ttdbserverd[1710]: iserase(): 78
> Apr 12 21:45:05 blade rpc.ttdbserverd[1710]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:45:05 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:45:06 blade rpc.ttdbserverd[1725]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting...
> Apr 12 21:45:06 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
> Apr 12 21:45:07 blade rpc.ttdbserverd[1735]: iserase(): 78
> Apr 12 21:50:21 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &"
> Apr 12 21:50:21 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &"
> Apr 12 21:51:11 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &"
> Apr 12 21:51:12 blade last message repeated 3 times
> Apr 12 22:06:48 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &"
> Apr 12 22:08:38 blade last message repeated 5 times
------------------------------------------------------------------------------
From: Tim Carlson <tim@santafe.edu>
had bunches of these statd attempted exploits myself.. make sure you are
up to rev on the statd patches. Otherwise you will have some
./inetd -s /tmp/bob
things floating around. then they well add a "rt" or "rewt" user to your
password file with UID '0' and away they go.. I had these people around
for a couple of days until I got everthing patched. They didn't seem to do
any harm considering they had root on my home directory server.
------------------------------------------------------------------------------
From: Jonathan Vafai <jjv200@acf2.nyu.edu>
For those of you noticing rpc.ttdbserverd acting funny....
We saw this on our system, and it turns out that we
were attacked. You can tell if you were attacked
if your /usr/sbin/inetd is 39544 long.
Also, there is a sniffer put in /usr/man/tmp/update
and the log is in /usr/man/tmp/output.
I don't think Sun has a patch for this, because
we were pretty up-to-date up until now.
------------------------------------------------------------------------------
From: "Dan L. Ostrom" <dostrom@umich.edu>
Paul,
One other VERY important thing I neglected to include:
Check your passwd and shadow files for the users rt and rewt (usually at
the bottom.)
>Date: Tue, 13 Apr 1999 12:24:50 -0400
paul@katana.com>
>From: "Dan L. Ostrom" <dostrom@umich.edu>
>Subject: Re: ttdb hack?
>
>Paul,
> We have had this attack at the UofM.
>
> The intruder blows the stack on the ingreslock service (this is all
theory and speculation on my part) to come out the other end as root.
>
> Look for the following on your system(s):
>
> /tmp/.X11-unix (there may sub-dirs/files in here they used to install with)
> /tmp/.X11-unix/john (this is a password cracking program)
> /tmp/bob (logging file for their process)
>
> /usr/sbin/rpcsh (this is a sniffer program)
>
> /usr/sbin/stats (this is one of the log files they use to collect info
they gather from the sniffers)
>
> First thing to do is vi the /etc/services file and comment out the
ingreslock entry. Then restart the inet process (kill -HUP <pid-of-inet>
>
> kill the rpcsh process. Then tar up all the evidence you have found an
keep it in a secure spot on your system so you can look at it later. Get a
snapshot of your wtmpx, wtmp and lastlog files and store them in s secure
spot as well.
>
> You have to assume at this point the the hacker already has seen any
passwords sent across the network in clear text. All you users will need to
change their passwords. You may also assume that your system binaries for
ps, ls, w, who, login, etc are also compromised and react accordingly.
>
> If you have any specific questions after wading throough all this, feel
free to email me.
>
> Hope this helps.
------------------------------------------------------------------------------
From: Michael Cunningham <malice@exit109.com>
http://geek-girl.com/bugtraq/1998_3/0843.html
YES this is a known vulnerability... I highly suggest you
disconnect the box from all networks NOW. And begin to investigate.
Make sure you perform a level 0 dump of the system before you
touch anything.
Grab the latest patch clusters from sun and start installing them
NOW on all your other boxes. You must assume that this person
has comprimised and probably rootkit'd your box. Once a box
has been rootkit'd you arent going to know the hacker is even
on the box. Make sure you get another box that has the same os
rev and compare binary file sizes to see if any have been replaced.
Also.. Since you have probably been compromised then I suggest
you reload that box from scratch.. I can think of 2 dozen ways
right off the top of my head to put in backdoors into a system.
Make sure you watch all your other systems over the next few
weeks and turn on all the logging you can. No firewall? ..
Might wanna look into setting up a quick linux box to record
traffic to that specific box.
------------------------------------------------------------------------------
From: Dean Humphrey <dean@cec.wustl.edu>
paul,
another admin i work with knows alot more about this. contact john roman;
jrr@cts.wustl.edu
look over the following notes of his.
i have had the same problem. but the hacker knows we are watching. i suggest
current patches for inetd, automountd, lockd and statd.
We found:
We got problems on root 4139 1 0 22:48:27 ? 0:00
/usr/sbin/inetd -s /tmp/bob
We have been running NFR and capturing all packets to and from several machines
(including this one). We have not configured NFR beyond the standard
configuration.
104654-05 SunOS 5.5.1: automount/automountd patch
104334-01 SunOS 5.5.1: lockd patch
104166-03 SunOS 5.5.1: /usr/lib/nfs/statd patch
104331-07 SunOS 5.5.1: /usr/sbin/rpcbind patch
and the various Openwin patches.
I have talked with CERT about the original incident. They indicated
that
- there are no known new holes related to statd, which was apparently
related to the original problem.
- There may be some interaction between the hole closed by the
automount patch and lockd and statd (which were previously patched)
and the tooltalk problem (patched on Sunday).
Otherwise, they did not have much to add.
>From this we don't have a lot of information to understand the pattern
of the attack. Our hope is that this set of patches will close the
opening.
------------------------------------------------------------------------------
From: Casper Dik <casper@holland.sun.com>
>I had the following show up in my logs last night; seems obvious someone is up
>to no good. My question is, is this a known hack/vulnerability? I don't
>*think* the attempt was successful, although I'll be watching things quite
>closely to make sure...
>
>Anyone know what exactly happened here?
Have you installed the most recent patches?
>Apr 12 21:42:11 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped
Appears someone is trying a tooltalk daemon exploit; the buffer overflow
results in a coredump o your system.
>Apr 12 21:50:21 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp
nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &"
This is a standard statd/automountd ineteraction exploit.
You need to have the latest automountd patches installed; it's also
safer to run statd as daemon:
/etc/init.d/nfs.client stop
edit /etc/init.d/nfs.client and run statd as daemon:
su daemon -c '/usr/lib/nfs/statd ...... ' &
chown -R daemon /var/statmon
/etc/init.d/nfs.client start
------------------------------------------------------------------------------
From: John Roman <jrr@cts.wustl.edu>
They were successful. You have an extra inetd process running, which
allows anyone to telnet to the ingresslock port and have a root shell.
Very Bad. If you have not killed the extra inetd processes do so right
away. It would be best to reload the operating system & patch as indicated
or upgrade to Solaris 7.
This is the information we got from Sun about this exploit:
[Forwarded message]
Some info on ... /tmp/bob and inetd issues....
I found out what the problem is. It deals with some problems with ff.core,
under the openwindows system.
Here's the workaround:
# chmod ug-s /usr/openwin/bin/ff.core
with ff.core, one is able to copy files around even if they don't have access to
them. What this entails is that one can create symlinks and such and change
what inetd actually calls. in other words instead of spawning in.rlogind, it
can spawn sh. Boom. Instant root shell.
===============================================================================================================
The problem turns out not to be so much a bug in statd as a
combination of things in statd and automountd. It is possible
to induce statd to relay a call to automountd. Automountd accepts
the request from statd since statd is running as root.
The behavior of statd is intentional, so the solution is to not
have it be root.
A patch will be forthcoming, but in the meantime there is a workaround.
The first step is to insure that you are up to rev on the following
patches:
Solaris 2.5: 103187-42 or later
Solaris 2.5.1: 104654-05 or later
Solaris 2.6: no patch required.
Solaris 7: not vulnerable
Then, on Solaris 2.5, 2.5.1, and 2.6, do the following:
Run statd as non root; this requires the following changes to the os:
adding a "statd" account, uid > 0
chown -R statd /var/statmon; chmod -R og-w /var/statmon
the following change to /etc/init.d/nfs.client (remember to
preserve hardlinks):
28c28
< /usr/lib/nfs/statd > /dev/console 2>&1
--- > su statd -c /usr/lib/nfs/statd > /dev/console 2>&1This procedure will bring you up to date on all known security problems with statd. [End Forwarded message]
------------------------------------------------------------------------------
From: Line Printer <lps@rahul.net>
That is DEFINITELY the evidence of a system cracker at work!!
You should inspect your machine(s) carefully and have a look at the following WWW page:
http://www.unc.edu/~smythe/security/issues.htm
------------------------------------------------------------------------------
From: Dave Reichard <dave_reichard@ncsu.edu>
Our University recently was hit with a massive hack job on UNIX boxes from 2.4 up to 2.6. It involves the statd process and the nfs client. I attached a security alert for you some of the info. won't apply to you since :
Security Alert : April 7, 1999
Updated 4/8/99 at 2:10pm
This document provides additional technical details on the recent breakins on Unity workstations running Solaris (Sun) operating systems connected to the university network. Linux systems connected to the university network may have also been affected.
How did the break in occur? Systems were compromised via a bug in the "statd" program. This service is started when NFS client services are started on the system from the "nfs.client" startup script. It appears that the system changes were intended to capture login id's and passwords, and not to damage systems or files. Compromised passwords may allow future attacks, which emphasizes the importance of changing your password.
Look under /usr/share/man for a tmp/bob directory - delete it.
Do a ps grep for inetd, statd, update processes and kill them.
The following files may have been changed : /bin/ls, /bin/ps, /usr/ucb/ps and /usr/sbin/inetd
The known sizes of invalid binaries are :
/bin/ps /usr/ucb/ps - bad size 24356 /usr/sbin/inetd - bad size 39544
If these files have the sizes listed above, they have been compromised and need to be replaced.
Look for multiple copies of inetd running (ps -ef | grep inetd) Look for a modified inetd.conf Look for services similar to 'ingreslo' in either inetd.conf or attempts to connect to them in logs Look for these files in /usr/share/man/tmp : update, output Using the ps command, look for the process ./update running (ps -ef | grep update). This is the sniffer. Note that these were the changes that we observed from the break-ins.
Again the file sizes listed above are only relevant for Solaris 2.51. If you are not running 2.51 you should edit the /etc/inetd.conf file and
comment out any unneeded services. If your system is running a "update" process that has been running for more than a day, it should be killed.
For any Sun Unix or Red-Hat Linux machine: If /bin/ps /usr/ucb/ps or /usr/sbin/inetd have recent unexplained modification dates, you should assume that they have been compromised.
A new service patch from Red Hat software: nfs-server-2.2beta29-7.i386.rpm is supposed to fix this particular problem on Red Hat Linux systems.
Last updated by twk on 4/8/99
-- Paul H. Yoshimune paul@katana.com
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:18 CDT