Hello, all. :-)
Sorry to have been so long in putting this together: I was hoping to hear
back from Sun before I summarized the problem. I'd like to take this
opportunity to thank the following indviduals who promptly responded to
my initial inquiry:
thurst_p@adm-srv.sat.mot.com
wilhelm@lexmark.com
reineman1@llnl.gov
Stuart Beck stuart@cs.adelaide.edu.au
Thomas Bastian bastian@germany.cerf.net
John Benjamins johnb@Soliton.COM
Marc S. Gibian gibian@stars1.hanscom.af.mil
Viet Q. Hoang vqh@dwrock.dw.lucent.com
Danny Johnson djohnson@nbserv2.dseg.ti.com
Jim Harmon jharmon@telecnnct.com
Carter Gilmer hgilmer@regis.edu
(I think thats it.. let me know if I missed you.)
And for the sake of those who are going through the Summary archives
however many months from now, my original message is included at the
bottom of this post.
Alright, now to business! :-) Many of the responses I recieved
understandably expressed concern that we had been 'hacked' or exploited in
some way. This is a perfectly fair conclusion and one that certainly at
the forefront of my mind when I initially discovered the problem. I
neglected to mention that the find scripts I ran on the system showed that
no files in /usr/bin /usr/sbin/ /sbin/ had been altered in the last week
and that all the modifications in /etc/ were accounted for. I also
neglected to mention that we run ASET reports nightly and had generated a
list of suid executables which I also check against a filesystem wide
find. If someone *had* managed to gain the root access neccessary to
delete the 400 set shadow file, they certainly could have done something a
bit more creative, and damaging...
The suggestions made for stepping up security on the system was
appreciated and also rather good, however. I am including all the
responses so that anyone doing a summary search can benefit from them.
Most of these, including the removal of unused services from inetd.conf,
running ASET, and keeping current with Sun patches, we already do at this
site, but a refresher is always useful. Thank you all very much for your
advice!
So, what was the problem, then? Well, there appears to be a problem with
large passwd and shadow that allows them to become corrupted when being
updated. This strikes me as being an insanely dangerous bug to have
survived to 2.5.1 and I was a little doubtful at first, but a number of
folks seem to have encountered the same thing. Here are some of the
responses I received:
John Benjamins wrote:
--- since this is a development machine for CS students, i'd wager the passwd/shadow files have a lot of entries (ie. >1000 entries). if this is the case, there is a bug report on this to sun from more than a year ago. probably a student changed their password, and since that's stored in /etc/shadow, the file got trashed. same reason your expect script clobbered it, it has nothing to do with expect, per se, but to do with solaris internals. the bug report states that this seems to happen only on machines with large passwd files. if you need it i can look up the bug report. no patches that i'm aware of (though i haven't looked in a long time). ---Thomas Bastian added: --- I have experienced the same problem in conjunction with expect. Was not able to reproduce it though. I believe it is a bug in the passwd command. You can probably trigger by interrupting the command at the right moment. Fortunatly it keeps a file called oshadow... ---- Phew! Thanks, Thomas. Indeed, the oshadow file *did* exist and was created at a point that coincided with the 'weird stuff' happening. Additionally, Stuart Beck remarked: --- You might be being caught by the sync problem that we encountered when we first used solaris 2.5.1 We found that the password and shadow files were being corrupted (very regularly on the x86 boxes as people insist on treating them like a pc box (Grr)). One of the people here found that things were not being written back to disk properly (or something like that, i'm not sure), he setup a cron job on each machine to sync every minute which cleared the problem. --- I thought that was what fsflush was for, but if it works! :) Mr. Carter Gilmer made me feel a little better, as it sounds like we shared common experiences and dread: --- This exact same problem happened to me one week ago. I have an expect script which is run via CGI for email users to change their passwords w/o giving them shell access. At the time, I figured I had been exploited somehow, because there were some problems with permissions, etc. Your experience is very interesting to me, because I am spending most of my time now locking down the system, and maybe the problem wasn't wily hacking after all... --- So... what now? Sunsolve nor Access1 have any patches for this problem, it seems. I am waiting to actually get the University password to Sunsolve so I can get you all the bug numbers related to this problem. I really appreciate all the help and advice: This is a really solid and mature community. The other responses I will list below, as they contain some valuable pointers for both improving security and recovering from an exploit:
---
From: thurst_p@adm-srv.sat.mot.com Dan:
You might want to look for a soft link from some unrelated file directory to /etc/shadow.
This is an old hack used by to gain access to passwds.
Good Luck,
Paul
---
From: wilhelm@lexmark.com This may or may not help you but if you really think your security has been compromised, a very likely suspect is that one of the "common" commands has been replaced w/ a trojan horse. This provides an intruder a way back into your system if he is accidentally discovered. Use 'sum' to compare these commands on your live filesystem with known safe installation media (preferably cdrom or something else read only).
I would definately check all the most commonly used commands, (ls, chmod, cat, grep, ...) especially those used during your last session.
As far as having a shadow file disappear w/o something malicious or directly accidental (rm -r *) at play, I have never seen or heard of it.
Hope this helps
Nick Wilhelm-Olsen
---
From: gibian@stars1.hanscom.af.mil (Marc S. Gibian) Please excuse the cynicism (sp?), but are you sure you don't have a student hacking the system? Maybe a timebomb/landmine left behind that you executed when you were su-ed to root?
-Marc
--- From: vqh@dwrock.dw.lucent.com (Viet Q. Hoang) No ideas what caused it but I'd suggest hard linking the shadow file to another name just in case it happens again.
Viet Hoang ---
From: djohnson@nbserv2.dseg.ti.com (Danny Johnson) my guess is that you are being hacked, but you should check the permissions of /etc in case you haven't.
--- From: reineman1@llnl.gov This sort of thing certainly shouldn't happen. I would be VERY surprised and concerned also. As you're in a university environment I would persue this as if you have been compromised.
1) Go through inetd.conf, turn off everything you do not absolutely need. Add extra logging options to anything that will accept them.
2) Send syslog to another machine, in addition to the local one.
3) Install TCPwrappers
4) If possible install SSH, and disable telnet, rsh, and rlogin from inetd.conf. Don't let anyone in unless they are an authorized SSH user.
5) Get the latest sendmail.
6) Current patches
7) Double check any NFS shares, share to systemnames only. Also consider getting rpcbind from the same place TCPwrappers comes from
8) Start running ASET in high security mode.
Good luck, Rick
**NOTE: Thanks Rick! Good tips, All of which are implemented here already, but a very nice summary of how to improve overall security on a Sun machine. I appreciate the time you took to write it. :-)
--- From: Jim Harmon <jharmon@telecnnct.com> Is the machine in question in use by any classes or students who are learning about UNIX?
Chances are you will find the missing file somewhere else on your system, where someone has gotten hold of an admin account and moved the file to evaluate it and try to spoof or edit it.
First thing *I* would do would be a find on the system, starting at root, for any occurance of "shadow", like so:
find / -name shadow -print
And see how many copies show up and from where.
Chances are the person who's "stealing" the file is putting it in /tmp or even in their own home directory, which should tell you who it was, or at least WHEN, and from there you can determine who was using the system and from where, so you can confront them about it.
The BIGGEST insult a would-be hacker can get is being asked about something he thought he was being slick about, and it's also a great deterrent from further stupidity. **NOTE: Thanks Jim, another well thought out response.
--
Well, thats about it. Hope this is useful to those out there who also have had this problem. Hopefully this isn't an issue in Solaris 2.6. I have implemented a shell script that copies crucial files like passwd shadow and some others ever 15 minutes to a seperate (secure) filesystem, so if it occurs again, recovery should be swift. I will keep my ears and eyes open and post any new developments as I encounter them. Again, thanks to everyone who responded and heres hoping that not too many of y'all encounter this rather nasty snafu. :)
Cheers,
Dan Mercer ACRS UNIX Administration Western Kentucky University Here is my original post:
-----------------------------------------------------------------------------
We've had a rather disturbing problem here that hopefully someone else out in Sun land will recognize and help us resolve. On a Sparc/20 running Solaris 2.5.1 we run as a development machine for CS students, we have had our /etc/shadow file dissapear without explanation twice now. I will do my best to detail the circumstances with as much brevity as possible. :)
The first occurance was this Spring. At the time, we were running an expect script which automatically created new user accounts, and so attributed the rather disconcerting event to some bug that occured during this process. We were able to restore the file from the mornings backups, so we recovered quickly.
The second occurance happened about 2 hours ago. I was gathering statistical data from a number of log files and had su'd to root to access some of the files. I ran a few basic file manipulation commands on messages and syslog, outputting to a text file in our admin directory. The entire time I was root, I was being watched by another administrator who was curious to see what commands I was doing. Just for the sake of being complete, here are some examples:
grep SOMESTRING /var/log/syslog* >> /export2/admin/report.0
last|awk '/SOMESTRING/ {print $2"\t"$3}' >> /export2/admin/report.1
grep SOMESTRING /var/adm/messages*|grep -v SOMESTRING \ >> /export2/admin/report.3
Nothing fancy... and nothing having to do with the shadow file... But I *was* root. I have gone through the history and it shows nothing which would have brought about the shadow file's dissapearance.
I'm really at a loss. As with last time, we were able to restore a recently backed up /etc/shadow from tape, but this is very disconcerting. Have we been exploited in some way? Am I an absolutely horrible typist and my colleague blind? Am I missing something here?
I will provide additonal information upon request and will, of course, summarize. ---------------------------------------------------------------------------
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:12:04 CDT