Cool, my first SUMMARY message ever.
My original question was:
Recently we had a software pirate upload 30 megs of software to a directory on our FTP server that was writeable. This pirate has been back many times since the first event in December, 1996. I keep deleting the software and am keeping logs, and am in communication with the individual's Internet access provider... but other than consuming a minor portion of our bandwidth and temporarily taking up hard disk space, this does no harm.
This pirate is connecting through a slow link, possibly 28.8, and it takes all night for them to send software ... nobody else ever comes to get copies, so I'm astonished that the person keeps sending the stuff to our server only for it to be deleted. I suppose that 14 year-olds don't have anything better to do.
At any rate, within the past two days I've noticed the CPU and packet meters on our FTP server running at or near maximum 24-hours per day. A ps -e -o pid -o c -o comm reveals that it is our in.named process that is consuming the CPU cycles (at 80% or higher normally) but I have not been able to track down the source of the packet traffic.
We're running Solaris 2.5.
netstat -a reveals no active UDP nor unusual TCP activity in progress on the machine. I'm not very experienced when it comes to monitoring network traffic, but there would seem to be a flood of UDP packets coming from somewhere and going to our DNS server. This might be connected to my deleting the files of this computer pirate, sort of a weak attempt at payback.
How can I determine whether this theory of a "UDP denial of service attack" directed at the DNS server on our system is in fact occuring?
Are there other things that might cause a nameserver to max-out the CPU and generate network traffic?
-----------------------------------------------------------------
The answer was to use the snoop command to log all of the packets on the network. I used the following:
snoop -d le0 -o snoop.le0.1.10.97.11.23.am.txt -V -N
And was able to determine very easily that there was, indeed, a flood of UDP traffic zooming to and from our DNS server. The culprit was not, however, the pirate who has been visiting our FTP server. Instead it was a combination of our secondary nameservers, backup UUCP feed, and newsfeeds all furiously sending and receiving UDP packets on port 53. The command that interprets the snoop log file is:
snoop -i snoop.le0.1.10.97.11.23.am.txt
which produces output similar to:
38 0.00062 ns.cts.com -> 207.92.75.27 DNS R port=53
39 0.00054 ns.cts.com -> 207.92.75.27 DNS R port=53
40 0.00061 ns.cts.com -> 207.92.75.27 DNS R port=53
41 0.00058 ns.cts.com -> 207.92.75.27 DNS R port=53
42 0.00057 ns.cts.com -> 207.92.75.27 DNS R port=53
43 0.00054 ns.cts.com -> 207.92.75.27 DNS R port=53
44 0.00691 hydrogen -> donews.cts.com DNS R port=53
45 0.01671 titanium -> hydrogen TELNET C port=3391
46 0.00066 hydrogen -> titanium TELNET R port=3391 /dev/le (promiscuo
us
47 0.00538 hydrogen -> donews.cts.com DNS R port=53
48 0.00923 207.92.75.27 -> ns.cts.com DNS R port=53
49 0.01551 207.92.75.27 -> ns.cts.com DNS R port=53
50 0.01312 207.92.75.27 -> ns.cts.com DNS R port=53
51 0.01333 207.92.75.27 -> ns.cts.com DNS R port=53
52 0.01459 207.92.75.27 -> ns.cts.com DNS R port=53
53 0.01273 207.92.75.27 -> ns.cts.com DNS R port=53
54 0.01574 207.92.75.27 -> ns.cts.com DNS R port=53
55 0.01311 207.92.75.27 -> ns.cts.com DNS R port=53
56 0.01888 207.92.75.27 -> ns.cts.com DNS R port=53
57 0.01872 207.92.75.27 -> ns.cts.com DNS R port=53
58 0.01636 hydrogen -> donews.cts.com DNS R port=53
59 0.01906 hydrogen -> donews.cts.com DNS R port=53
60 0.01318 titanium -> hydrogen TELNET C port=3391
Killing and restarting in.named solved the problem. I figured that it might, but didn't want to just kill the process without a better understanding of what was going on. I don't have a complete understanding of what went wrong, as I didn't have time to track it down further and coordinate with the other hosts involved in this packet barrage, but thanks to the help of the following people I was able to learn about using snoop:
Trevor Kirby [Trevor.Kirby@newcastle.ac.uk]
K. Ravi [RAVKRISH.IN.ORACLE.COM.ofcmail@in.oracle.com]
Mark Bergman [bergman@phri.nyu.edu]
Iván Villalobos Hernández [denp@acnet.net]
Karl E. Vogel [vogelke@c17.wpafb.af.mil]
The following people suggested installing the software called tcp-wrappers or wuftpd ( so that I could filter out the pirate's IP address and keep him/her from returning. The pirate is using an Internet access provider with dynamic IP address assignment, and I don't want to exclude every one of their customers. Besides, 22-year-olds (namely, me) don't seem to have anything better to do, either, than track down a pirate who is, at the very least, wasting his/her time and talent and is, at worst, going to cause somebody some harm if they ever learn anything useful.
Dr. Peter Watkins [peter@jrc.nl]
Mark Bergman [bergman@phri.nyu.edu]
Karl E. Vogel [vogelke@c17.wpafb.af.mil]
The best way to get rid of this twit is to run the tcp wrapper suite. The
source is on ftp.cert.org under one of the tools directories.
Noah Vawter [noah@intel-sol.com]
A suggestion by K. Ravi [RAVKRISH.IN.ORACLE.COM.ofcmail@in.oracle.com] introduced me to the wonders of truss -p <pid>. The truss command can be used to debug the system calls performed by a process, as well as signals and machine faults it encounters. With the -p option, an existing process can be debugged. Otherwise, truss launches a specified executable and debugs it.
Glenn Popelka [popelka@cnaindex.mtb.phil.mop.com] had a very interesting observation, as well:
One possibility is that the intruder might have installed an 'fsp' daemon
on your machine. fsp is a transfer protocol that runs over UDP. I've
forgotten what fsp stands for, but some say 'filched software and
pornography'. It can be slow, which might account for the long upload. After
all, his 28.8 link is to his ISP, not to you.
You can look for fsp with archie, the distribution will fill you in on
the details. Look carefully for unknown programs running on your system,
and the dates and checksums of your system executables. Also, get tcp
wrappers. I've forgotten off hand where it is or how to search for it.
Peter Marelas [maral@phase-one.com.au] suggested the following:
Use the debug option in in.named to see whats being received. (-d)
Rick Fincher [rnf@spitfire.tbird.com] said:
I think some of Adrian Cockroft's performance monitoring software from Sun will let you do that too. It is available through Sunworld online (http://www.sun.com/sunworldonline) and is free.
Thanks also to Bob Woodward [bobw@kramer.filmworks.com] for input.
Regards,
Jason Coombs
Scientist, SCIENCE.ORG
http://www.science.org/jasonc/
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: 2.6.2
mQCNAzH47i4AAAEEAKjzazn04nZvnPQo/Rj+O026OyDi6kw6Rvl13YcfjgiaE76t
qU0COmWDpDLtDgrYSosDRwFdIPJuo4esXhUfgpWFEF2BXTqO+r9sKV9A3VBKGP3A
wXMhcSZH7PQR96KMPlLEl/DoQa9KfcUPfb3stf2UfNrOmaYyZEfm6QRcLcyhAAUR
tCFKYXNvbiBDb29tYnMgPGphc29uY0BzY2llbmNlLm9yZz6JAJUDBRAx+O6bR+bp
BFwtzKEBARp0A/wKb3WAhILkc30bOhvxrMab50iUYWPQaJObrBpQc3CsbCartFZe
Xf5LStYtBrjXNlW+r0yJVn0ETE3MHDLE2wEWGiuEIpGcXYMaigajxAI+RuIxdK5c
Xx4BM/7vKIhvFHEHvBKVP35xeJK2r9lAa0RQGp05I3t81T9EuumBg/StrA==
=aUAR
-----END PGP PUBLIC KEY BLOCK-----
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:11:42 CDT