[SUMMARY] Automating the movment of log files across the network

Thanks to the numerous replies, all with good ideas. In the end I was
looking down the right path but discounted too quickly.

I will use scp with RSA encryption and a null passphrase with additional
restricting options similar to:

from="some.host.foo",no-port-forwarding,no-X11-forwarding,no-pty,command="sc
p -v -t ." <public key>

(thanks, specifically kevin@nome.net & Dan Astoorian <djast@cs.toronto.edu>
& the sshd man page - look for AUTHORIZED_KEYS FILE FORMAT )

Numerous other people suggested this as well but Kevin & Dan were the most
detailed.

Other comments and suggestions I received were:

>Also, ver. 1 of ssh has all sorts of known weaknesses -- you should really
>switch to version 2.  I can highly recommend OpenSSH.
Not true if you compile it correctly and without certain options. Also ssh2
from ssh.fi/ssh.com costs money for commercial use. Limited experience with
OpenSSH so I will stick with what I know for now, I have started playing
with it though.

>I have done this on a small scale, using scp. It is pretty secure, but the
encryption + transfer time increased by a factor of about >3 to 4.  Be
prepared to wait using this.
Not an issue for me file size is fairly small and it goes in the middle of
the night, as long as a 5M file takes less then 6 hours I am fine ;-D

> Maybe set it up so that all your servers mount a directory on the central
server and you can just drop them there.
Not across the internet. I do this already within each data center. I have
an admin node that collects all logs from each server that are stored on a
shared NetApp.


Thanks everybody I love this list,
Michael DeSimone
Computers & Stuff

 Original Message below

> I have a large number of servers scattered around the US in multiple
> datacenters. There are a number of log files, web & app servers mostly,
that
> I want to move back to a central location to perform analysis on. I am
> currently using a set of scripts to roll the logs and then move the via
ftp
> and using a .netrc file, all being via cron jobs. The ftp user I use has
no
> shell and can essentially only transfer files to and from his directories.
>
> I want a more secure way to do this preferably encrypting the entire
> transfer and not having any plain text passwords anywhere. I have thought
> about using ssh's (1.2.27 from ssh.fi is what I have everywhere) scp with
an
> empty (no password) RSA authentication but that doesn't sit well either.
>
> I have also been playing with netcat (Conner McCleod's SUMMARY kind of
> sparc'd this) but I can't find a way to encrypt the channel or
authenticate
> without having passwords laying around.
>
> All boxes are either 220's or Netra T1s running 2.7.
>
> So my question is:
> Is there a way to automate the transfer of files from multiple servers
back
> to a single point. Not having any clear text passwords laying around over
a
> preferably encrypted channel.
> Has anybody ever set something like this up? Or looked into it any way?
Any
> pointers to anything?
> I haven't been able to find anything on the net regarding this.
>
> Thanks,
> Michael DeSimone
> Computers & Stuff