Question: We have a Linux server (2.6.28-11-generic #42-Ubuntu) that's misbehaving on a client site, gobbling up an entirely unacceptable percentage of the client's bandwidth, and we're trying to figure out what the heck it's doing. And the guy who had the sysadmin skillset has yet to be replaced.

We're at a loss for what could be causing all that network traffic, and need to figure it out SOON. What log files should I be looking at to find this information? What analysis tools would you recommend for this task?

Please note that I'm not looking for a tool that will allow me to analyze FUTURE traffic. The client is on the verge of shutting the machine off entirely; I need to figure out what it's been doing with the data I already have, if that's at all possible.

My thanks in advance for helping a development monkey play sysadmin.

Answer: 
If its moving alot of data quickly, i'd see if there's any heavy log activity in /var/log/\*, namely httpd/ and maybe FTP. Otherwise check /var/log/security and /var/log/messages, those will contain a large portion of your system logs.

Warner recommended some good tools that i would suggest turning on and monitoring the traffic as it is happening. If you want the system down take it off the network and run the above tools with it disconnected from the primary network.
If you have already isolated the server without a doubt..

[iptraf](http://iptraf.seul.org/) is useful for bandwidth measurement on a local server.

[ntop](http://www.ntop.org/) can be useful to breakdown on a higher level. If you install and start to running now, it can analyze past traffic.

Ultimately, [tcpdump](http://www.tcpdump.org/tcpdump_man.html) and [wireshark](http://www.wireshark.org/) should be all you need.

Otherwise, I would spend time on the network equipment to isolate and analyze the traffic. [Netflow](http://en.wikipedia.org/wiki/Netflow) is something network admins like to pay a lot of money for but the OSS tools aforementioned can accomplish the same.

Ultimately, you're unlikely to have evidence specific to traffic in logs if you have not already configured something to do that. Depending on the protocol, you could analyze Apache logs and other application logs for historical patterns. If you have not isolated on any level, you are going to have a difficult time without enabling additional tools.