Security on Open Source Networks.
Do you run your own website? Think you know security? Think again.
This week's Best of IRC column is all about security, what follows below are highlights
of the live! event entitled "Security on Open Source Networks." The panelists on this
discussion include Pat Lynch, Yazz Alta, Elizabeth Palomino and David Ford. I strongly
recommend you read the complete log of the event, the link is at the bottom of this article.
I'd also like to take this opportunity to plug the next Live! event, "Snoopy, Peanuts' dog or logger?" that will be occuring on December 19th. Don't let the name fool you, it's a talk by Marius and myself (the authors of snoopy logger) about the security of shared libraries and using them to monitor users. The details of the event can be found here.
Without futher delay here's the abrigded log of the previous Live! event:
<dlewis> What do you all consider as the most important points for people trying to secure their web site?
<Blu3> paying attention to detail
<BSD-Pat> well there are several things, usually I have a mental checklist for a lockdown, usually done on OS install, shutting down unneeded services, having sshd run at startup, and having a good snap/checksum of machines. I admit it was something in the...
<Aaton> To secure a site you need to also know what the developer will be running and what access they require. If some one needs for example FTP it should be locked down to just the system that is required to connect.
<Blu3> attention to detail in my manner means accomplishing your checklist and making sure you follow it all the way down to the period at the end of the sentence
<Blu3> use common sense. if you don't know what a service is, find out, shut it off if it's not required
<Blu3> internal services should be filtered from public access
<Aaton> Be aware that there is always a new exploit out there don't think your safe just because your last audit of your system looked good. be always on alert for changes.
<chipmunk> staying informed of whats going on what exploits have been discovered. Monitoring the sites for suspicious activity. shutting down unneded services,making the sure the versions of daemons your running are current. the trick is to give the users the ability to do their work without comprimising security
<BSD-Pat> heheh the Slashdot DDos was more fun :p
<dlewis> would you like to discuss that?
<BSD-Pat> yeah I guess, Liz just started that day, I'm sure she was more awake than I was....
<chipmunk> yeah heck of a way to start -)
<dlewis> I would assume so... :P
<Aaton> I wasn't there yet.
<BSD-Pat> but essentially a mid-sized DDoS attack , mostly SYN floods came through the new network, obviously someone wanting to see what we could handle...
<BSD-Pat> the problem mostly was, we had a small arrowpoint on loan....
<dlewis> for those of the audience who may be new to the security scene...
<BSD-Pat> and the poor thing had a stroke under the load...
<dlewis> can you explain what a DDos attack is?
<muks> what is a SYN flood?
<BSD-Pat> sure, essentially a Distributed Denial of service attack, alot of "slave" machines send out malicious packets to a target under control of a "master", they get particularly nasty of you consider that its a bunch of machines doing the devastating things
<Blu3> A DDos (distributed denial of service) is likened to a thousand mice nawing on you. You can easily kick away one mouse with little damage but a thousand are very hard to handle.
<BSD-Pat> that one or two machines would do before, like smirfs (icmp directed spoofed broadcast attack)...
<BSD-Pat> I mean smurfs....
<BSD-Pat> or simple SYN floods
<BSD-Pat> like we got that day,.
<BSD-Pat> SYN flood is....
<BSD-Pat> well to understand it, you have to know what TCP is....
<Blu3> A SYN flood is a storm of small packets. A SYN is the first part of a three way TCP session handshake.
<muks> heh, ok. what steps do we take when a DoS packet attack occurs?
<BSD-Pat> the first packet a machine sends out is a SYN packet, requesting to initiate the handshake...
<BSD-Pat> the problems with a DoS or DDos is that if you weren;t prepared to begin with, it could suck.
<BSD-Pat> there are several ways to prepare
<Blu3> The problem is that the victim server waits for the entire session to be established, thus this attack can bring a server down by consuming all the socket resources on the server.
<BSD-Pat> like QoS on the routers, allowing priority to some packets over others, and also good filtering.
<BSD-Pat> we use a combination of both, filtering mostly offloaded onto a FreeBSD firewall at Exodus.... (or actually two of them)
<BSD-Pat> I'm sure we get DoS'd but rarely do we notice it, but if the big one comes, its also good communication with an upstream (in our case, exodus)
<Blu3> Some of the steps you can take are to lower the initial timeout for the session. There are numerous ways of accomplishing that via /proc tuning, kernel mod, etc. By lowering the timeout, the server's resources are freed more rapidly and it can handle the onslaught more easily
<marius> 2.4 filtering also has some interesting rate limiting functions
<Blu3> Have on hand a list of your upstream provider's tech numbers and names.
<Aaton> Have that list posted in your office and where the machines are located... Never know when you will need it.
<Blu3> yes. [Linux] 2.4 has a plethura of tunable options in the networking area now. /proc/sys/net/ipv4/
<BSD-Pat> however I don;t know if I'd recommend .0 or pre .0 releases for such an essential thing, I'm sure its fine on a very specific set of hardware though.
<BSD-Pat> but linux 2.4 promises to be interesting in this area.
<Blu3> 2.4.0 should be fairly good, we've been hammering on the pre releases with a fairly wide audience
Linux 2.4.0 is a reference to the next stable release of the Linux kernel, currently in development. At the the
time of writing the kernel is currently at 2.4.0-test12-pre7 and a stable release is expected before the end of the year.
<dlewis> What are the absolute bare minimum services one needs on his / her boxen?
<Blu3> that's a loaded question
<BSD-Pat> depends on what you are running...
<Beret> depends on the purpose of the machine in question
<BSD-Pat> but if its a webserver , only the webserver :p
<Blu3> a web server needs port 80...or what it's listening on
<BSD-Pat> you can use serial to get in a console, or even be on the console itself, thats not optimal, but... :p
<marius> console servers are neat.
<Blu3> a mail server...25, possibly 110. the best way to answer your question is to say start with nothing then add what you need.
<BSD-Pat> yep, when setting a machine up, start with an empty inetd.conf :p
<BSD-Pat> work your way up.
<Aaton> well thoses services are daemon but there are others in /etc/inetd.conf that should be turned off..
<BSD-Pat> usually we have nothing in it anyway, some apps rely upon other things like cvs pserver, etc.
<Blu3> I have at most 'ident' in inetd.conf
<BSD-Pat> mailservers have port 25 open, sometimes port 110, sometimes the ssh port, etc.
<Blu3> A server shouldn't have fast cycling services in inetd. They should be stand-alone
<Aaton> SSH should be the ONLY way anyone should be getting shell access. there is no reason theses day to run telnet
<BSD-Pat> well, unless its SSL telnet or SRP telnet.
<Blu3> I usually have ssh and I tend set it to listen on a non-standard port in the 5 digit range.
<BSD-Pat> which are pretty secure auth and stream crypto apps.
<marius> another approach to this is to have two network interfaces, one for configuration (that has ssh or telnet on an unrouteable internal network that only few people [i..e sysadmins can reach]) then run the services _only_ on the external interfaces (that is reacheable from anywhere)
<BSD-Pat> we have an interesting setup for OSDN, the arrowpoint in conjunction with the firewall/packet filter....
<BSD-Pat> only really allows ports in from the outside we specify....
<Blu3> Agreed, an internal network relieves many issues, another being that internal traffic is kept to a different network and not sniffable on the public network
<BSD-Pat> there are in actuality 3-4 actual subnets at exodus, 3 of which most people never see.
<BSD-Pat> thats going to change in the future... as we start to put each projects (Slashdot, Freshmeat, etc) on thier own subnets.
<BSD-Pat> that way theres less danger of having any communication if one cluster does, god forbid, get compromised.
<Blu3> Using different physical networks helps not just for security but for traffic load and easy isolation of traffic statistics.
Originally a commercial product it has since been reverse engineered and open sourced.
<dlewis> How do you find rogue servers on your network?
<BSD-Pat> I'll let Yazz answer that, but sometimes you are just taken by surprise, something weird happens, what we thought was a T1 issue at one point was a machine compromise of someone's workstation.
<Blu3> If you have a secured network, pay attention to the traffic on it.
<Blu3> It's a good idea to put a machine that has two network cards on it and the public card has no services bound to it's interfaces
<BSD-Pat> we got draconian after that :p
<BSD-Pat> or a bridge like we do, we can see traffic go through the bridge. thats how we found the trinoo daemon on a developers machine.
<Aaton> its your friend and so is lsof when it comes to verifying which one is running the rogue program
<Blu3> It is considered a listen-only system and used for sniffing. Learn what your traffic patters are and be able to identify what is valid traffic. Be alert for suspicious traffic that you're not familiar with.
<dlewis> As we have explained what lsof is, Yazz, can you explain what tcpdump is?
<BSD-Pat> Yazz's script is pretty nifty and tripwire is really good too.
<marius> yes, also dsniff is a good way to monitor
<Blu3> Yes. There are a few good tools to help for watching traffic. Ethereal is one of the better GPL fashioned programs
<marius> and also enforce the use of secure communications
<Aaton> We implement snort which is similar to NFR to look for sigantures comming into and out of out network
<BSD-Pat> I tend to like NFR, even though its commercial...
<Blu3> (ethereal is a gui that decodes packets very similar to tcpdump)
<marius> yes, and they've got awesome 'network police traffic tickets' to go with it
<BSD-Pat> it takes advantage of OpenBSD's packet capture
<Aaton> sniffit is another tool for decoding packets
<Blu3> I strongly suggest using several tools as one may not show what another does
<Aaton> there are afew out there its amatter of admin choice...
<marius> aonther way to monitor - and make sure traffic is generally secure and encrypted on a network is to find a way to measure the network entropy
<BSD-Pat> like I said, when it comes to our network, we want what works, religion or licensing doesn;t enter into it as much...
<Aaton> Forensic Computing & Analysis
Read the complete log.
A collection of network sniffing utilities for many popular protocols.
Ethereal is a free network protocol analyzer for Unix and Windows.
LiSt Open Files. You'll find lsof to be an invaluable tool in tracking down both file and port usage.
Network Flight Recorder.
Snort is a lightweight network intrusion detection system, capable of performing
real-time traffic analysis and packet logging on IP networks.