When it comes to squeezing every ounce of performance from an Apache web server it is easy to make changes that can have dire consequences to your sites. Sometimes you can see your web server load go up right away so it’s easy to tell you are headed in the wrong direction. But, some modifications can make a real mess of things in a much more subtle way.

The Problem:

Load on my web server went up when several of my competitor’s sites got booted from the top 10 of Google which left my site as the lone survivor for a very competitive key word. That day my poor little 2.8 dual Xenon with 2GB of RAM web server went from a average load of 2.7 to well over 80 and had spikes in the 100+ range.

How I “Fixed” It:

I went in and started messing with the httpd.conf file. I changed just about everything under the sun and then thought it would be smart to turn KeepAlive off. Now I could free up all those idle connections hogging memory until they time out. It was supposed to be a win-win. My visitors will get a better experience and load will go down. Or so the theory goes…

Why This Went Horribly Wrong:

If you run web sites that have lots of pictures each time one of the photos was grabbed from the server it made a new connection. So if you have a page that has 60 images you can see how the number of connections each user was making adds up quickly. If they have one of those web caching programs/plugins that will follow links and download pages automatically in order to speed up dial up browsing it gets even worse.

As part of my server hardening I run (D)DoS Deflate a great DDOS mitigating script written by the good folks a Medialayer. Since KeepAlive was off these visitors were making hundreds of completely legitimate connections to the web server. (D)DoS Deflate would see this and then ban their IP. I can’t blame the script it was doing exactly what I told it to do. The effect was that thousands of visitors and potential customers were getting their IP’s banned from my server. Yeah Ouch!

Fixing the “Fixed”:

In the first thirty minutes of the Apache reconfiguration I received easily 100 banned IP notifications. I thought that my server was getting DDOS’ed and was comforted by my superior server hardening and Apache tweaking skills. And then the next half an hour I got another 100. I knew something was up. Then I got an e-mail from a friend of mine alerting me to one of my sites being unavailable. That was odd since I was literally writing a post in Wordpress. Taking a look at the log, sure enough he had been banned. Luckily, I made the connection between (D)DoS deflate and KeepAlive and quickly turned it back on. Everything was right with the world again.

Lessons Learned:

1. If you run sites with lots of photos on individual pages do not turn KeepAlive off if you are running (D)Dos Deflate
2. Always remember that Apache is a part of your server eco-system. Mess one up, mess them all up. You’re Apache ninja skills are never good enough to prevent all screw ups so be careful.
3. It’s great when your competitors lose their ranking leaving you in the top spot in Google for competitive keywords.

P.S. No actual clients were harmed. This event happened on a server reserved for my own personal affiliate work.