An old client of ours called us yesterday evening and told us that his website was extremely slow to load. Our first guess was that it was hacked, but it was not; his website was very clean. What could it be?
We then thought, well, maybe he has some huge queries running in the background and slowing down the whole website, so we enabled the MySQL Slow Query log on his Joomla website, but the log file was not populated with any slow query. We thought there was something wrong with the logging system, so we saved all the queries run by the server into a text file (all we needed to do was to slightly modify the setQuery function in the JDatabase class to write all the queries to a text file), and then we pasted all the queries into the SQL page in phpMyAdmin and ran them all in one shot, but they ran in under a second. Odd…
Maybe the load on the server was very high, we thought. So we checked the load on the server using top, and it was below 0.3 – an extremely comfortable load level (meaning that there are no load issues on the server whatsoever). Hmmm…
Our next attempt was to contact the host and tell them about the problem, maybe it’s a routing/DNS issue on their end. Their reply was immediate and clear: “The website loads extremely fast on our end, so it might be the script or it might be that your ISP is experiencing connection issues. Please take a look at your script’s code and/or check with your ISP.” A typical first response from any hosting company to throw the blame on anything but their servers. We were relentless, though… So we replied back: “We’re sure it’s not the script and we have checked the website from multiple places and multiple ISPs, and the problem is always there. Could you please check the website from outside your network? Maybe it’s a routing issue…” We got a reply that they’re going to take a “closer” look.
Sure enough, 10 minutes later, we got the following reply: “We’ve updated and tweaked your firewall… It looks like it was just the firewall blocking a little more than was necessary. It looks like the SYN_FLOOD protection was enabled, which will cause these sorts of issues sometimes. We’re sorry for the inconvenience this must have caused you. Please confirm that the issue is resolved and please let us know if you run into any more issues.” Ahaaaa!
So it was something on their network! Now, You might be wondering, what is this SYN_FLOOD thing? Well, for those of you who have some technical knowledge in the web, you might already know that there is something called DoS (Denial of Service attack). Denial of Service attack, in short, is a kind of malicious attack that will send many simultaneous connections from different IPs to the web server, until the server is brought down to its knees. One method of dealing with this is to limit the number of connections from an IP to a defined number (this is a setting that relates to the SYN_FLOOD term and is defined at the firewall level). Now remember, every file that is requested from the server and every image is a connection, which means that a web page that has 10 images, and that calls about 5 CSS files and 7 JavaScript files means that there are at least 28 connections (one master connection to the page, 12 for loading files, and 10 for loading images). If the aforementioned defined number is less than 28, then in nearly all cases, the page loading will be very slow, because the firewall will think that this is an attack (the firewall has no way to distinguish legitimate connections from malicious connections). What happened, in the case of the client, is that the maximum number of connections was always above that number – hence the extreme slowness of the website – increasing that number solved the issue.
Now, in case you’re experiencing the same problem then we suggest you call your host and ask them to check your firewall settings. If your host confirms that everything’s OK with your website (it’s important to be persistent), then the problem might be on your actual website, and this where we are very useful! Just contact us and we’ll sure fix the problem for you in no time and for a very affordable fee. By the way, we are the friendliest programmers you’ve ever worked with (and you’ll ever work with) – at least that’s what our clients tell us!