Back in May (today is the 21st of June, the day that should have been the summer solstice, but instead it was yesterday, since this year is a leap year [you learn something new every day!]) we had a client who approached us with a very weird problem. The client’s problem, to put it simply, was the sudden inability to login to both the backend and the frontend. In short, the login worked, and then, all of a sudden, it didn’t (it just timed out). After a while, it started working again only to stop working shortly after.
From the get-go, we felt that this had something to do with an aggressive firewall, as firewall issues are usually characterized by erratic problems. What confirmed our theory was the fact that the problems started happening not long after the website was temporarily moved to a different server.
Since we didn’t have root server access to the temporary hosting (and neither did the client, for that matter), we asked the client to authorize us to move her website to another high-end host that we trust, and she did. So, we moved the website, we tested the website (including the login), and everything worked smoothly! We were happy and the client was happy, for a while…
The next morning the client called us and told us that she was experiencing the exact same issue: her clients were not able to login. That, of course, meant that our firewall theory was wrong, and there was something else causing this problem. So, we checked what was going on, and we noticed that there was an authentication plugin connecting to Paradiso which is an online LMS. The way how that plugin worked was a bit complex: it authenticated the login on Paradiso, and then Paradiso authenticated back the login on the joomdle extension on the Joomla website. Disabling the plugin fixed the problem, but, as you might have probably guessed, it also broke the most important functionality that the website had, which was the LMS integration (needless to say, this made the website useless).
Investigating the issue further proved that the login was timing out when the Paradiso authentication plugin (on the Joomla website) was connecting to the Paradiso website (in order to post the login credentials), so we thought that the issue was definitely on Paradiso‘s end. So we asked Paradiso to check the logs on their end to see what’s going on, but they told us that everything was normal on their end, and that the client’s server IP was not blocked.
So, we just took the Paradiso URL and tried to load it directly (from the browser), and it worked. Hmmm… We were still not convinced and we still thought it was Paradiso‘s problem. Nevertheless, we decided to check our logs, and, to our surprise, we found the following:
2016-05-19 15:02:22 SMTP connection from [52.36.77.46]:38053 (TCP/IP connection count = 1) 2016-05-19 15:02:32 SMTP call from us-west-2.compute.amazonaws.com [52.36.77.46]:38053 dropped: too many syntax or protocol errors (last command was "?4?2?\016?\r?\031?\v?\f?\030? ?") 2016-05-19 15:22:53 SMTP connection from [52.36.77.46]:38397 (TCP/IP connection count = 2) 2016-05-19 15:23:03 SMTP call from us-west-2.compute.amazonaws.com [52.36.77.46]:38397 dropped: too many syntax or protocol errors (last command was "?4?2?\016?\r?\031?\v?\f?\030? ?") 2016-05-19 15:23:06 1b3TXG-0005Wz-GJ <= root@host.[ourclientdomain].com U=root P=local S=2865 T="lfd on host.[ourclientdomain].com: blocked 52.36.77.46 (US/United States/us-west-2.compute.amazo" for root
Aha! So, it was our client's server that was blocking Paradiso's IP, and not the other way around, and it was blocking their IP because it was sending malformed syntax to the mail server, which explained this whole mess. Finally! After working on this problem for days we found out what the exact cause was!
So, how did we fix the problem?
Fixing the problem was easy - we just whitelisted Paradiso's IP and we then told Paradiso to fix their mail sending program, which they said they will do immediately (we can't confirm whether they fixed it or not).
And why wasn't this a problem for our client before?
Most likely because our client's website resided on a server with no firewall, or, at best, with very loose firewall settings. Any half-decent firewall should block malformed requests to any server application, which incidentally, brings us to this question: why didn't other Paradiso customers complain about this and why was our client the only one affected with this? Obviously, this is a rhetorical question, but a valid question nonetheless.
In any case, we hope that you found this post helpful. If you are facing the same problem on your Joomla website, then check your server logs as the problem may be at the server level. If you can't find anything in the logs or if you need help, then please contact us. We will find what the problem is, we will fix it for you, you won't pay us much, and most likely, you will gain new friends (that's us)!