Yesterday morning was unsettling to us – to say the very least, as were faced with a blank page when we were checking the homepage of a Joomla 4 website of one of our regular clients.
Our first thought, of course, was to enable error reporting, so, we went to the login page of the Joomla website, and then entered the login credentials, and, to our horror, we saw the following:
1 Can’t create/write to file ‘/tmp/#sql-temptable-5aaf-117a9e3-b8e8.MAI’ (Errcode: 30 “Read-only file system”)
As you may have imagined if you’re a fellow developer, the only thing that we ever read on that error message was “sql” – and so we thought that there was corruption at the database level. But then we checked every other website on this server, and they all had the same error, could it be that the whole database server is corrupted – we know that the database server is not down because Joomla was able to serve the login page, it’s just wasn’t allowing us to login and wasn’t displaying any content on the frontend.
So, we re-read the error message, and now we saw the “Can’t create/write to file” bit, which means that the problem is that the disk is full? Right?
Maybe, but we’re sure it wasn’t the case as we maintained the whole server and we knew for a fact that all disks/partitions on that server were < 20% full, but in this day and age, everything’s possible.
Everything was normal when we logged in to the server, and no disk/partition was full. In fact, we had plenty of space on every disk/partition on that server. Could it be that the system is suddenly unable to write to the “/tmp” folder?
So, the next step was checking the “/tmp” folder, and the moment we issued the following command in the Linux shell:
cd /tmp
We were greeted with the following error:
-bash: cannot create temp file for here-document: Read-only file system
Oh! “Read-only file system”, what does that mean? The permissions were OK (in fact, the /tmp folder’s permission were set to 777), so it’s not that. Could it be what every system administrator dread?
We checked the logs, particularly the /var/log/messages file, and we saw the following:
Aug 28 05:24:09 prod02 kernel: JBD2: Detected IO errors while flushing file data on sdb5-8
It was, as we feared, a hard disk error (sdb5 was a disk partition). In fact, we felt it was a hard disk error from the get-go, but we chose to ignore our feelings, which is obvious when you notice that we ignored seeing the flashing “Read-only file system” in every error message until we had no other option but to see it! (By the way, when we first saw the blank page, we thought that the server was hacked, which was even a much more dreadful scenario).
So, how did we fix the problem?
Restarting the server (through the client’s hosting provider) fixed the problem – but we all know it’s temporary. A permanent solution is to replace the problematic hard disk, and, better yet, replace the whole server, which is what we’re doing at the moment.
We hope that you found this post useful. If you’re running into the same problem on your website(s)/server and you want help, then don’t hesitate to contact us. We will solve the problem for you and we’ll get your website(s)/server up and running in no time (well, in as little time as possible), and for very little money.