Reiserfs and Decent Uptimes

December 3rd, 2007

Recently we had a server who had been up for approximately 170 days running Gentoo Linux, and using Resierfs. We had been dealing with a strange problem of /var filling up when it appeared to be fairly empty for a while, but since both apache and mysql were running on the server we didn't think too much of it. One day, I'd finally had enough and decided to finally solve the problem. It appeared at first that something was seriously wrong as after I cleaned out the old mysql databases and conveniently relocated the apache log files to another partition (and placed the appropriate symlinks) du and df were giving very different measurements of the used space on /var. The report from df confirmed that the partition was nearly full, which is typical of the behavior we had been experiencing for some time, but he report from a du -s said that /var was only about 3% filled. I immediately tried to determine if we could tell if the machine had been compromised, but no forensics tools were in use, and still aren't. That changes within the next week...

Anyway, I start googling for anything strange from Reiserfs and du df report mismatches, but nothing comes up. I consult my boss, and we decide to try a reboot and see what happens. Low and behold, after the reboot the df and du reports are in sync, and both reporting 3% usage. It seems (and I need to confirm this) that the version of Reiserfs that we are using can't handle the php session files being created and destroyed in a steady stream for such a long period of time. It just stops deallocating blocks, and makes them unavailable for new files. Again, this is just a guess, and if anyone has the real answer I would appreciate hearing about it in the comments.

1 Response to “Reiserfs and Decent Uptimes”

  1. Andy Says:

    Sorry you have to work for MSUM IT :) When I was working there...perhaps on the same server...I experienced the same problem. It took quite a bit of poking and prodding, but what it turned out to be was a process keeping the log files open, so the files weren't deleted, just unlinked. Try doing an 'lsof' to find the removed mysql binlogs and deal with the process that is keeping them open.

    I hope this was actually the problem, and not actually a reiserfs bug. If not, then sorry for sending you in the wrong direction :)

Sorry, comments are closed for this article.