After using KVM for almost couple of years (ConVirt as front end), there has been almost no problems with it. But with quite regular, and still random, interval one virtual machine has put it’s root file system into read-only state. The file system just keeps getting corrupted. Couple other virtual machines have done this also, but none have repeated this behaviour much. It seems so random in the end. Only things common are that these virtual machines are all Ubuntu 10.04 as is the host, all have their disks located on LVM and they are using virtio. Host hardware is different and so on.
Possible reasons
I have found quite many articles about virtio doing things described above. But all of these cases seem to involve file system larger than 1TB and usually large files being written on them. We have no such setups. Mostly simple web content serving machines with separate host for database. Also a fix for the problem with large file system situation had already been applied and in use by Ubuntu 10.04.
One other post was implying that kernel 2.6.32 would have this kind of a problem and upgrading to kernel 3.2 would resolve the issue. Unfortunately there is only 3.0 kernel available in Ubuntu 10.04 repositories currently. So I will leave this post open and finish this if/when I find solution for this problem.
Update (21.12.2012): World didn’t end (damn, back to work) and this seems to be fixed in 12.04 LTS.
Update (25.5.2012): Waiting for Ubuntu 12.04 LTS server to be released as final (July 2012). It has 3.2 kernel as default.