VMWare + Java + overcommitment= sad… + SSD = ?
We are doing a overall of our core customer systems… unfortunately the developers have no idea as to what they are actually going to need. They have us configuring bunches and bunches of VM guests with 8GB of ram in them to run Java. Our VMWare guys dutifully looked at vmware memory commit rates on the existing and a sample of the new guests and said oh yeah we’ve got lots of space and let them do it. They deployed a whole bunch of them over a week and turned them all over to devel to use. While that was obviously bad it wasn’t because they were being complete idiots, we already purchased enough to double our vmware farm but had not received licenses yet. So they did a quick look around and VMWare hosts said they had active memory free, devel was wanting their new environment up before the new hardware was ready to try and get ahead of things and the perfect storm happened.
I’ve been rummaging in the storage closet mainly here lately, trying to find the best way to do a large Oracle XMLDB installation… like 20TB in the next few months and growing past that very quickly there after. So while I’m the only VCP on premise that’s not been my cup of tea lately… So I get a call right before Christmas that our new devel environment is having network problems, seems to be available than not; once people start looking at it the problem goes away. So stepping into the unknown… I start digging poking and prodding, tcpdumps, network traces, switch diags, etc and after turning over the right stone I find out we had overcommitted memory on lots of our hosts to the tune of 60-80GB. However VirtualCenter is reporting that the host memory is only consuming 16-20GB of a 32GB… So we have a wonderful example of a guests allocating memory to their apps but not really using it. Java is pretty much universally thought of as offender #1 with this. Java has a wonderful/horrible memory managment (depends upon who you talk to), in that you give it a chunk of ram and it manages it. VMWare has a balloon driver which underload opens an app that consumes a chunk of ram having the OS push unused ram to disk and then kills itself freeing up the memory. The problem with java is that it is managing it’s own memory so the OS has no idea what to swap out. The OS trys to be intelligent with pushing out parts untouched for awhile; but garbage collection kicks in which has to touch all parts of java memroy and you are now swapping in which is the bad part as memory speed is == disk speed.
In the end we set the memory reservation in all the java guests = to the size of ram; which prevented VMWare from swapping out it memory but mean that if you had a 8GB guest it would require 8GB of ram no matter if the guest was using 1MB or 8GB. This was great for development as their problems went away but the business is now trying to figure out how to deal with the new unexpected IT costs to support their systems. Even worse development is behind so while they say yes they can skinny up their RAM requirements they aren’t ready to do it as they are squishing bugs still and don’t want to introduce more variance.
This morning I had an “aha” moment, VMWare has a hardly touched option as to where ESX should swap a guest to, I’ve got a few GB of unused SSD drives in my DMX4; what if we were to allocate a chunk of it to ESX swap space. While it will continue to be much slower than actual ram, the rotation latency should be removed; hopefully making it not nearly as bad as it was (was doing this on raid-6 SATA pumping 100+ iops per drive before so it was really bad).
I don’t have enough to configure a swap partition in hundreds of guests, so this would have to be when ESX itself pages. I’m thinking that I’ll probably need to disable swap inside the guest OS as the balloon driver would kick in pushing it to the guests existing drives before ESX will swap it out using the SSD swap location…. but I don’t know exactly how bad things have to get for the other guests before ESX will swap vs ballooning guests.
I’ll see if people are interested in trying this at work or not… will let you know on any details that come out of it. This should only be thought of as a short-term solution as having 60+GB of memory overcommitted is not going to be good anyway you think about it.
I’ve found one article this evening but it doesn’t really go into the whens/where’s more just on the performance benchmark (which did validate my general suposition): http://communities.vmware.com/blogs/chethank/2009/12/22/using-solidstate-drives-to-improve-performance-of-sql-databases-on-vsphere-hosts-when-memory-is-overcommitted