The professional blog of Christopher Randel

VMWare Server hits a wall.

System:

• Supermicro 6015P-8RB
• Super Micro Low-Profile All-in-One Zero Channel RAID Card
• 1x Intel Xeon L5320 1.86GHz Quad-Core
• 4X 4GB FB-DIMM DDR2-667
• 4x 146.8GB Cheetah ST3146855LC 15K Ultra 320 SCSI SCA 80 Pin in RAID-10
• Windows Server 2003 X64 running VMWare Server 2.0
• Running 12-16 Ubuntu/Redhat/FreeBSD/W2K3 instances

Problem: Adding additional instances make the server literally hit the wall for disk access. CPU and memory of host system is within acceptable limits. Host OS is zippy. All instances are running at an incredibly slow speed. Logs per instance show information like so:

May 19 13:35:41.417: vmx| scsi0:0: Command READ(10) took 1.681 seconds (ok)
May 19 13:35:41.417: vmx| scsi0:0: Command READ(10) took 1.681 seconds (ok)
May 19 13:35:41.417: vmx| scsi0:0: Command WRITE(10) took 1.192 seconds (ok)
May 19 13:35:42.823: vmx| scsi0:0: Command WRITE(10) took 1.332 seconds (ok)
May 19 13:35:50.402: vmx| scsi0:0: Command WRITE(10) took 1.105 seconds (ok)
May 19 13:35:53.496: vmx| scsi0:0: Command WRITE(10) took 1.133 seconds (ok)

This is a theory from what I’ve read, but it fits my data, and shutting down one or two VM’s seems to restore disk I/O speeds. Other people have reported similar issues with similar enough hardware. [1] [2]

It seems any VMware server is limited to a certain number of concurrent running clients, due to a limitation for ATA in a guest operating system to read/write to a given number of files simultaneously.

Of course I don’t have any real hard evidence, but it seems to have fixed the issues for now.

Note that installing VMware ESXi hypervisor, and using the correct hardware RAID controller, you can apparently get around this multi-I/O problem