Large I/O block size operations show high latency in Windows 2008 in ESX 4.x and ESXi 4.x / 5.x
Article ID: 311584
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
In VMware ESX 4.x and ESXi 4.x/5.x, when performing a task that utilizes large I/O block sizes in Windows 2008, performance charts in vCenter or ESXTOP display very high device latency or DAVG, even though the actual throughput is excellent.
Environment
VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable
Cause
Unlike previous Windows releases, Windows 2008 can issue much larger I/O block sizes for certain operations, such as file copy. However, this can be reproduced with other applications, such as IOmeter with large I/O block size test parameters.
This issue can also be reproduced in a Linux virtual machine with the dd command and issuing commands with a large block size (i.e. - 1M).
This issue can also be reproduced in a Linux virtual machine with the dd command and issuing commands with a large block size (i.e. - 1M).
Resolution
High latency is due to a large I/O block size that needs to be split up into smaller I/O sizes in order to be transmitted to the storage device. When this I/O split occurs, the measure of latency is against the entire command, not the individual chunks.
Therefore, you must wait for all chunks to complete to the array before reporting back on the latency. This results in a false positive when high latency is observed to the array, when a performance problem does not actually exist.
Note: Reducing the Disk.DiskMaxIOSize advanced setting in the ESX/ESXi host does not improve the latency results because the Guest operating system is the one issuing large I/O block size. For more information, see Tuning ESX/ESXi for better storage performance by modifying the maximum I/O block size (1003469).
The Windows registry may be altered to issue smaller I/O block sizes, resulting in lower latency. However, this is merely a false positive. For more information, see:
Note: Reducing the Disk.DiskMaxIOSize advanced setting in the ESX/ESXi host does not improve the latency results because the Guest operating system is the one issuing large I/O block size. For more information, see Tuning ESX/ESXi for better storage performance by modifying the maximum I/O block size (1003469).
The Windows registry may be altered to issue smaller I/O block sizes, resulting in lower latency. However, this is merely a false positive. For more information, see:
- Large I/Os Are Split into 64KB Units When Using LSILogic Driver (9645697)
- The Guest Operating System Storage Considerations section in the Performance Best Practices for VMware vSphere 5.0.
While one may think that issuing a virtual machine migration or virtual machine deployment can be considered a large block operation, in reality the vmkernel issues I/Os block size of 32k. So, this is significantly smaller than what the guest operating system issues, resulting in no high latency for this operation.
Additional Information
Feedback
Yes
No
Powered by
