ESXi hosts using iSCSI/FC/FC/FCoE storage experiences latency issues with no signs of latency on the SAN side.
I got a client request to setup his storage infrastructure to get high performance for his few applications that requires high reads and writes (IOPs) like database and sap applications etc.
Now Setup is ESXi hosts, iSCSI SAN with 10G backend card and 10G midline switches.
As per vendor recommendation we have to change the settings on storage and VMware environment.So, I decided to change the setting on ESXi for IOPS.
ESXi Round Robin PSP (Path Selection Plug-in) uses a round-robin algorithm to balance the load across all active storage paths. A path is selected and used until a specific quantity of data has been transferred. After that quantity is reached, the PSP selects the next path in the list.
The quantity at which a path change triggered is known as the limit.
ESXi Round Robin PSP supports two types of limits:
IOPS limit: The Round Robin PSP defaults to an IOPS limit with a value of 1000. In this default case, a new path is used after 1000 I/O operations are issued.
Bytes limit: The bytes limit is an alternative to the IOPS limit. The bytes limit allows for a specified amount of bytes to be transferred before the path is switched.
Adjusting the limit can provide a positive impact to performance and is recommended by some storage vendors to change IOPS limit to 1.
Reference: VMware KB 2069356
All My datastores are using Round Robin policy.
Now Login into ESXi hosts via putty.

Run this command to see what is default or current value set.
>esxcli storage nmp device list you can see below output:
eui.018c7bf1773f0b88
Device Display Name: STARWIND iSCSI Disk (eui.018c7bf1773f0b88) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=1,TPG_state=AO}} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=3,numBytesPending=1524224} Path Selection Policy Device Custom Config: Working Paths: vmhba65:C0:T0:L1 Is USB: false
Now we are going to run the command to change the IOPS.
for i in `esxcfg-scsidevs -c |awk ‘{print $1}’ | grep eui.018c7bf1773f0b88`; do esxcli storage nmp psp roundrobin deviceconfig set –type=iops –iops=1 –device=$i; done
replace eui or naa id as per your organisation configuration.
Output will be:

Note:
You have to run this command on each ESXi hosts and per LUN basis.
You do not need to restart the host for the changes to take effect.
Testing the result:
I am running IOMeter to test the speed.