[plug] Cheap iSCSI .vs. NFS for VM's

[Tim White]

I've been doing lots of research recently into iSCSI .vs. NFS for our 
virtual machines at work. I now have a preference based on my research, 
but am wondering if I've missed anything that others can point out for 
me, based on their industry experience.

At work we are using QNAP 6 bay NAS, currently in RAID6 but aiming to 
migrate to RAID10 eventually. We have 2 Virtual Hosts connected via a 
dedicated storage network. We'll also be adding another NAS in the near 
future, possibly a QNAP as well.

Our current setup is with ESXi and iSCSI. However I'm migrating to 
Proxmox (KVM) and so have the option to migrate to NFS at that point.

Currently, iSCSI is thick provisioned, and then the VM HDD is thin 
provisioned in that iSCSI. This means we can never "over subscribe" our 
storage. However it also means we are using all the disk space in the 
QNAP even though we've actually only used 2/3 of that (or less). I know 
that iSCSI can be thin provisioned, so this is a moot point.

iSCSI and ESXi (on QNAP, I assume higher end systems do it differently) 
is essentially a file system within a file system within a filesystem. 
You have the QNAP filesystem, which you then create a sparse file in, 
(iSCSI backing), which you then export via iSCSI and create a filesystem 
in (VMFS) which is then filled with your disk images (VMDK) which is 
then shown to the guest who then creates another filesystem in. With 
Proxmox you can reduce this by using it as LVM with iSCSI backing, but 
then you don't get all the qcow features (although LVM is pretty good).
To me, a qcow on NFS seems like the least "filesystem within filesystem" 
that you can get.

NFS .vs. iSCSI speed? Apparently without offloader devices, they are 
basically the same now days.

Handling of network issues. Apparently iSCSI does this better than NFS, 
but if we have a network issue on the storage network, I believe it's 
going to cause problems regardless of protocol.

 From everything I've been reading, I struggle to see why iSCSI is used 
to much. I can see if the iSCSI is exporting a raw raid array for 
example, (no filesystem), then the filesystem within a filesystem issue 
is not really there. But on low end NAS's it seems to me that NFS is 
just as good. And I don't have to worry about iSCSI provisioning, I just 
create the qcow's as needed, and manage the free space so that a sudden 
spike in usage doesn't crash a VM (which will happen regardless of 
protocol as discovered by one of the iSCSI lun's being provisioned 
slightly smaller than the disk inside it by the previous network manager).

So for all those out in the industry, what do you use, and why? How does 
iSCSI make your life better, or is it a left over from when it's 
performance was better than NFS?

Thanks

Tim