<div dir="ltr">We'll be using ceph soon too, but for vms we just put them all on local storage and make them stateless (user writable data all goes to remote cephfs fileshare, databases stored locally but backed up frequently via bacula). That's the cheapest layout we found for high performance VM's while still having redundancy (if we lose a server coz their stateless its like a 15 minute rebuild).<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 24, 2013 at 10:43 AM, William Kenworthy <span dir="ltr"><<a href="mailto:billk@iinet.net.au" target="_blank">billk@iinet.net.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am using "ceph" - well putting a ceph/libvirsh setup in place. Anyone<br>
else playing with "ceph"?<br>
<br>
BillK<br>
<div class="im"><br>
<br>
<br>
On 24/01/13 08:34, Leon Wright wrote:<br>
> Tim,<br>
><br>
> We're using NFS off our NetApp boxes, hooked up to ESXi. Everything is<br>
> thin provisioned, but we still used cooked file systems (vmdks). As<br>
> we've yet to get better performance by direct NFS mounting to the filer<br>
> inside the VM. We haven't finished testing that though as most of the<br>
> VMs were migrated off old iSCSI fibre channel sans. The NetApps also<br>
> dedup, so inefficient space usage isn't so much of an issue for us.<br>
><br>
> Regards,<br>
><br>
> Leon<br>
><br>
> --<br>
> DRM 'manages access' in the same way that jail 'manages freedom.'<br>
><br>
> # cat /dev/mem | strings | grep -i cats<br>
> Damn, my RAM is full of cats... MEOW!!<br>
><br>
><br>
> On Thu, Jan 24, 2013 at 4:58 AM, Tim White <<a href="mailto:weirdit@gmail.com">weirdit@gmail.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:weirdit@gmail.com">weirdit@gmail.com</a>>> wrote:<br>
><br>
> I've been doing lots of research recently into iSCSI .vs. NFS for<br>
> our virtual machines at work. I now have a preference based on my<br>
> research, but am wondering if I've missed anything that others can<br>
> point out for me, based on their industry experience.<br>
><br>
> At work we are using QNAP 6 bay NAS, currently in RAID6 but aiming<br>
> to migrate to RAID10 eventually. We have 2 Virtual Hosts connected<br>
> via a dedicated storage network. We'll also be adding another NAS in<br>
> the near future, possibly a QNAP as well.<br>
><br>
> Our current setup is with ESXi and iSCSI. However I'm migrating to<br>
> Proxmox (KVM) and so have the option to migrate to NFS at that point.<br>
><br>
> Currently, iSCSI is thick provisioned, and then the VM HDD is thin<br>
> provisioned in that iSCSI. This means we can never "over subscribe"<br>
> our storage. However it also means we are using all the disk space<br>
> in the QNAP even though we've actually only used 2/3 of that (or<br>
> less). I know that iSCSI can be thin provisioned, so this is a moot<br>
> point.<br>
><br>
> iSCSI and ESXi (on QNAP, I assume higher end systems do it<br>
> differently) is essentially a file system within a file system<br>
> within a filesystem. You have the QNAP filesystem, which you then<br>
> create a sparse file in, (iSCSI backing), which you then export via<br>
> iSCSI and create a filesystem in (VMFS) which is then filled with<br>
> your disk images (VMDK) which is then shown to the guest who then<br>
> creates another filesystem in. With Proxmox you can reduce this by<br>
> using it as LVM with iSCSI backing, but then you don't get all the<br>
> qcow features (although LVM is pretty good).<br>
> To me, a qcow on NFS seems like the least "filesystem within<br>
> filesystem" that you can get.<br>
><br>
> NFS .vs. iSCSI speed? Apparently without offloader devices, they are<br>
> basically the same now days.<br>
><br>
> Handling of network issues. Apparently iSCSI does this better than<br>
> NFS, but if we have a network issue on the storage network, I<br>
> believe it's going to cause problems regardless of protocol.<br>
><br>
> From everything I've been reading, I struggle to see why iSCSI is<br>
> used to much. I can see if the iSCSI is exporting a raw raid array<br>
> for example, (no filesystem), then the filesystem within a<br>
> filesystem issue is not really there. But on low end NAS's it seems<br>
> to me that NFS is just as good. And I don't have to worry about<br>
> iSCSI provisioning, I just create the qcow's as needed, and manage<br>
> the free space so that a sudden spike in usage doesn't crash a VM<br>
> (which will happen regardless of protocol as discovered by one of<br>
> the iSCSI lun's being provisioned slightly smaller than the disk<br>
> inside it by the previous network manager).<br>
><br>
> So for all those out in the industry, what do you use, and why? How<br>
> does iSCSI make your life better, or is it a left over from when<br>
> it's performance was better than NFS?<br>
><br>
> Thanks<br>
><br>
> Tim<br>
</div></div>> _________________________________________________<br>
> PLUG discussion list: <a href="mailto:plug@plug.org.au">plug@plug.org.au</a> <mailto:<a href="mailto:plug@plug.org.au">plug@plug.org.au</a>><br>
> <a href="http://lists.plug.org.au/__mailman/listinfo/plug" target="_blank">http://lists.plug.org.au/__mailman/listinfo/plug</a><br>
> <<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a>><br>
> Committee e-mail: <a href="mailto:committee@plug.org.au">committee@plug.org.au</a> <mailto:<a href="mailto:committee@plug.org.au">committee@plug.org.au</a>><br>
> PLUG Membership: <a href="http://www.plug.org.au/__membership" target="_blank">http://www.plug.org.au/__membership</a><br>
> <<a href="http://www.plug.org.au/membership" target="_blank">http://www.plug.org.au/membership</a>><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
><br>
><br>
> _______________________________________________<br>
> PLUG discussion list: <a href="mailto:plug@plug.org.au">plug@plug.org.au</a><br>
> <a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
> Committee e-mail: <a href="mailto:committee@plug.org.au">committee@plug.org.au</a><br>
> PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank">http://www.plug.org.au/membership</a><br>
><br>
<br>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank">http://www.plug.org.au/membership</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Regards,<br><br>Adon Metcalfe<a href="mailto:adon.metcalfe@dec.wa.gov.au" target="_blank"></a>
</div></div>