[plug] ZFS and deduplicaton?

Andrew Furey andrew.furey at gmail.com
Fri Dec 27 05:27:53 UTC 2013


Edit: err gave up after 3 hours, not 30. Wasn't QUITE that bad.


On 24 December 2013 13:40, Andrew Furey <andrew.furey at gmail.com> wrote:

> Thanks Leon,
>
> Yes, interesting read; I tested with ONLY the schema files, not the L0
> files, and I get dedupe of 4 as I would have expected.
>
> The block size gave me an idea. ZFS uses 128k by default whereas SDFS is a
> default 4k; presumably this sort of data requires the lower size for that
> dedupe ratio.
>
> I eventually managed to get a test 100Gb ZFS filesystem with 4k blocks
> ("zfs set recordsize=4k backup/admin", once I eventually worked that out).
> The copying process of the 80Gb is far far slower than either of the
> previous tests (I gave up after nearly 30 hours, it having only finished
> 30Gb; it was much slower once it hit that second 20Gb file, as I'd expect).
>
> A "zpool list" of that then gives
>
>
> NAME          SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
> backup       99.5G  31.7G  67.8G    31%  1.12x  ONLINE  -
>
> So it DID make some difference, but it's so much slower and impractical
> that it looks like I'll stick with my initially-tested sdfs. (At least
> there is a .deb for it, in GoogleCode, which works well.)
>
> Andrew
>
>
>
> On 23 December 2013 17:39, Leon Wright <techman83 at gmail.com> wrote:
>
>> I've been using ZFS for a while and the deduplication pretty much "Just
>> works" from what I can tell.
>>
>> root at kitten:/home/leon# zfs list
>>
>> NAME       USED  AVAIL  REFER  MOUNTPOINT
>> zfs        506G   133G    30K  /zfs
>> zfs/data   505G   133G   505G  /data
>>
>> root at kitten:/home/leon# zpool list
>>
>> NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>> zfs    496G   353G   143G    71%  1.56x  ONLINE  -
>>
>> Filesystem            Size Used Avail Use% Mounted on
>> zfs/data              639G  506G  134G  80% /data
>>
>> I'm using more than the disk size and have 134G free :-)
>>
>> Though It may depend on the size of the files and the block sizes. This
>> site had some interesting info:
>>
>> https://blogs.oracle.com/scottdickson/entry/sillyt_zfs_dedup_experiment
>>
>> Leon
>>
>> --
>> DRM 'manages access' in the same way that jail 'manages freedom.'
>>
>> # cat /dev/mem | strings | grep -i cats
>> Damn, my RAM is full of cats... MEOW!!
>>
>>
>> On Mon, Dec 23, 2013 at 5:06 PM, Andrew Furey <andrew.furey at gmail.com>wrote:
>>
>>> Looks like it does it with hard-linking identical files and relying on
>>> most of them not changing (which is what I'm already doing successfully
>>> [with scripts by hand] for other aspects of the server backup).
>>>
>>> Unfortunately these 25Gb database files are GUARANTEED to change one to
>>> another (even 5 minutes apart, they'd have internal log pointers etc that
>>> would have changed; they're Informix IDS L0 backup files). Given that a
>>> difference of even 1 byte means it needs a different copy of the file...
>>>
>>> I'm relying on the fact that while SOME of the file will have changed,
>>> MUCH of it won't at block level. I just seem to be doing it wrong for ZFS
>>> when compared to the compression opendedup obtained (which I would have
>>> expected for the data in question).
>>>
>>> Further; running "zdb -S backup" to simulate the deduplication with the
>>> data, returned all the same numbers; so it looks like it thinks it IS
>>> deduping. Might the two systems use differing block sizes for comparison,
>>> or something?
>>>
>>> Andrew
>>>
>>>
>>> On 23 December 2013 16:25, William Kenworthy <billk at iinet.net.au> wrote:
>>>
>>>> Rather than dedupe after, is this something dirvish may be better at?
>>>>
>>>> http://www.dirvish.org/
>>>>
>>>> BillK
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 23/12/13 15:59, Andrew Furey wrote:
>>>> > Hi all,
>>>> >
>>>> > I'm testing different deduplicating filesystems on Wheezy for storing
>>>> > database backups (somewhat-compressed database dumps, average of
>>>> about 25Gb
>>>> > times 12 clients, ideally 30 days worth, so 9 terabytes raw). To test
>>>> I
>>>> > have a set of 4 days' worth from the same server, of 21Gb each day.
>>>> >
>>>> > I first played with opendedup (aka sdfs) which is Java-based so loads
>>>> up
>>>> > the system a bit when reading and writing (not near as bad on
>>>> physical as
>>>> > on a VM, though). With that, the first file is the full 21Gb or near
>>>> to,
>>>> > while the subsequent ones are a bit smaller - one of them is down to
>>>> 5.4Gb,
>>>> > as reported by a simple du.
>>>> >
>>>> > Next I'm trying ZFS, as something a bit more native would be
>>>> preferred. I
>>>> > have a 1.06Tb raw LVM logical volume, so I run
>>>> >
>>>> > zpool create -O dedup=on backup /dev/VolGroup00/LogVol01
>>>> >
>>>> > zpool list gives:
>>>> >
>>>> > NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>>>> > backup  1.05T   183K  1.05T     0%  1.00x  ONLINE  -
>>>> >
>>>> > I then create a filesystem device under it (I've tried without it
>>>> first,
>>>> > made no difference to what's coming):
>>>> >
>>>> > zfs create -o dedup=on backup/admin
>>>> >
>>>> > Now zfs list gives:
>>>> >
>>>> > NAME           USED  AVAIL  REFER  MOUNTPOINT
>>>> > backup         104K  1.04T    21K  /backup
>>>> > backup/admin    21K  1.04T    21K  /backup/admin
>>>> >
>>>> > Looks OK so far.
>>>> >
>>>> > Trouble is, when I copy my 80Gb-odd set to it with plain rsync (same
>>>> as
>>>> > before), I only get a dedupe ratio of 1.01x (ie nothing at all):
>>>> >
>>>> > NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>>>> > backup  1.05T  78.5G  1001G     7%  1.01x  ONLINE  -
>>>> >
>>>> > I also found "zdb backup | grep plain", which indicates that there is
>>>> no
>>>> > deduping being done on any files on the disk, including the schema
>>>> files
>>>> > also included (column 7 should be something less than 100):
>>>> >
>>>> >        107    2    16K   128K  2.75M  2.75M  100.00  ZFS plain file
>>>> >        108    2    16K   128K  2.13M  2.12M  100.00  ZFS plain file
>>>> >        109    1    16K     8K     8K     8K  100.00  ZFS plain file
>>>> >        110    1    16K   9.5K   9.5K   9.5K  100.00  ZFS plain file
>>>> >        111    1    16K   9.5K   9.5K   9.5K  100.00  ZFS plain file
>>>> >        112    1    16K  12.0K  12.0K  12.0K  100.00  ZFS plain file
>>>> >        113    1    16K   9.5K   9.5K   9.5K  100.00  ZFS plain file
>>>> >        114    4    16K   128K  19.9G  19.9G  100.00  ZFS plain file
>>>> >        115    1    16K    512    512    512  100.00  ZFS plain file
>>>> >        116    1    16K     8K     8K     8K  100.00  ZFS plain file
>>>> >        117    1    16K   9.5K   9.5K   9.5K  100.00  ZFS plain file
>>>> >        118    1    16K   9.5K   9.5K   9.5K  100.00  ZFS plain file
>>>> >        119    1    16K  14.5K  14.5K  14.5K  100.00  ZFS plain file
>>>> >        120    1    16K  14.5K  14.5K  14.5K  100.00  ZFS plain file
>>>> >        121    1    16K  3.50K  3.50K  3.50K  100.00  ZFS plain file
>>>> >
>>>> > 95% of those schema files are in fact identical, so filesystem hard
>>>> links
>>>> > would dedupe them perfectly...
>>>> >
>>>> >
>>>> > I must be missing something, surely? Or should I just go ahead with
>>>> > opendedup and be done with? Any others I should know about (btrfs
>>>> didn't
>>>> > sound terribly stable from what I've been reading)?
>>>> >
>>>> > TIA and Merry Christmas,
>>>> > Andrew
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > PLUG discussion list: plug at plug.org.au
>>>> > http://lists.plug.org.au/mailman/listinfo/plug
>>>> > Committee e-mail: committee at plug.org.au
>>>> > PLUG Membership: http://www.plug.org.au/membership
>>>> >
>>>>
>>>> _______________________________________________
>>>> PLUG discussion list: plug at plug.org.au
>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>> Committee e-mail: committee at plug.org.au
>>>> PLUG Membership: http://www.plug.org.au/membership
>>>>
>>>
>>>
>>>
>>> --
>>> Linux supports the notion of a command line or a shell for the same
>>> reason that only children read books with only pictures in them.
>>> Language, be it English or something else, is the only tool flexible
>>> enough to accomplish a sufficiently broad range of tasks.
>>>                           -- Bill Garrett
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au
>>> http://lists.plug.org.au/mailman/listinfo/plug
>>> Committee e-mail: committee at plug.org.au
>>> PLUG Membership: http://www.plug.org.au/membership
>>>
>>
>>
>
>
> --
> Linux supports the notion of a command line or a shell for the same
> reason that only children read books with only pictures in them.
> Language, be it English or something else, is the only tool flexible
> enough to accomplish a sufficiently broad range of tasks.
>                           -- Bill Garrett
>



-- 
Linux supports the notion of a command line or a shell for the same
reason that only children read books with only pictures in them.
Language, be it English or something else, is the only tool flexible
enough to accomplish a sufficiently broad range of tasks.
                          -- Bill Garrett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20131227/d94dc72d/attachment.html>


More information about the plug mailing list