[plug] ZFS and deduplicaton?
techman83 at gmail.com
Mon Dec 23 09:39:51 UTC 2013
I've been using ZFS for a while and the deduplication pretty much "Just
works" from what I can tell.
root at kitten:/home/leon# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zfs 506G 133G 30K /zfs
zfs/data 505G 133G 505G /data
root at kitten:/home/leon# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zfs 496G 353G 143G 71% 1.56x ONLINE -
Filesystem Size Used Avail Use% Mounted on
zfs/data 639G 506G 134G 80% /data
I'm using more than the disk size and have 134G free :-)
Though It may depend on the size of the files and the block sizes. This
site had some interesting info:
DRM 'manages access' in the same way that jail 'manages freedom.'
# cat /dev/mem | strings | grep -i cats
Damn, my RAM is full of cats... MEOW!!
On Mon, Dec 23, 2013 at 5:06 PM, Andrew Furey <andrew.furey at gmail.com>wrote:
> Looks like it does it with hard-linking identical files and relying on
> most of them not changing (which is what I'm already doing successfully
> [with scripts by hand] for other aspects of the server backup).
> Unfortunately these 25Gb database files are GUARANTEED to change one to
> another (even 5 minutes apart, they'd have internal log pointers etc that
> would have changed; they're Informix IDS L0 backup files). Given that a
> difference of even 1 byte means it needs a different copy of the file...
> I'm relying on the fact that while SOME of the file will have changed,
> MUCH of it won't at block level. I just seem to be doing it wrong for ZFS
> when compared to the compression opendedup obtained (which I would have
> expected for the data in question).
> Further; running "zdb -S backup" to simulate the deduplication with the
> data, returned all the same numbers; so it looks like it thinks it IS
> deduping. Might the two systems use differing block sizes for comparison,
> or something?
> On 23 December 2013 16:25, William Kenworthy <billk at iinet.net.au> wrote:
>> Rather than dedupe after, is this something dirvish may be better at?
>> On 23/12/13 15:59, Andrew Furey wrote:
>> > Hi all,
>> > I'm testing different deduplicating filesystems on Wheezy for storing
>> > database backups (somewhat-compressed database dumps, average of about
>> > times 12 clients, ideally 30 days worth, so 9 terabytes raw). To test I
>> > have a set of 4 days' worth from the same server, of 21Gb each day.
>> > I first played with opendedup (aka sdfs) which is Java-based so loads up
>> > the system a bit when reading and writing (not near as bad on physical
>> > on a VM, though). With that, the first file is the full 21Gb or near to,
>> > while the subsequent ones are a bit smaller - one of them is down to
>> > as reported by a simple du.
>> > Next I'm trying ZFS, as something a bit more native would be preferred.
>> > have a 1.06Tb raw LVM logical volume, so I run
>> > zpool create -O dedup=on backup /dev/VolGroup00/LogVol01
>> > zpool list gives:
>> > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
>> > backup 1.05T 183K 1.05T 0% 1.00x ONLINE -
>> > I then create a filesystem device under it (I've tried without it first,
>> > made no difference to what's coming):
>> > zfs create -o dedup=on backup/admin
>> > Now zfs list gives:
>> > NAME USED AVAIL REFER MOUNTPOINT
>> > backup 104K 1.04T 21K /backup
>> > backup/admin 21K 1.04T 21K /backup/admin
>> > Looks OK so far.
>> > Trouble is, when I copy my 80Gb-odd set to it with plain rsync (same as
>> > before), I only get a dedupe ratio of 1.01x (ie nothing at all):
>> > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
>> > backup 1.05T 78.5G 1001G 7% 1.01x ONLINE -
>> > I also found "zdb backup | grep plain", which indicates that there is no
>> > deduping being done on any files on the disk, including the schema files
>> > also included (column 7 should be something less than 100):
>> > 107 2 16K 128K 2.75M 2.75M 100.00 ZFS plain file
>> > 108 2 16K 128K 2.13M 2.12M 100.00 ZFS plain file
>> > 109 1 16K 8K 8K 8K 100.00 ZFS plain file
>> > 110 1 16K 9.5K 9.5K 9.5K 100.00 ZFS plain file
>> > 111 1 16K 9.5K 9.5K 9.5K 100.00 ZFS plain file
>> > 112 1 16K 12.0K 12.0K 12.0K 100.00 ZFS plain file
>> > 113 1 16K 9.5K 9.5K 9.5K 100.00 ZFS plain file
>> > 114 4 16K 128K 19.9G 19.9G 100.00 ZFS plain file
>> > 115 1 16K 512 512 512 100.00 ZFS plain file
>> > 116 1 16K 8K 8K 8K 100.00 ZFS plain file
>> > 117 1 16K 9.5K 9.5K 9.5K 100.00 ZFS plain file
>> > 118 1 16K 9.5K 9.5K 9.5K 100.00 ZFS plain file
>> > 119 1 16K 14.5K 14.5K 14.5K 100.00 ZFS plain file
>> > 120 1 16K 14.5K 14.5K 14.5K 100.00 ZFS plain file
>> > 121 1 16K 3.50K 3.50K 3.50K 100.00 ZFS plain file
>> > 95% of those schema files are in fact identical, so filesystem hard
>> > would dedupe them perfectly...
>> > I must be missing something, surely? Or should I just go ahead with
>> > opendedup and be done with? Any others I should know about (btrfs didn't
>> > sound terribly stable from what I've been reading)?
>> > TIA and Merry Christmas,
>> > Andrew
>> > _______________________________________________
>> > PLUG discussion list: plug at plug.org.au
>> > http://lists.plug.org.au/mailman/listinfo/plug
>> > Committee e-mail: committee at plug.org.au
>> > PLUG Membership: http://www.plug.org.au/membership
>> PLUG discussion list: plug at plug.org.au
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
> Linux supports the notion of a command line or a shell for the same
> reason that only children read books with only pictures in them.
> Language, be it English or something else, is the only tool flexible
> enough to accomplish a sufficiently broad range of tasks.
> -- Bill Garrett
> PLUG discussion list: plug at plug.org.au
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the plug