[plug] Deduplication

[Scott Middleton]

Hi

Very coincidental conversation

I have been looking into dedupe for a while.

I am currently testing BtrFS on duperemove. Bedup is more file level
and I didn't get a lot of saving with all the virtual images I am
using.

I am testing on:
root at Ubuntu-14:~# uname -a
Linux Ubuntu-14 3.14.1-031401-generic #201404141220 SMP Mon Apr 14
16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root at Ubuntu-14:~# btrfs --version
Btrfs v3.14.1

My client data is 1.7 TB mainly of virtualbox images, shadow protect
and standard Microsoft backup images.

Duperemove:
https://github.com/markfasheh/duperemove

Posting to the BtrFS mailing list they say that I should be using a
3.15 RC kernel to get the latest patches.

http://article.gmane.org/gmane.comp.file-systems.btrfs/35078


What I am trying to achieve is an offsite backup capability for VM
images. Hopefully to be able to backup offsite on a block level. ZFS
(ZoL), BtrFS and SDFS all do this.

In the past I tested Opendedupe. http://opendedup.org/ they have a new
version now so I may try them again. The Java part of SDFS is annoying
though!

ZFS Dedupe is a constant and reading about it, they say it requires
20GB RAM for 1TB of data.

Opendedup is different again. BtrFS seems the go but not stable enough
yet for the features I want.



On 12 May 2014 18:25, Will Dowling <will at autodeist.com> wrote:
>> Yeah I'd say BTRFS itself is very stable on kernel 3.12+, and bedup had
>> a bug with sqlite (https://github.com/g2p/bedup/pull/45) fixed in the
>> latest version which was causing some issues, since those 2 changes we
>> have found it pretty nice.
>
>
> I'll have to upgrade to the latest code then, this was a bit of a drama for
> me a few months back.
>
> With regards to btrfs stability, I wouldn't be comfortable running it at
> $job, but am more than comfortable running it for noncritical applications
> (ie: home).
>
> Russell Coker posts regularly about his experiences with btrfs and is always
> worth a read:
> http://etbe.coker.com.au/tag/btrfs/
>
>
>
> Will Dowling
>
> F: +61 (08) 6364 4880
> F: +61 (08) 6364 4881
> E: will at autodeist.com
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership