[plug] Tapes

[Brad Campbell]

On 23/4/23 18:34, John McCabe-Dansted wrote:
> On Thu, 30 Mar 2023 at 17:36, Brad Campbell <brad at fnarfbargle.com <mailto:brad at fnarfbargle.com>> wrote:
> 
>     Having had a play for a week or so, I've found pigz -i compresses nearly as well as lbzip2 and 3 times faster. Admittedly the CPU is an old AMD FX-8350, so it's not exactly a screamer.
>     I'm using a 996GB source directory of 28 rsync backups from various machines ranging from ~250M to ~590G. I tar and compress these to the staging drive before writing them to tape.
> 
> 
> Is this to avoid the shoe-shining effect?

Yep. The machines and archives I'm staging from are not fast. The Rsync backups are particularly pathological and tar can drop to ~20MB/s while reading bits of them even from a local drive. Even then the fast bits would be still limited by the network if I didn't stage locally.

I figure staging to an SSD gives me the best chance of avoiding any issues with tape speed. The HP drive logs any repositions and thus far I've managed to avoid them. It happily spools ~1.3TB to tape at full speed.
  
> 
>     tar through lbzip2 gets them down to 753G in 5 hours 23 minutes. That is limited by compression.
>     tar through pigz -i gets them down to 763G in 1 hour 50 minutes. That's not limited by compression, rather tar reading the data from the "super cheap and nasty budget SSD" I chose to play with.
> 
> 
> You could also try `zstd -T8`
>  

I could, but being unfamiliar I did a bit of a search on zstd and wasn't comforted with the amount of "it ate my data" and "test your backups actually decompress properly" I found. I also couldn't find any conclusive "we can split the output into independent blocks so if you corrupt one we can still decompress the rest". I did find some relatively recent info on the new "block splitter", but most of that was "it's new, has sharp edges and we'll fix it when we get a chance".

At least with bzip2 and pigz -i these are solved problems. I'm happy to trade speed for reliability, but as I said with pigz -i the compression isn't the bottleneck.

Thanks for the pointer though, I'll keep an eye on it as it matures.

Regards,
Brad