[plug] pigz --independent, gzrt/gzip file recovery, was Re: Tapes

[Nick Bannon]

On Wed, Mar 08, 2023 at 09:43:39PM +0800, Andrew Furey wrote:
> Also pigz is a must for getting data to the tape fast enough to avoid
> shoeshining. (At least in our case with LTO3/4 it made a huge difference…
> LTO8 I couldn’t seem to get it there fast enough regardless.)
> 
> (tar -cf /dev/st0 -I pigz source, rather than tar -czf.)

`pigz` has another superpower for seekability/partial recovery, in
combination with https://www.urbanophile.com/arenn/hacking/gzrt/gzrt.html /
DebianPkg:gzrt , namely `pigz --independent` .

What if you had gigabytes of vital backup with missing, corrupt blocks
partway through? can you recover some of the rest? I think with hardware
tape compression that should usually be possible, but it's sadly missing
from most software-compression-format options.
( Honourable mention of `bzip2`/`bzip2recover`; and `zstd --block-size`
is promising but the decoding won't work easily . `--rsyncable` is also
awesome, but not enough for this situation. It needs Z_FULL_FLUSH and
the ability to skip past some "garbage". )

$ seq 1 9999999|pv|pigz -i|tail -c2M |gzrecover -vp|tail
Opened input file for reading: stdin
75.2MiB 0:00:01 [60.9MiB/s] [   <=> ]
Found error at byte 10552 in input stream
Found good data at byte 0 in input stream
Found error at byte 2097144 in input stream
Found good data at byte 2043904 in input stream
Total decompressed output = 7978936 bytes
9999990
9999991
9999992
9999993
9999994
9999995
9999996
9999997
9999998
9999999

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal