[plug] Extreme text editing

[Bernard Blackham]

On Tue, Oct 07, 2003 at 09:52:59PM +0800, J Michael Gilks wrote:
> Today I had an unfortunate experience with a large text file. A guy had 
> concatenated a number of files to form a MEGA velocity file which 
> unfortunately had an error on line 10,557,152.
> The total file was 4.3GB on a remote NFS partition and contained 66,709,648 
> lines. I eventually opened the file with VI navigated to line 10,557,152, 
> changed the fifteen offending characters to spaces and closed the file after 
> saving.
> Unfortunately each operation took about 45 minutes. So 1.5 hours to load, edit 
> and save the file.
> Does anyone have any ideas about how this could be done without loading the 
> entire file.

A crude way would be something like:

$ (head -10557151 bigfile ; echo 'new line here' ; tail +10557153 bigfile) > newbigfile
$ mv newbigfile bigfile.

Though this requires creating a copy of the file which is possibly
what took so long in the first place.

My solution #2, if the changes didn't involve changing the filesize
(ie, no insertions), then you could write a small program (C, Perl,
whatever bloats your float) to open it in rw mode, seek to the right
offset (test by reading the data at the seek point), write the new
bytes, and close... but it was probably safer with vi :) Though this
method would be much faster.

Ideally, you'd use a hexediter to do it for you, if you could find a
hex editor which didn't read the entire file on load.

Regards,

Bernard.

-- 
 Bernard Blackham 
 bernard at blackham dot com dot au
_______________________________________________
plug mailing list
plug at plug.linux.org.au
http://mail.plug.linux.org.au/cgi-bin/mailman/listinfo/plug