[plug] importing a large text database for fast search

Bret Busby bret.busby at gmail.com
Fri Sep 2 14:58:29 WST 2011

On 02/09/2011, Michael Holland <michael.holland at gmail.com> wrote:
> Suppose you had a large database - 1.7GB, with about 250k records in a CSV
> file.
> Each record has 8 fields - 7 headers plus a body.
> You might use a PERL script to split to files, sort into folders by
> embassy name, convert the ALLCAPS to more legible case, and remove the
> quote escaping from the body.
> Maybe add links to a glossary for the more obscure military/diplomatic
> terms and acronyms.
> But greping all this data is still slow. What is a good way to store
> it in Linux, with a full text index?

Why not just instead use the Perl scripting to insert the data into a
MySQL database with a Perl frontend to retrieve the data and run
whatever queries are wanted on it?

Bret Busby
West Australia

"So once you do know what the question actually is,
 you'll know what the answer means."
- Deep Thought,
 Chapter 28 of Book 1 of
 "The Hitchhiker's Guide to the Galaxy:
 A Trilogy In Four Parts",
 written by Douglas Adams,
 published by Pan Books, 1992


More information about the plug mailing list