[plug] importing a large text database for fast search
bret.busby at gmail.com
Fri Sep 2 14:58:29 WST 2011
On 02/09/2011, Michael Holland <michael.holland at gmail.com> wrote:
> Suppose you had a large database - 1.7GB, with about 250k records in a CSV
> Each record has 8 fields - 7 headers plus a body.
> You might use a PERL script to split to files, sort into folders by
> embassy name, convert the ALLCAPS to more legible case, and remove the
> quote escaping from the body.
> Maybe add links to a glossary for the more obscure military/diplomatic
> terms and acronyms.
> But greping all this data is still slow. What is a good way to store
> it in Linux, with a full text index?
Why not just instead use the Perl scripting to insert the data into a
MySQL database with a Perl frontend to retrieve the data and run
whatever queries are wanted on it?
"So once you do know what the question actually is,
you'll know what the answer means."
- Deep Thought,
Chapter 28 of Book 1 of
"The Hitchhiker's Guide to the Galaxy:
A Trilogy In Four Parts",
written by Douglas Adams,
published by Pan Books, 1992
More information about the plug