[plug] importing a large text database for fast search

[Bret Busby]

On 02/09/2011, Michael Holland <michael.holland at gmail.com> wrote:
> Suppose you had a large database - 1.7GB, with about 250k records in a CSV
> file.
> Each record has 8 fields - 7 headers plus a body.
> You might use a PERL script to split to files, sort into folders by
> embassy name, convert the ALLCAPS to more legible case, and remove the
> quote escaping from the body.
> Maybe add links to a glossary for the more obscure military/diplomatic
> terms and acronyms.
> But greping all this data is still slow. What is a good way to store
> it in Linux, with a full text index?

Why not just instead use the Perl scripting to insert the data into a
MySQL database with a Perl frontend to retrieve the data and run
whatever queries are wanted on it?

-- 
Bret Busby
Armadale
West Australia
..............

"So once you do know what the question actually is,
 you'll know what the answer means."
- Deep Thought,
 Chapter 28 of Book 1 of
 "The Hitchhiker's Guide to the Galaxy:
 A Trilogy In Four Parts",
 written by Douglas Adams,
 published by Pan Books, 1992

....................................................