[plug] importing a large text database for fast search

[Michael Holland]

Thanks folks. Perl frontend with SQL datbase it is then.
I'll go look at DBD::SQLite and DBIx::Class  My PERL is rusty, but I like it.

On Fri, Sep 2, 2011 at 4:30 PM, Alexander Hartner <alex at j2anywhere.com> wrote:
> When importing large amounts of data there are some things to consider which are common to all databases.
>
> If you are using individual insert statements would would want to disable AUTOCOMMIT and manuall commit every X-thousand entries as well as use a prepared statement to avoid the SQL statement from being processes over and over again.
>
> If you not going to use insert statements and opt to use something like postgresql's COPY or DB2's LOAD command you should bet much better performance for the import. Typically these import the data without applying triggers, referential integrity etc. You should have a look at the options available by the database engine you decide on.
>
> Regards
> Alex
>