[plug] importing a large text database for fast search
michael.holland at gmail.com
Sun Sep 4 12:56:20 WST 2011
Thanks folks. Perl frontend with SQL datbase it is then.
I'll go look at DBD::SQLite and DBIx::Class My PERL is rusty, but I like it.
On Fri, Sep 2, 2011 at 4:30 PM, Alexander Hartner <alex at j2anywhere.com> wrote:
> When importing large amounts of data there are some things to consider which are common to all databases.
> If you are using individual insert statements would would want to disable AUTOCOMMIT and manuall commit every X-thousand entries as well as use a prepared statement to avoid the SQL statement from being processes over and over again.
> If you not going to use insert statements and opt to use something like postgresql's COPY or DB2's LOAD command you should bet much better performance for the import. Typically these import the data without applying triggers, referential integrity etc. You should have a look at the options available by the database engine you decide on.
More information about the plug