---

Welcome to the pg_bulkload Project Home Page

---

pg_bulkload is a high speed data loading utility for PostgreSQL. Here is the pgFoundry page, where you can find downloads, documentation, bug reports, mailing lists, and a whole lot more.

日本語ページはこちら

Documentation

• High speed data loader: pg_bulkload documentation
  • pg_bulkload provides high-speed data loading capability to PostgreSQL users.
• Optional tool : pg_timestamp documentation
  • pg_timestamp ia an optional tool to skip parsing overhead of timestamp string.

Performance Results

Here are comparisons between COPY and pg_bulkload. Performance was measured with basic-tuned PostgreSQL server.

Table definition

Customer table in DBT-2 benchmark, that is an implementation of TPC-C.

Index definition

There are 2 indexes. The first one is a primary key with one ascending integer column. The second one is a non-unique index with one random integer column.

There are the following measurement patterns.

1. Initial data loading to an empty table for 4GB of data
2. Appended data loading to a table with 4GB of data for 1GB of new data
3. Performance efficiencies by PARALLEL and FILTER features

Result 1: Initial data loading

COPY has better performance on initial data loading if the destination table is TRUNCATEd in the same transaction with COPY. In addition, performance of COPY is improved when the data are loaded without indexes and create indexes after the loading. However, pg_bulkload has completed in 85% of loading time compared with the tuned COPY.

Result 2: Appended data loading

The TRUNCATE hack is not available on appended data loading. Also, creating indexes after loading is not always faster than loading with indexes. Therefore, pg_bulkload has completed in 35% of loading time compared with COPY on appended data loading.

Result 3: PARALLEL and FILTER features

The parallel loader is used when WRITER = PARALLEL is specified. Performance would be improved on multi-CPU server because reading an input file and writing rows to a table are done with two processes. The actual measurement shows the loading time is shortened to 70%.

FILTER feature transforms input data in various operations, but it's not free. The actual measurement shows the loading time is increased to 250-300% with SQL functions and 150% with C functions.

PostgreSQL 9.0b2 + pg_bulkload 3.0b1 was used for all measurements, with indexes.

Conditions

Copyright (c) 2007-2010, NIPPON TELEGRAPH AND TELEPHONE CORPORATION