Re: pg_restore enhancements

On Wed, Nov 22, 2023 at 2:28 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> "Efrain J. Berdecia" <ejberdecia(at)yahoo(dot)com> writes:
> > Thanks, the issue we've run into, which I guess could be really a setup
> issue, with running a COPY command while executing pg_restore, is that if
> we are restoring a large table (bigger than 500GB) our WAL directory can
> grow to be very large.
> > I would think that if the pg_restore or COPY command was able to support
> a batch-size option, this should allow postgres to either archive or remove
> wal files and prevent having to re-size the WAL directory for a one time
> refresh operation.
> > I'm trying to gage how feasible would be to start looking at
> contributing to add such a feature to either the COPY command or pg_restore.
>
> Given the shortage of other complaints, I tend to agree with Adrian
> that there's not likely to be much interest in adding complexity
> to pg_restore (or COPY) to address this. You should probably look
> harder at the idea that you have some configuration problem that's
> triggering your WAL bloat. If COPY can run you out of WAL space,
> then so could any future bulk insert or update.
>

What OP needs, I think, since I'd use it, too, is "pg_bulkload without the
intrusive hacks and restrictions".