Re: speed concerns with executemany()

On Thu, Jan 19, 2017 at 02:23:15PM +0200, Aryeh Leib Taurog wrote:
> >>
> >> Adrian, have you got a benchmark "classic vs. joined" on remote
> >> network? Thank you.
> >
> > With NRECS=10000 and page size=100:
> >
> > aklaver(at)tito:~> python psycopg_executemany.py -p 100
> > classic: 427.618795156 sec
> > joined: 7.55754685402 sec
>
> This is really interesting. I have long been using a utility I put
> together to insert using BINARY COPY. In fact I just brushed it up a
> bit and put it on PyPi: <https://pypi.python.org/pypi/pgcopy>

I added binary copy to the benchmark:
<https://gist.github.com/altaurog/84668e034646fb354b5de81bb86a580d>

On local machine, with NRECS=100000 and page size=100:
classic: 4.26264309883 sec
joined: 2.34096288681 sec
pgcopy: 0.412513971329 sec

Over network, with NRECS=10000 and page size=100:
classic: 716.759769917 sec
joined: 15.141461134 sec
pgcopy: 3.70594096184 sec