COPY TO STDOUT Apache Arrow support

From: Adam Lippai <adam(at)rigo(dot)sk>
To: pgsql-hackers(at)postgresql(dot)org
Subject: COPY TO STDOUT Apache Arrow support
Date: 2022-04-21 14:41:17
Message-ID: CAGrfaBVyfm0wPzXVqm0=h5uArYh9N_ij+sVpUtDHqkB=VyB3jw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

would it be possible to add Apache Arrow streaming format to the copy
backend + frontend?
The use case is fetching (or storing) tens or hundreds of millions of rows
for client side data science purposes (Pandas, Apache Arrow compute
kernels, Parquet conversion etc). It looks like the serialization overhead
when using the postgresql wire format can be significant.

Best regards,
Adam Lippai

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2022-04-21 17:16:54 Re: typos
Previous Message David G. Johnston 2022-04-21 14:39:17 Re: Add version and data directory to initdb output