Re: [HACKERS] memory problems in copying large table to STDOUT

Tom Lane wrote on Sat, 09 Oct 1999 11:54:34 EDT
>Martin Weinberg <weinberg(at)osprey(dot)astro(dot)umass(dot)edu> writes:
>> I've been struggling to copy a large table (200 million
>> records, 60GB) to tape using:
>> psql -qc "copy psc to STDOUT;" Winter99 | dd of=/dev/st0 bs=32k
>> After processing about 10 million records (this varies), I
>> get:
>> FATAL 1: Memory exhausted in AllocSetAlloc()
>
>Hmm. What is the exact declaration of the table?
>
>The only explanation I can think of offhand is that the output
>conversion function for one of the column types is leaking memory...
>copy.c itself looks to be pretty careful not to.
>

The table def is:

CREATE TABLE psc (
hemis text,
date date,
scan int2,
id int4,
ra float4,
dec float4,
glon float4,
glat float4,
err_maj float4,
err_min float4,
err_ang int2,
xscan float4,
cnf_flag text,
j_m float4,
h_m float4,
k_m float4,
j_msig float4,
h_msig float4,
k_msig float4,
j_m_psf float4,
h_m_psf float4,
k_m_psf float4,
j_psfchi float4,
h_psfchi float4,
k_psfchi float4,
j_skyval float4,
h_skyval float4,
k_skyval float4,
j_blend int2,
h_blend int2,
k_blend int2,
j_m_stdap float4,
h_m_stdap float4,
k_m_stdap float4,
j_msig_stdap float4,
h_msig_stdap float4,
k_msig_stdap float4,
j_prob_pers float4,
h_prob_pers float4,
k_prob_pers float4,
j_prg_flg text,
h_prg_flg text,
k_prg_flg text,
j_mrg_flg text,
h_mrg_flg text,
k_mrg_flg text,
j_pix_flg text,
h_pix_flg text,
k_pix_flg text,
j_cal float4,
h_cal float4,
k_cal float4,
gal_contam int2,
id_opt text,
dist_opt float4,
b_m_opt float4,
r_m_opt float4,
j_h float4,
h_k float4,
j_k float4,
dup_src int2,
use_src int2,
ext_key_1 int4
);

Thanks for taking a look,

--Martin

===========================================================================

Martin Weinberg Phone: (413) 545-3821
Dept. of Physics and Astronomy FAX: (413) 545-2117/0648
530 Graduate Research Tower weinberg(at)astro(dot)umass(dot)edu
University of Massachusetts http://www.astro.umass.edu/~weinberg/
Amherst, MA 01003-4525