Re: [ERROR] Copy from CSV fails due to memory error.

On 1/19/22 14:01, Kostas Chasialis wrote:
> Hey.
>
> I am facing an issue when I try to run the following command
>
> COPY <table_name> FROM <file> WITH DELIMITER E',’;
>
> This file, is rather large, it's around 178GBs.
>
> When I try to run this COPY command I get the following error:
>
> ERROR: out of memory
> DETAIL: Failed on request of size 2048 in memory context "AfterTriggerEvents".
> CONTEXT: COPY ssbm300_lineorder, line 50796791
>
> Clearly a memory allocation function is failing but I have no clue how to fix it.
>
> I have tried experimenting with shared_buffers value in postgresql.conf file but after searching a bit I quickly realized that I do not know what I am doing there so I left it with default value. Same with work_mem value.
>
> Did you face this issue before? Can you help me resolve it?
>

Well, it's clearly related to "after" triggers - do you have anything
such triggers on the table? AFAIK it might be related to deferred
constraints (like unique / foreign keys). Do you have anything like that?

If yes, I guess the only solution is to make the constraints not
deferred or split the copy into smaller chunks.

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company