Re: speedup COPY TO for partitioned table.

From: Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com>
To: jian he <jian(dot)universality(at)gmail(dot)com>
Cc: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, torikoshia <torikoshia(at)oss(dot)nttdata(dot)com>, Álvaro Herrera <alvherre(at)kurilemu(dot)de>, Kirill Reshke <reshkekirill(at)gmail(dot)com>, vignesh C <vignesh21(at)gmail(dot)com>, David Rowley <dgrowleyml(at)gmail(dot)com>, Melih Mutlu <m(dot)melihmutlu(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: speedup COPY TO for partitioned table.
Date: 2025-10-10 01:02:26
Message-ID: 8316BFA1-D32A-4EDF-A190-A555B98D1EE4@gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

> On Oct 9, 2025, at 22:50, jian he <jian(dot)universality(at)gmail(dot)com> wrote:
>> 3
>> ```
>> + if (RELKIND_HAS_PARTITIONS(relkind))
>> + children = foreach_delete_current(children, childreloid);
>> + }
>> ```
>>
>> I wonder if there is any specially consideration of using RELKIND_HAS_PARTITIONS() here? Because according to the function comment of find_all_inheritors(), it will only return OIDs of relations; while RELKIND_HAS_PARTITIONS checks for both relations and views. Logically using this macro works, but it may lead to some confusion to code readers.
>>
>
> find_all_inheritors comments says:
> * Returns a list of relation OIDs including the given rel plus
> * all relations that inherit from it, directly or indirectly.
>
> CREATE TABLE pp (id int,val int) PARTITION BY RANGE (id);
> CREATE TABLE pp_1 (val int, id int) PARTITION BY RANGE (id);
> ALTER TABLE pp ATTACH PARTITION pp_1 FOR VALUES FROM (1) TO (5);
>
> If we copy partitioned table "pp" data out, but partitioned table "pp_1"
> don't have storage, so we have to skip it, using RELKIND_HAS_PARTITIONS
> to skip it should be fine.

My point is that RELKIND_HAS_PARTITIONS is defined as:

#define RELKIND_HAS_PARTITIONS(relkind) \
((relkind) == RELKIND_PARTITIONED_TABLE || \
(relkind) == RELKIND_PARTITIONED_INDEX)

It just checks relkind to be table or index. The example in your explanation seems to not address my concern. Why do we need to check against index?

>
>> 4
>> ```
>> @@ -722,6 +754,7 @@ BeginCopyTo(ParseState *pstate,
>> DestReceiver *dest;
>>
>> cstate->rel = NULL;
>> + cstate->partitions = NIL;
>> ```
>>
>> Both NULL assignment are not needed as cstate is allocated by palloc0().
>>
> I guess this is just a code convention. Such not necessary is quite common
> within the codebase.

I don’t agree. cstate has a lot of more fields with pointer types, why don’t set NULL to them?

>
>> 5
>> ```
>> +static void
>> +CopyRelTo(CopyToState cstate, Relation rel, Relation root_rel,
>> + uint64 *processed)
>> ```
>>
>> Instead of using a pointer to pass out processed count, I think it’s better to return the process count. I understand the current implementation allows continuous increment while calling this function in a loop. However, it’s a bit error-prone, a caller must make sure “processed” is well initialized. With returning a unit64, the caller’s code is still simple:
>>
>> ```
>> processed += CopyRelTo(cstate, …);
>> ```
>>
> pgstat_progress_update_param was within CopyRelTo.
> so we have to pass (uint64 *processed) to CopyRelTo.
> Am I missing something?
>

Make sense. I didn’t notice postage_progress_update_param. So, “processed” is both input and output. In that case, I think the comment for parameter “processed” should be enhanced, for example:

```
* processed: on entry, contains the current count of processed count;
* this function increments it by the number of rows copied
* from this relation and writes back the updated total.
```

Or a short version:

```
* processed: input/output; cumulative count of tuples processed, incremented here.
```

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Thomas Munro 2025-10-10 01:44:04 Re: Potential deadlock in pgaio_io_wait()
Previous Message Michael Paquier 2025-10-10 00:52:36 Re: Replace O_EXCL with O_TRUNC for creation of state.tmp in SaveSlotToPath