Re: Add a test for index_rebuild_count of REPACK (CONCURRENTLY)

From: Manu <manuelreyesbravo(at)gmail(dot)com>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Cc: Álvaro Herrera <alvherre(at)kurilemu(dot)de>, Michael Paquier <michael(at)paquier(dot)xyz>, Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com>
Subject: Re: Add a test for index_rebuild_count of REPACK (CONCURRENTLY)
Date: 2026-09-23 18:29:45
Message-ID: 179018818545.3031865.5809261654654798774@gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

The CFBot still could not apply the series, and this time it was the
mail, not the tree: the attachments of my last message went out as
text/plain without base64, and on the way the first line of each was
escaped from "From <sha1>" to ">From <sha1>", which git am does not
recognize ("Patch format detection failed" in the CFBot log). They
are base64-encoded this time.

v3, rebased on master (374522aa63a), changes two things.

0005 now runs every command that reports progress, which Álvaro asked
for. New since v2:

- REINDEX TABLE and REINDEX INDEX CONCURRENTLY
- CREATE INDEX on a partitioned table (partitions_total/done)
- ANALYZE of a partitioned table with extended statistics
- COPY FROM with WHERE and ON_ERROR ignore (tuples_excluded/skipped)
- VACUUM FULL
- disabling and enabling data checksums online, launcher and workers

Data checksums needed one addition to the rules: their counters start
at -1, which the view shows as NULL, so a counter may now go to and
come back from a command's NULL value.

The new cases earn their keep on the old bugs, too. With 0001-0004
reverted, the checker used to fail 10 checks; it now fails 14, the new
ones in REINDEX TABLE, REINDEX CONCURRENTLY, CREATE INDEX on a
partitioned table and VACUUM FULL, all of them the same unreset
counters.

0006 changes in three ways:

- It no longer compares the order of the phase tables in
monitoring.sgml with the order of execution. Sami made the point in
the REPACK phases thread [1] that the documented order is not a
guarantee, and 001 already checks the actual succession of phases.
What remains is the drift check: every phase a command reports must
be named by the view and listed in the documentation.
- That check used to run only when it found a problem, so a clean run
passed without testing anything; it is now one test per command.
- 002_doc_phases.pl only ran when PROGRESS_DEBUG_BUILD=1 was set in the
environment, and nothing sets it, so on a server compiled with
-DPROGRESS_DEBUG -- the buildfarm animal Álvaro suggested -- it would
have been skipped every time. It now detects the build the same way
001_progress.pl does; that check moved into ProgressCheck.pm as
compiled_with_progress_debug().

0001-0004 are unchanged apart from the rebase.

Tested with meson:

- with -DPROGRESS_DEBUG: 001 passes 80 subtests (53 in v2), and 002
passes 7;
- 0005 alone passes the same 80;
- 002 fails, naming the phase, when a row of a phase table is renamed;
- without PROGRESS_DEBUG: 001 only checks progress.h against its
description, and 002 skips, saying why;
- the main regression suite passes, with no compiler warnings.

Regards,
Manu

[1]
https://postgr.es/m/CAN12+YKo-vjvdPQts6QnHoB3ET5A2or137oN_FGMviQdtYLN6Q@mail.gmail.com

Attachment Content-Type Size
v3-0001-Fix-blocks_done-of-an-index-build-s-heap-scan-on-.patch text/x-patch 1.6 KB
v3-0002-Reset-the-index-build-progress-counters-for-every.patch text/x-patch 2.8 KB
v3-0003-Reset-VACUUM-s-dead-item-progress-counters-after-.patch text/x-patch 1.8 KB
v3-0004-Don-t-count-the-last-key-twice-in-a-parallel-GIN-.patch text/x-patch 1.6 KB
v3-0005-Add-PROGRESS_DEBUG-to-test-the-whole-sequence-of-.patch text/x-patch 55.2 KB
v3-0006-Check-the-documented-progress-phases-against-the-.patch text/x-patch 14.0 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Sami Imseih 2026-09-23 18:41:13 Re: REPACK (CONCURRENTLY) loses missing values of columns added without a rewrite
Previous Message Masahiko Sawada 2026-09-23 18:27:15 Re: REPACK (CONCURRENTLY) can silently lose updates when the toast table is rewritten