| From: | Rafael Chacón <rafael(dot)chacon(at)supabase(dot)io> |
|---|---|
| To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | Table synchronization cannot resume after a subscriber crash |
| Date: | 2026-11-06 16:00:45 |
| Message-ID: | CACX14KDdvL5nftHt8Ga5qB9KgA4dZe-N9GMjMNXo8ejHmCQbqg@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hello!
I'm an engineer working on Multigres at Supabase. While debugging flaky
tests in Multigres, we found what we believe is a bug in PostgreSQL logical
replication.
I reproduced this on PostgreSQL 17.11, built from source on macOS arm64.
After a subscriber restart, a table remains stuck in FINISHEDCOPY
(pg_subscription_rel.srsubstate
= 'f'), while already synchronized tables continue receiving changes.
The table sync worker repeatedly reports:
ERROR: could not start WAL streaming: ERROR: replication slot
"pg_16405_sync_16398_7704351398798576014" does not exist
To reproduce this, I paused the table sync worker immediately after the
publisher confirmed deletion of its synchronization slot, then stopped and
restarted the subscriber. Both servers have fsync=on, and the subscription
uses synchronous_commit=on.
I expected synchronization to resume after the restart, but the table
remains stuck.
Looking at process_syncing_tables_for_sync()
<https://git.postgresql.org/cgit/postgresql.git/tree/src/backend/replication/logical/tablesync.c?h=REL_17_11#n313>,
my understanding is that the worker updates the table state to SYNCDONE,
deletes the synchronization slot on the publisher, and only then commits
the local transaction.
This seems to leave a crash window where the slot has been deleted, but the
SYNCDONE state has not been committed. After a restart, the table is still
in FINISHEDCOPY, but the slot it needs to resume synchronization no longer
exists.
I put together a draft patch that commits and flushes the SYNCDONE state
before deleting the slot. It also allows a restarted worker to complete the
cleanup if interrupted between those steps.
I'm attaching the patch and a test using injection points to reproduce the
issue.
Does this understanding look correct, or is there a recovery path I'm
overlooking? If this is indeed a bug, I'd appreciate some guidance on the
proposed approach. Happy to refine the patch for REL_17_STABLE.
| Attachment | Content-Type | Size |
|---|---|---|
| 03-durable-handoff-fix.patch | application/octet-stream | 9.5 KB |
| 02-crash-test.patch | application/octet-stream | 10.0 KB |
| 01-injection-points.patch | application/octet-stream | 1.5 KB |
| From | Date | Subject | |
|---|---|---|---|
| Previous Message | Robert Haas | 2026-11-06 15:13:30 | Re: WAL_LOG CREATE DATABASE strategy broken for non-standard page layouts |