Re: Orphaned Files in PostgreSQL

On Wednesday, September 23rd, 2026 at 8:39 AM, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:

> Hi Ashutosh,
>
> On 23 Sep 2026, Ashutosh Sharma wrote:
> > Please take a look and let me know.

Hey Ashutosh!

Thanks for the excellent email kicking off this thread, I couldn't agree
more about the issue. I started in a different place, asking myself if
I could resurrect UNDO from ZHEAP without modifying HEAP at all (because
I tried and it didn't help, other different table AMs might find benefit
but HEAP is rather solid as it is) and if I could how would I demonstrate
and justify it without adding a new table AM or modifying HEAP?

I used to work on Berkeley DB and one of the features was its ability to
WAL log UNDO/REDO records for "filesystem operations" and then during
recovery tidy up and make things consistent. That struck me as a solid
first application of UNDO in Postgres, so I created that and called it
FILEOPS (because I'm not creative at all).

> At the design level, one manifest per XID still means a pg_fsync() for
> each appended record, including during redo. Have you considered WAL
> plus delayed manifest synchronization, along the lines Andres
> suggested [0]? It would be useful to compare small-DDL and replay
> costs before settling on synchronous per-record writes.
>
> From reading v2, I am concerned about mapped catalog rewrites.
> write_relmap_file() flushes XLOG_RELMAP_UPDATE before calling
> RelationPreserveStorage(), where the patch now records PRESERVE.
> A crash between those steps leaves the new mapping durable, but the
> creating transaction uncommitted and its manifest without PRESERVE.
> relmap_redo() does not preserve the storage either. Wouldn't the new
> end-of-recovery cleanup then remove files needed by the mapped catalog?
>
> Could preservation be part of the relmap update's recovery semantics?
> A crash test in that window during VACUUM FULL of a mapped catalog
> seems particularly important. I haven't run that reproducer yet.
>
> The truncated-manifest test expects startup to fail. Can a crash during
> a normal append leave that state without replay repairing it? If so,
> could we retain the uncertain files rather than refuse startup?
>
> Also, Greg recently mentioned renewed UNDO/FILEOPS work [1]. It may
> be worth coordinating the scope with him. Preventing new orphans and
> handling existing ones, as needed for online checksums, are separate
> parts of the problem.

I'm spending a lot of time this week cleaning up the UNDO and FILEOPS
patches in hopes of posting them soon as either an RFC or a proposed
patch set. I do have new table AMs that use it, but I don't think
they are ready for prime time yet. There are other use cases for UNDO
also like BLOB/CLOBs etc. that I think might be interesting and if I
can get the integration with nbtree and hash correct there are also
benefits for indexes.

That said, it's a large change and one that has philosophical and
technical challenges before the community could even consider merging
it in. I have hope, but it'll be a long road.

Your approach has less overhead/history to deal with. I'll need to
dig into it more to appreciate the direction you've taken but I do
agree that it needs to happen somehow and so I don't see this as a
competing idea at all.

best.

-greg

> Thank you!
>
>
> Best regards, Andrey Borodin.
>
> [0] https://postgr.es/m/20170814185632.zodm5qykgss7ud32@alap3.anarazel.de
> [1] https://postgr.es/m/5d89549c-117e-45ae-b934-a2bb71c82a79@app.fastmail.com