Re: Orphaned files in base/[oid]

On 14.08.2017 23:56, Andres Freund wrote:
> Alternatively we could do something without marker files, with some
> added complexity: Keep track of all "uncommitted new files" in memory,
> and log them every checkpoint. Commit/abort records clear elements of
> that list. Since we always start replay at the beginning of a
> checkpoint, we'd always reach a moment with such an up2date list of
> pending-action files before reaching end-of-recovery. At end-of-recovery
> we can delete all unconfirmed files. To avoid out-of-memory due to too
> many tracked relations, we'd possibly still have to have marker files...
>

Hi, hackers.

I'm sorry, but I want to bump this thread, because there is still no
good solution to solve the problem. I see there are few threads with
undo-based approaches, which looks preferable, but have some pitfalls.
Is there any chance to return to non-undo approaches partially discussed
here? What do you think about the following solutions?
1) Make `pendingDeletes` shared and let postmaster clean all garbage in
case of child process dying. Cons: Not works in case of postmaster
dying. Should care about `pendingDeletes` pointers validity.
2) Catch and store all records with relfilenode during WAL replay,
delete all orphaned nodes at the end of replaying. Cons: The final
delete may use an incomplete list of nodes, as there was something
before the latest checkpoint. The general opacity - we remove something
without a corresponded WAL record (or possibly do it in wrong place in
general).
3) This way is close to one I quoted and a combination of two above.
`pendingDeletes` is shared. Each checkpoint creates a WAL record with a
list of open transactions and created nodes. WAL replaying can use this
list as base, adding nodes to it from newer records. The final delete
operation has a complete list of orphaned nodes. Cons: Complexity(?).
Others(?).

Can it work? Are any of this approaches still relevant?

I'm sorry for the last html-formatted message. Our web-based app is too
smart.

--
Regards,
Alex Go, C developer
goa(at)arenadata(dot)io, www.arenadata.tech