Re: BF mamba failure

We just had the same sudden replica shutdown, this time with version 17.3.

2025-08-02 22:10:02.229 UTC,,,473966,,67b3d76c.73b6e,14,,2025-02-18
00:42:20 UTC,8111/0,0,FATAL,XX000,"trying to drop stats entry already
dropped: kind=relation dboid=16420 objoid=4169049057 refcount=1",,,,,"WAL
redo at 6337E/9FF275E8 for Transaction/COMMIT: 2025-08-02
22:10:02.163756+00; dropped stats: 2/16420/4169049057 2/16420/4169049056
2/16420/4169049053 2/16420/4169049071 2/16420/4169049070 2/16420/4169049068
2/16420/4169049064 2/16420/4169049154 2/16420/4169049153 2/16420/4169049152
2/16420/4169049149; inval msgs: catcache 80 c
atcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 cat
cache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 32 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache 79
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache
54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 ca
tcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache
6 catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache 55
catcache 54 catcache 55 catcache 54 catcache 7 catcache 6 catcache 32
catcache 55 catcache 54 catcache 55 catcache 54 cat
cache 32 catcache 7 catcache 6 catcache 55 catcache 54 catcache 32 catcache
7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 cat
cache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 cat
cache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache
80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 c
atcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catca
che 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 32
catcache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 55 catcache 54 catcache 7 catcache 6 catcache 32 catcache 55
catcache 54 catcache 55 catcache 54 catcache 32 catca
che 7 catcache 6 catcache 55 catcache 54 catcache 32 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcac
he 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 80 catcache 79 catcache 80 catcache 79 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcac
he 6 catcache 7 catcache 6 catcache 55 catcache 54 snapshot 2608 relcache
4169049053 snapshot 1214 relcache 4169049056 relcache 4169049057 relcache
4169049056 snapshot 2608 relcache 4169049056 relcache 4169049057 relcache
4169049053 snapshot 2608 relcache 4169049057 rel
cache 4169049056 snapshot 2608 relcache 4169049056 snapshot 2608 snapshot
2608 snapshot 2608 relcache 4169049053 snapshot 2608 snapshot 1214 snapshot
2608 relcache 4169049064 snapshot 1214 relcache 4169049068 relcache
4169049070 relcache 4169049068 snapshot 2608 relcach
e 4169049068 relcache 4169049070 relcache 4169049064 snapshot 2608 relcache
4169049071 relcache 4169049064 snapshot 2608 relcache 4169049064 relcache
4169049071 relcache 4169049071 relcache 4169049064 snapshot 2608 relcache
4169049070 relcache 4169049068 snapshot 2608 r
elcache 4169049068 snapshot 2608 snapshot 2608 snapshot 2608 relcache
4169049064 snapshot 2608 snapshot 1214 snapshot 2608 relcache 4169049149
snapshot 1214 relcache 4169049152 relcache 4169049153 relcache 4169049152
snapshot 2608 relcache 4169049152 relcache 4169049153
relcache 4169049149 snapshot 2608 relcache 4169049154 relcache 4169049149
snapshot 2608 relcache 4169049149 relcache 4169049154 relcache 4169049154
relcache 4169049149 snapshot 2608 relcache 4169049153 relcache 4169049152
snapshot 2608 relcache 4169049152 snapshot 2608
snapshot 2608 snapshot 2608 relcache 4169049149 snapshot 2608 snapshot
1214",,,,"","startup",,0
2025-08-02 22:10:02.389 UTC,,,473962,,67b3d76b.73b6a,27,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"startup process (PID 473966) exited with exit
code 1",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.389 UTC,,,473962,,67b3d76b.73b6a,28,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"terminating any other active server
processes",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.516 UTC,,,473962,,67b3d76b.73b6a,29,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"shutting down due to startup process
failure",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.563 UTC,,,473962,,67b3d76b.73b6a,30,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"database system is shut
down",,,,,,,,,"","postmaster",,0

На пт, 15.11.2024 г. в 6:45 Michael Paquier <michael(at)paquier(dot)xyz> написа:

> On Thu, Nov 14, 2024 at 10:07:41AM +0000, Bertrand Drouvot wrote:
> > === 1
> >
> > Maybe use "generation" instead of generation in the comments (where it's
> not done,
> > some comments do it already).
>
> I've tweaked things to be more consistency, and applied that down to 15.
>
> > === 2
> >
> > We could think about adding a test. That should be doable with
> replication slots
> > or custom pgstats kinds and probably injection points. But I'm not sure
> that's
> > worth it, as the code in the patch looks "simple" enough. Thoughts?
>
> The tricky part is that this path involves a backend shutdown. We
> should be able to do something with a wait before the dshash_find() in
> pgstat_release_entry_ref() with a BackgroundPsql object that gets
> stopped, but I'm not sure if we can guarantee a lookup of
> pg_stat_activity at this stage. Let me see..
> --
> Michael
>