[PATCH] Fall back to serializing when parallel apply runs out of DSM segments

Hi,

When the leader apply worker launches a parallel apply worker,
pa_setup_dsm() checks dsm_create() for NULL and returns false, and the
caller then serializes the streamed transaction to a file and applies it
itself. That's what the docs [1] describe for streaming = parallel: "If no
parallel apply worker is free to handle streaming transactions then the
changes are written to temporary files and applied after the transaction
is committed."

But dsm_create() is called with flags = 0, so it never returns NULL. If
no DSM segment is available, it raises "too many dynamic shared memory
segments" instead. The leader exits, restarts, and fails again at the
same STREAM START, so replication makes no progress until segments are
freed:

ERROR: too many dynamic shared memory segments
CONTEXT: processing remote data for replication origin "pg_16394" during
message type "STREAM START" in transaction 665
LOG: background worker "logical replication apply worker" (PID 2514)
exited with exit code 1

I reproduced it by holding all DSM slots on the subscriber from a test
extension, then running a 100k-row transaction on the publisher with
logical_decoding_work_mem = 64kB. On master the apply worker restarted
13 times in a minute with nothing applied. With the patch the leader
serialized the transaction and applied all rows with no error.

The patch passes DSM_CREATE_NULL_IF_MAXSEGMENTS, as
InitializeParallelDSM() does, so the existing fallback is used. The code
dates from 216a784829c and the patch applies to 16 and later.

[1]:
https://www.postgresql.org/docs/devel/sql-createsubscription.html#SQL-CREATESUBSCRIPTION-PARAMS-WITH-STREAMING

Thanks,
Baji Shaik