Re: Proposal: Limitations of palloc inside checkpointer

Hi Alexander,

Thanks a lot for reviewing!

> I have few notes about that:
> 1) Should we make CompactCheckpointerRequestQueue() process the queue
> of checkpoint requests in smaller parts for the same reason we do this
> in AbsorbSyncRequests()? That would require significant redesign of
> the algorithm, but still.

In AbsorbSyncRequests, we process requests incrementally in batches to
avoid allocating more than 1 GB of memory, which would lead to
repeated failure. I think this is less concerning in
CompactCheckpointerRequestQueue, because if we caps num_requests at 10
million, the hash table peaks at ~500 MB and skip_slot[] at ~10
MB—both under 1 GB.

> 2) That's pretty independent to the changes by the patch, but should
> CompactCheckpointerRequestQueue() fill the gaps with entries from the
> tail instead of rewriting the whole queue? That might be a bit
> faster.
This optimization would be quite helpful for compacting large queues.
For small ones, it may also add extra costs. Can we use a hybrid
approach? If it's independent, should we create a standalone patch for
it?

> 3) For sure, we wouldn't backpatch this. Can we prepare some simple
> solution for back branches? Perhaps, just introduction of
> MAX_CHECKPOINT_REQUESTS is enough to save us from allocations larger
> than 1GB.
>
I think this would work well for back branches.