Re: Weird failure with latches in curculio on v15[

On Sun, Feb 19, 2023 at 08:06:24PM +0530, Robert Haas wrote:
> I mean, my idea was to basically just have one big callback:
> ArchiverModuleMainLoopCB(). Which wouldn't return, or perhaps, would
> only return when archiving was totally caught up and there was nothing
> more to do right now. And then that callback could call functions like
> AreThereAnyMoreFilesIShouldBeArchivingAndIfYesWhatIsTheNextOne(). So
> it would call that function and it would find out about a file and
> start an HTTP session or whatever and then call that function again
> and start another HTTP session for the second file and so on until it
> had as much concurrency as it wanted. And then when it hit the
> concurrency limit, it would wait until at least one HTTP request
> finished. At that point it would call
> HeyEverybodyISuccessfullyArchivedAWalFile(), after which it could
> again ask for the next file and start a request for that one and so on
> and so forth.

This archiving implementation is not completely impossible with the
current API infrastructure, either? If you consider the archiving as
a two-step process where segments are first copied into a cheap,
reliable area. Then these could be pushed in block in a more remote
area like a S3 bucket? Of course this depends on other things like
the cluster structure, but redundancy can be added with standby
archiving, as well.

I am not sure exactly how many requirements we want to push into a
callback, to be honest, and surely more requirements pushed to the
callback increases the odds of implementation mistakes, like a full
loop. There already many ways to get it wrong with archiving, like
missing a flush of the archived segment before the callback returns to
ensure its durability..
--
Michael