Re: Review: Extra Daemons / bgworker

Alvaro,

in general, please keep in mind that there are two aspects that I tend
to mix and confuse: one is what's implemented and working for
Postgres-R. The other this is how I envision (parts of it) to be merged
back into Postgres, itself. Sorry if that causes confusion.

On 12/03/2012 04:43 PM, Alvaro Herrera wrote:
> Oh, I understand that. My question was more about what mechanism are
> you envisioning for new bgworkers. What process would be in charge of
> sending such signals? Would it be a persistent bgworker which would
> decide to start up other bgworkers based on external conditions such as
> timing? And how would postmaster know which bgworker to start -- would
> it use the bgworker's name to find it in the registered workers list?

Well, in the Postgres-R case, I've extended the autovacuum launcher to a
more general purpose job scheduler, named coordinator. Lacking your
bgworker patch, it is the only process that is able to launch background
workers.

Your work now allows extensions to register background workers. If I
merge the two concepts, I can easily imagine allowing other (bgworker)
processes to launch bgworkers.

Another thing I can imagine is turning that coordinator into something
that can schedule and trigger jobs registered by extensions (or even
user defined ones). Think: cron daemon for SQL jobs in the background.
(After all, that's pretty close to what the autovacuum launcher does,
already.)

> The trouble with the above rough sketch is how does the coordinating
> bgworker communicate with postmaster.

I know. I've gone through that pain.

In Postgres-R, I've solved this with imessages (which was the primary
reason for rejection of the bgworker patches back in 2010).

The postmaster only needs to starts *a* background worker process. That
process itself then has to figure out (by checking its imessage queue)
what job it needs to perform. Or if you think in terms of bgworker
types: what type of bgworker it needs to be.

> Autovacuum has a very, um,
> peculiar mechanism to make this work: avlauncher sends a signal (which
> has a hardcoded-in-core signal number) and postmaster knows to start a
> generic avworker; previously avlauncher has set things up in shared
> memory, and the generic avworker knows where to look to morph into the
> specific bgworker required. So postmaster never really looks at shared
> memory other than the signal number (which is the only use of shmem in
> postmaster that's acceptable, because it's been carefully coded to be
> robust).

In Postgres-R, I've extended exactly that mechanism to work for other
jobs that autovacuum.

> This doesn't work for generic modules because we don't have a
> hardcoded signal number; if two modules wanted to start up generic
> bgworkers, how would postmaster know which worker-main function to call?

You've just described above how this works for autovacuum: the
postmaster doesn't *need* to know. Leave it to the bgworker to determine
what kind of task it needs to perform.

> As posted, the feature is already useful and it'd be good to have it
> committed soon so that others can experiment with whatever sample
> bgworkers they see fit. That will give us more insight on other API
> refinements we might need.

I completely agree. I didn't ever intend to provide an alternative patch
or hold you back. (Except for the extra daemon issue, where we disagree,
but that's not a reason to keep this feature back). So please, go ahead
and commit this feature (once the issues I've mentioned in the review
are fixed).

Please consider all of these plans or ideas in here (or in Postgres-R)
as extending on your work, rather than competing against.

> I'm going to disappear on paternity leave, most likely later this week,
> or early next week; I would like to commit this patch before that. When
> I'm back we can discuss other improvements. That will give us several
> weeks before the end of the 9.3 development period to get these in. Of
> course, other committers are welcome to improve the code in my absence.

Okay, thanks for sharing that. I'd certainly appreciate your inputs on
further refinements for bgworkers and/or autovacuum.

Regards

Markus Wanner