Re: bg worker: overview

Hey Dimitri,

On 07/24/2010 07:26 PM, Dimitri Fontaine wrote:
> Trying to figure out how it would fit the PGQ and pgagent needs. But
> maybe user defined daemons should be sub-coordinators (I used to think
> about them as "supervisors") able to talk to the coordinator to get a
> backend connected to some given database and distribute work to it.

Hm.. sounds like an awful lot of work to me, but if you need the
separation and security of a separate process...

To simplify, you might want to start a bgworker on database 'postgres',
which then acts as a sub-coordinator (and doesn't really need to use its
database connection).

> You're using iMessage as the data exchange, how are you doing the work
> distribution? What do you use to tell the backend what is the processing
> you're interrested into?

Well, there are different types of imessages defined in imsg.h. If you
are coding something within Postgres, you'd just add all the required
messages types there. There's no such thing as an external registration
for new message types.

For example, for autovacuum, there are two message types:
IMSGT_PERFORM_VACUUM, that's sent from the coordinator to a bgworker,
and initiates a vacuum job there. Then there's IMSGT_FORCE_VACUUM, which
is sent from a backend to the coordinator to inform it that a certain
database urgently needs vacuuming.

For Postgres-R, things are a bit more complicated. The first IMSGT_CSET
messages starts the application of a remote transaction. Further
IMSGT_CSET messages may follow. The IMSGT_ORDERING message finally
completes the job.

So, imessage types cannot be mapped to jobs directly. See
include/postmaster/coordinator.h, enum worker_state. Those are the
possible states a worker can be in (job types).

Adding a job would consist of adding a worker_state, plus at least one
imessage type. Once the worker is done with its job, it returns
IMSGT_READY to the coordinator.

I'm open to refinements, such as assigning a certain range of message
types to external use or some such. However, have no idea how to avoid
clashing message type ids, then. Maybe those should still be part of imsg.h?

>> Go ahead, re-use the background workers. That's what I've published
>> them for
>
> Hehe :) The aim of this thread would be to have your input as far as
> designing an API would go, now that we're about on track as to what the
> aim is.

Oh, sure. :-)

> In very short, the idea is a clock that ticks and associate
> current_txid() to now(), so that you're able to say "give me 3s worth of
> transactions activity from this queue". It then provides facilities to
> organise a queue into batches at consumer request, and for more details,
> see there:
>
> http://github.com/markokr/skytools-dev/blob/master/sql/ticker/pgqd.c
> http://github.com/markokr/skytools-dev/blob/master/sql/ticker/ticker.c

Okay, thanks for the pointers. However, comments are relatively sparse
in there as well...

> But the important thing as far as making it a child of the coordinator
> goes would be, I guess, that it's some C code running as a deamon and
> running SQL queries from time to time. The SQL queries are calling C
> user defined functions, provided by the PGQ backend module.

You could certainly define jobs, which don't ever terminate. And calling
SQL queries certainly sounds more like a background job to me, than
something belonging to the sphere of the coordinator. Sorry, if my first
impulse has been misleading.

So, the bgworker infrastructure could probably satisfy the internal
communication needs. But how does this ticker daemon talk to the
outside? Does it need to open a socket and listen there? Or do the
requests to that queue come in via SQL?

Regards

Markus