Re: relaying errors from background workers

On Thu, May 22, 2014 at 6:33 AM, Petr Jelinek <petr(at)2ndquadrant(dot)com> wrote:
> On 22/05/14 06:21, Robert Haas wrote:
>> The main thing I'm not sure about is how to format the message that we
>> write to the shm_mq. One option is to try to use the good old FEBE
>> protocol. This doesn't look entirely straightforward, because
>> send_message_to_frontend() assembles the message using pq_sendbyte()
>> and pq_sendstring(), and then sends it out to the client using
>> pq_endmessage() and pq_flush(). This infrastructure assumes that the
>> connection to the frontend is a socket. It doesn't seem impossible to
>> kludge that infrastructure to be able to send to either a socket or a
>> shm_mq, but I'm not sure whether it's a good idea. Alternatively, we
>> could devise some other message format specific to this problem; it
>> would probably look a lot like an ErrorData protocol message, but
>> maybe that's doesn't really matter. Any thoughts?
>
> I played with this a bit already and even have some (very much hacked up)
> prototype based on ErrorData as it seemed better solution than using
> sockets. Obviously some of the ErrorData fields don't really have meaning
> during transport (send_to_client for example) as those must be set based on
> the connection options and I didn't get to making this nicer yet.

I didn't mean the ErrorData structure that the backend uses
internally; it doesn't seem practical to use that because it contains
pointers. Even if we stored all the data in the DSM (which seems
rather hard without a shared-memory allocator), it might not be mapped
at the same address in both backends. I meant the libpq wire protocol
used to transmit errors to the client; I guess the message is actually
called ErrorResponse rather than ErrorData:

http://www.postgresql.org/docs/current/static/protocol-message-formats.html

ErrorResponse of course wouldn't contain anything like send_to_client,
as that's a server-internal thing.

>> Another thing to think about is that, most likely, many users of the
>> background worker facility will want to respond to a relayed error by
>> rethrowing it. That means that whatever format we use to send the
>> error from one process to the other has to be able to be decoded by
>> the receiving process. That process will probably want to do
>> something like add add a bit more to the context (e.g. "in background
>> worker PID %d") and throw the resulting error preserving the rest of
>> the original fields. I'm not sure exactly what make sense here, but
>> the point is that ideally the message format should be something that
>> the receiver can rethrow, possibly after tweaking it a bit.
>>
>
> This is one advantage of using ErrorData or ErrorData-like structure based
> messaging, rethrowing is simpler, but I guess if we really needed we could
> provide some conversion api.
>
> One side effect of the rethrowing that maybe deserves a bit of thought is
> that we are going to log the same error from both bgworker and backend if we
> rethrow it.

Yeah, I'm not sure how to handle that. It probably wouldn't be hard
to make the routine that re-throws the error suppress sending it to
the server log, but that could also be confusing if someone is trying
to follow the progress of a particular session. Having the background
worker suppress sending it to the log also seems bad; it has no
guarantee that it will successfully relay the message, or that the
recipient will re-throw it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company