Wait for parallel workers to attach

From: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
To: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Cc: Peter Geoghegan <pg(at)bowt(dot)ie>, Robert Haas <robertmhaas(at)gmail(dot)com>, Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>
Subject: Wait for parallel workers to attach
Date: 2018-01-27 08:14:41
Message-ID: CAA4eK1+e2MzyouF5bg=OtyhDSX+=Ao=3htN=T-r_6s3gCtKFiw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

During the recent development of parallel operation (parallel create
index)[1], a need has been arised for $SUBJECT. The idea is to allow
leader backend to rely on number of workers that are successfully
started. This API allows leader to wait for all the workers to start
or fail even if one of the workers fails to attach. We consider
workers started/attached once they are attached to error queue. This
will ensure that any error after the workers are attached won't be
silently ignored by leader.

I have used wait event as WAIT_EVENT_BGWORKER_STARTUP similar to
WaitForReplicationWorkerAttach, but we might want to change it.

I have tested this patch by calling this API in nodeGather.c and then
introducing failuires at various places: (a) induce fork failure for
background workers (force_fork_failure_v1.patch), (b) Exit parallel
worker before attaching to the error queue
(exit_parallel_worker_before_error_queue_attach_v1.patch) and (c) Exit
parallel worker after attaching to the error queue
(exit_parallel_worker_after_error_queue_attach_v1.patch).

In all above cases, I got the errors as expected.

[1] - https://www.postgresql.org/message-id/CAA4eK1KgmdT3ivm1vG%2BrJzKOKeYQU2XLhp6ma5LMHxaG89brsA%40mail.gmail.com

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Attachment Content-Type Size
WaitForParallelWorkersToAttach_v1.patch application/octet-stream 4.0 KB
modify_gather_to_wait_for_attach_v1.patch application/octet-stream 475 bytes
force_fork_failure_v1.patch application/octet-stream 510 bytes
exit_parallel_worker_before_error_queue_attach_v1.patch application/octet-stream 557 bytes
exit_parallel_worker_after_error_queue_attach_v1.patch application/octet-stream 539 bytes

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Kapila 2018-01-27 08:20:25 Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)
Previous Message Erik Rijkers 2018-01-27 07:40:58 Re: Add RANGE with values and exclusions clauses to the Window Functions