Re: Multithreaded queue in PgSQL

From: valgog <valgog(at)gmail(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Multithreaded queue in PgSQL
Date: 2008-07-15 09:00:57
Message-ID: 677a6e52-a3ff-4c8a-8676-9078dca90e71@34g2000hsf.googlegroups.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

> Whole point is to have multiple services accessing same table and
> dividing the work, so locking with waiting for lock to be released is
> out of question.
>

We are doing the same (newsletter) and there is no problem to lock the
whole table for a short time with an advisory lock as the java id
fetching worker is locking the table (that does not lock the table for
reading or writing, it is only locking his java worker brothers that
are using the same advisory lock), fetches, let's say, 50 id's of
records marked as CREATED and changes their status to PROCESSING. Then
several workers are getting the id's and fetch the needed data from
the table independently and process and update them in parallel. We
have 3 java machines getting id's for 10 parallel workers and
everything works just fine.

Getting the IDs is much much faster usually then real processing.

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Ragnar 2008-07-15 09:28:50 Re: optimizer ignoring primary key and doing sequence scan
Previous Message Craig Ringer 2008-07-15 08:45:13 Re: 8.3.3 Complie issue