Re: Deadlock in libpq

On Fri, Mar 25, 2011 at 21:21, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> On Fri, Mar 25, 2011 at 3:26 AM, Erik Hesselink <hesselink(at)gmail(dot)com> wrote:
>>> hm, ISTM (I don't know haskell) that the hdbc driver isn't doing any
>>> type of synchronization at all unless it is using a non thread safe
>>> libpq...and in that case it uses a global mutex.  That doesn't look
>>> correct -- the hdbc driver should be locking around the PGconn always,
>>> and globally if you're stuck with a non thread safe libpq.
>>
>> No, that is not the case. If libpq is not thread safe, the library
>> uses a global lock. If it is thread safe, it uses a single lock per
>> connection. This lock is created on connect, and locked before
>> executing a statement. So it seems the library is doing the correct
>> things.
>>
>> (And yes, libpq is thread safe, I just checked).
>
> hm, I'm stumped.  Are you sure nothing else is using the crypto
> library?  There is an unlikely but possible case that you initialized
> crypto locks over somebody else.

No, I'm not sure at all. We have quite a few dependencies, since
Haskell libraries are very small. I've looked through all available
Haskell libraries and none that we use seem to depend on libcrypto or
libssl. However, it is possible that we transitively depend on
libraries that do. It is also possible that we depend on libraries
that use a C library that uses libcrypto or libssl (like we do with
libpq). It will take some time to analyze the full dependency tree.

> *something* is happening here, but I have no idea what.   I'm very
> skeptical it's a locking issue in libpq itself, because there is so
> little going on beyond counting the connections.  Let me ask you this:
> how much does your connection count range over time?  would it be
> possible to reserve a connection that stays open all the time and see
> if the issue re-occurs? (a wild stab in the dark, but I'm curious of
> ssl re-initialization is causing your problem if you always have at
> least one connection open, it wont re-initialize).

My tests were with two clients. Each client performs database queries
sequentially, and a background thread is also opening database
connections, so that makes 0 to 3 concurrent connections.

I will test with one connection always open; that should be very
simple to set up. I've also set up a connection pooler (pgbouncer) on
our live server, and turned off SSL. This seems to prevent/hide the
problem there at least, but I will continue to investigate on a
different server.

Thanks for all the help so far,

--
Erik Hesselink
http://silkapp.com