Re: small parallel restore optimization

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: Magnus Hagander <magnus(at)hagander(dot)net>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: small parallel restore optimization
Date: 2009-03-09 15:46:19
Message-ID: 49B539CB.3040703@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Magnus Hagander wrote:
> Andrew Dunstan wrote:
>
>> Alvaro Herrera wrote:
>>
>>> Andrew Dunstan wrote:
>>>
>>>
>>>
>>>> I have found the source of the problem I saw. dumputils.c:fmtId()
>>>> uses a static PQExpBuffer which it initialises the first time it's
>>>> called. This gets clobbered by simultaneous calls by Windows threads.
>>>>
>>>> I could just make it auto and set it up on each call, but that could
>>>> result in a non-trivial memory leak ... it's probably called a great
>>>> many times. Or I could provide a parallel version where we pass in a
>>>> PQExpBuffer that we create, one per thread, and is used by anything
>>>> called by the parallel code. That seems like a bit of a potential
>>>> footgun, though.
>>>>
>>>>
>>> Could you use a different static PQExpBuffer on each thread?
>>> pthread_getspecific sort of thing, only to be used on Windows.
>>>
>>>
>> For MSVC we could declare it with "_declspec(thread)" and it would be
>> allocated in thread-local storage, but it looks like this isn't
>> supported on Mingw.
>>
>
> Yeah, _declspec(thread) would be the easy way to do it. But you can also
> use the TlsAlloc(), TlsSetValue() and friends functions directly. We do
> this to use TLS in ecpg.
>
> It requires some coding around, but for ecpg we did a macro that makes
> it behave like the posix functions (see
> src/interfaces/ecpg/include/ecpg-pthread-win32.h)
>
>
>

Yeah, we'll just have to call TlsAlloc to set the thread handle
somewhere near program start, but that shouldn't be too intrusive.

cheers

andrew

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Umar Farooq 2009-03-09 16:39:47 PostgreSQL Memory Management
Previous Message Greg Sabino Mullane 2009-03-09 15:22:47 One less footgun: deprecating pg_dump -d