Re: pl/python long-lived allocations in datum->dict transformation

On 12/02/12 00:48, Tom Lane wrote:
> =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= <wulczer(at)wulczer(dot)org> writes:
>> This is annoying for functions that plough through large tables, doing
>> some calculation. Attached is a patch that does the conversion of
>> PostgreSQL Datums into Python dict objects in a scratch memory context
>> that gets reset every time.
>
> As best I can tell, this patch proposes creating a new, separate context
> (chewing up 8KB+) for every plpython procedure that's ever used in a
> given session. This cure could easily be worse than the disease as far

Yeah, that's not ideal.

> What's more, it's unclear that
> it won't malfunction altogether if the function is used recursively
> (ie, what if PLyDict_FromTuple ends up calling the same function again?)

I was a bit worried about that, but the only place where
PLyDict_FromTuple calls into some other code is when it calls the type's
specific I/O function, which AFAICT can't call back into user code
(except for user-defined C I/O routines). It's not very comfortable, but
I think PLyDict_FromTuple can be allowed to be non-reentrant.

> Can't you fix it so that the temp context is associated with a
> particular function execution, rather than being "statically" allocated
> per-function?

That would be cool, but I failed to easily get a handle on something
that's like the execution context of a PL/Python function... Actually,
if we assume that PLyDict_FromTuple (which is quite a low-level thing)
never calls PL/Python UDFs we could keep a single memory context in
top-level PL/Python memory and pay the overhead once in a session, not
once per function.

OTOH if we want to make it reentrant, some more tinkering would be in order.

Cheers,
Jan