Re: Regarding Postgres Dynamic Shared Memory (DSA)

Hi Thomas,

Thanks for taking time and explaining the things.

Postgres extensions can't rely on backends inheriting the postmaster's

memory map like this (other than the main shared memory areas which

the core code looks after). For one thing, new backends aren't

created with fork() on all platforms (mainly Windows AFAIK, but also

any build with EXEC_BACKEND defined). The next problem is that dsa.c

and dsm.c work with reference counts that will be wrong if you try to

use memory map inheritance like this. Another problem is that the

postmaster isn't allowed to create DSM segment: if it's working for

you then I think you must be building with asserts disabled?

Now I understood, i took wrong approach.

That's simply not how DSA is

designed to be used: you need to create DSA areas in a non-postmaster

backend, and then attach explicitly from every other backend that

wants to access the area. Each backend needs to get its own dsa_area

object (either by creating or attaching).

Initially i tried to design the same way.

I mean, i have created a background worker and created dsa in it.

I tried to attach/detach to the same dsa/dsm by all the backends(postgres clients/connections) during backend(client/connection) init/destroy.

I didn't find any triggers or callbacks during backend init/close to attach/detach the dsa/dsm. Hence, i took this approach.

If postgres have any such triggers/callbacks available, please let me know, that is of great great help for me.

Anyways now i understood, i have taken a wrong approach to use dsa. I'll try to figure out any other way to build my in-memory index over postgres.

Once again thanks a lot for taking time to help me.

@Dilip thank you for your response :)

Thanks & Best Regards,

-Mahi
Teamwork divides the task and multiplies the success.

---- On Wed, 21 Jun 2017 04:26:45 +0530 Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> wrote ----

On Sat, Jun 17, 2017 at 1:17 AM, Mahi Gurram <teckymahi(at)gmail(dot)com> wrote:

>> 3. Whether you are the backend that created it or a backend that

>> attached to it, I think you'll need to store the dsa_area in a global

>> variable for your UDFs to access. Note that the dsa_area object will

>> be different in each backend: there is no point in storing that

>> address itself in shared memory, as you have it, as you certainly

>> can't use it in any other backend. In other words, each backend that

>> attached has its own dsa_area object that it can use to access the

>> common dynamic shared memory area.

>

>

> In case of forked processes, the OS actually does share the pages initially,

> because fork implements copy-on-write semantics. which means that provided

> none of the processes modifies the pages, they both points to same address

> and the same data.

>

> Based on above theory, assume i have created dsa_area object in postmaster

> process(_PG_Init) and is a global variable, all the backends/forked

> processes can able to access/share the same dsa_area object and it's

> members.

>

> Hence theoretically, the code should work with out any issues. But i'm sure

> why it is not working as expected :(

>

> I tried debugging by putting prints, and observed the below things:

> 1. dsa_area_control address is different among postmaster process and

> backends.

> 2. After restarting, they seems to be same and hence it is working after

> that.

Postgres extensions can't rely on backends inheriting the postmaster's

memory map like this (other than the main shared memory areas which

the core code looks after). For one thing, new backends aren't

created with fork() on all platforms (mainly Windows AFAIK, but also

any build with EXEC_BACKEND defined). The next problem is that dsa.c

and dsm.c work with reference counts that will be wrong if you try to

use memory map inheritance like this. Another problem is that the

postmaster isn't allowed to create DSM segment: if it's working for

you then I think you must be building with asserts disabled?

I'm not sure exactly why you're seeing the symptoms you're seeing

(working on one flavour of Unix and not another, and then working

after a crash-restart -- I guess it has something to do with

coincidences of mapped address). That's simply not how DSA is

designed to be used: you need to create DSA areas in a non-postmaster

backend, and then attach explicitly from every other backend that

wants to access the area. Each backend needs to get its own dsa_area

object (either by creating or attaching).

--

Thomas Munro

http://www.enterprisedb.com

--

Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)

To make changes to your subscription:

http://www.postgresql.org/mailpref/pgsql-hackers