From: nilsocket <nilsocket(at)gmail(dot)com>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject:
Date: 2019-09-16 05:41:31
Message-ID: CAGkTAxs_+cs8OCu2pO7ns-NnsuC8DsNDSBBEiNnfhyhBANV9wA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

We want to export data from PG to Kafka,
We can't rely on extension which we have written as there could be any
problems which we are not aware of and PG might break.
We don't want our master to go down because of the extension we have
written.

So, we are okay with having a new PG instance whose work is just to export
data, as slave.

What we thought of doing is to pause recovery (start-up) process, on any
vacuum changes on system catalog tables and resume once, our
logical-decoding has caught up.
That way, we aren't bloating our master.

Our problem is, we aren't able to exactly identify what WAL records are
causing Vacuum changes, as far as our understanding goes, `HEAP_2 (rmgr),
CLEAN` and `BTREE (rmgr), VACUUM` records.

Inorder to see our understanding is right or not, IPC (inter-process
communication) between WAL_SENDER and START_UP process, is not efficient
enough.

I have seen Latches, but I'm not sure exactly how to use them, as from
comments, my understanding is START_UP process is not available in PGPROC
array.

For efficient inter-process communication, what would be ideal way to
communicate between two processes.

I'm new to PostgreSQL, and C - world.

What we are trying to achieve is something similar to this:

START_UP process goes to sleep, as soon as it sees any vacuum on catalog.
WAL_SENDER process will resume recovery (wakeup), as soon as it caught up
and goes to sleep.
START_UP process will wake up when THERE is something for WAL_SENDER to
send.

Basically, IPC, between two processes, where one process generates work,
and other consumes it.
producer should go to sleep, until consumer caught up.
consumer should signal producer of it's completion and wake up producer.
cycle goes on...

As I have indicated before, I new to C and PostgreSQL, I familiar with
GoLang, and I have written a sample program in Go (attached below).

Any suggestions and pointers would be greatly helpful.

--
Thank you

Responses

  • Re: at 2019-09-16 05:53:33 from nilsocket

Browse pgsql-hackers by date

  From Date Subject
Next Message Thomas Munro 2019-09-16 05:52:25 Re: POC: Cleaning up orphaned files using undo logs
Previous Message Tom Lane 2019-09-16 04:24:44 Re: Leakproofness of texteq()/textne()