Re: archive status ".ready" files may be created too early

From: "alvherre(at)alvh(dot)no-ip(dot)org" <alvherre(at)alvh(dot)no-ip(dot)org>
To: "Bossart, Nathan" <bossartn(at)amazon(dot)com>
Cc: Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>, "x4mmm(at)yandex-team(dot)ru" <x4mmm(at)yandex-team(dot)ru>, "a(dot)lubennikova(at)postgrespro(dot)ru" <a(dot)lubennikova(at)postgrespro(dot)ru>, "hlinnaka(at)iki(dot)fi" <hlinnaka(at)iki(dot)fi>, "matsumura(dot)ryo(at)fujitsu(dot)com" <matsumura(dot)ryo(at)fujitsu(dot)com>, "masao(dot)fujii(at)gmail(dot)com" <masao(dot)fujii(at)gmail(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: archive status ".ready" files may be created too early
Date: 2021-08-20 14:50:18
Message-ID: 202108201450.6aao43wd3ju3@alvherre.pgsql
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Two things.

1. We use a hash table in shared memory. That's great. The part that's
not so great is that in both places where we read items from it, we
have to iterate in some way. This seems a bit silly. An array would
serve us better, if only we could expand it as needed. However, in
shared memory we can't do that. (I think the list of elements we
need to memoize is arbitrary long, if enough processes can be writing
WAL at the same time.)

Now that I think about this again, maybe it's limited by
NUM_XLOGINSERT_LOCKS, since there can only be that many records being
written down at a time ...

2. There is a new LWLock acquisition that may be a new contention point.
We acquire the lock in these cases:
- WAL record insert, when a segment boundary is crosses (rare
enough).
- XLogWrite, when a segment needs to be notified.

Looking again, I think the last point might be a problem actually,
because XLogWrite is called with WALWriteLock held. Maybe we should
take the NotifySegmentsReadyForArchive() call outside the section
locked by WALWriteLock (so put it in XLogWrite callers instead of
XLogWrite itself).

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Cómo ponemos nuestros dedos en la arcilla del otro. Eso es la amistad; jugar
al alfarero y ver qué formas se pueden sacar del otro" (C. Halloway en
La Feria de las Tinieblas, R. Bradbury)

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Geoghegan 2021-08-20 15:06:20 Re: The Free Space Map: Problems and Opportunities
Previous Message Robert Haas 2021-08-20 14:45:13 Re: The Free Space Map: Problems and Opportunities