From: | Magnus Hagander <magnus(at)hagander(dot)net> |
---|---|
To: | Andres Freund <andres(at)anarazel(dot)de> |
Cc: | PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Daniel Gustafsson <daniel(at)yesql(dot)se> |
Subject: | Re: Online enabling of checksums |
Date: | 2018-02-22 20:28:31 |
Message-ID: | CABUevEwswbOb4+tPaS02AqKmy7mj2DP+i4GDxYU9Vt12-t=kXg@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Thu, Feb 22, 2018 at 9:23 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi,
>
> On 2018-02-22 21:16:02 +0100, Magnus Hagander wrote:
> > You could do that, but then you've moving the complexity to managing that
> > list in shared memory instead.
>
> Maybe I'm missing something, but how are you going to get quick parallel
> processing if you don't have a shmem piece? You can't assign one
> database per worker because commonly there's only one database. You
> don't want to start/stop a worker for each relation because that'd be
> extremely slow for databases with a lot of tables. Without shmem you
> can't pass more than an oid to a bgworker. To me the combination of
> these things imply that you need some other synchronization mechanism
> *anyway*.
>
Yes, you probably need something like that if you want to be able to
parallelize on things inside each database. If you are OK parallelizing
things on a per-database level, you don't need it.
> > I'm not sure that's any easier... And
> > certainly adding a catalog flag for a usecase like this one is not making
> > it easier.
>
> Hm, I imagined you'd need that anyway. Imagine a 10TB database that's
> online converted to checksums. I assume you'd not want to reread 9TB if
> you crash after processing most of the cluster already?
>
I would prefer that yes. But having to re-read 9TB is still significantly
better than not being able to turn on checksums at all (state today). And
adding a catalog column for it will carry the cost of the migration
*forever*, both for clusters that never have checksums and those that had
it from the beginning.
Accepting that the process will start over (but only read, not re-write,
the blocks that have already been processed) in case of a crash does
significantly simplify the process, and reduce the long-term cost of it in
the form of entries in the catalogs. Since this is a on-time operation (or
for many people, a zero-time operation), paying that cost that one time is
probably better than paying a much smaller cost but constantly.
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>
From | Date | Subject | |
---|---|---|---|
Next Message | Andres Freund | 2018-02-22 20:30:26 | Re: Allow workers to override datallowconn |
Previous Message | Tom Lane | 2018-02-22 20:24:50 | Re: Allow workers to override datallowconn |