"long" type is not appropriate for counting tuples

From: Peter Geoghegan <pg(at)bowt(dot)ie>
To: Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: "long" type is not appropriate for counting tuples
Date: 2019-04-28 22:30:39
Message-ID: CAH2-Wz=WbNxc5ob5NJ9yqo2RMJ0q4HXDS30GVCobeCvC9A1L9A@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Commit ab0dfc961b6 used a "long" variable within _bt_load() to count
the number of tuples entered into a B-Tree index as it is built. This
will not work as expected on Windows, even on 64-bit Windows, because
"long" is only 32-bits wide. It's far from impossible that you'd have
~2 billion index tuples when building a new index.

Programmers use "long" because it is assumed to be wider than "int"
(even though that isn't required by the standard, and isn't true
across all of the platforms we support). The use of "long" seems
inherently suspect given our constraints, except perhaps in the
context of sizing work_mem-based allocations, where it is used as part
of a semi-standard idiom...albeit one that only works because of the
restrictions on work_mem size on Windows.

ISTM that we should try to come up with a way of making code like this
work, rather than placing the burden on new code to get it right. This
exact issue has bitten users on a number of occasions that I can
recall. There is also a hidden landmine that we know about but haven't
fixed: logtape.c, which will break on Windows with very very large
index builds.

Also, "off_t" is only 32-bits on Windows, which broke parallel CREATE
INDEX (issued fixed by commit aa551830). I suppose that "off_t" is
really a variant of the same problem.

--
Peter Geoghegan

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2019-04-28 23:24:59 Re: "long" type is not appropriate for counting tuples
Previous Message Tomas Vondra 2019-04-28 19:52:54 Re: Data streaming between different databases