Re: BUG #15858: could not stat file - over 4GB

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Juan José Santamaría Flecha <juanjo(dot)santamaria(at)gmail(dot)com>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, williamedwinallen(at)live(dot)com, pgsql-bugs(at)lists(dot)postgresql(dot)org, Magnus Hagander <magnus(at)hagander(dot)net>
Subject: Re: BUG #15858: could not stat file - over 4GB
Date: 2019-06-19 18:02:36
Message-ID: 17044.1560967356@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs pgsql-hackers

I wrote:
> Another issue here is that pgwin32_safestat() probably needs revisited
> as to its scope and purpose. Its use of GetFileAttributesEx() can
> presumably be dropped. I don't actually believe the header comment
> claiming that stat() is not guaranteed to update the st_size field;
> there's no indication of that in the Microsoft documentation. What
> seems more likely is that that's a garbled version of the truth,
> that you won't get a correct value of _st_size for files over 4GB.

So after further digging around, it seems that this is wrong. The
existence of pgwin32_safestat() can be traced back to these threads:

https://www.postgresql.org/message-id/flat/528853D3C5ED2C4AA8990B504BA7FB850106DF10%40sol.transas.com
https://www.postgresql.org/message-id/flat/528853D3C5ED2C4AA8990B504BA7FB850106DF2F%40sol.transas.com

in which it's stated that

It seems I've found the cause and the workaround of the problem.
MSVC's stat() is implemented by using FindNextFile().
MSDN contains the following suspicious paragraph аbout FindNextFile():
"In rare cases, file attribute information on NTFS file systems
may not be current at the time you call this function. To obtain
the current NTFS file system file attributes, call
GetFileInformationByHandle."
Since we generally cannot open an examined file, we need another way.

I'm wondering though why we adopted the existing coding in the face of
that observation. Couldn't the rest of struct stat be equally out of
date?

In short it seems like maybe we should be doing something similar to the
patch that Sergey actually submitted in that discussion:

https://www.postgresql.org/message-id/528853D3C5ED2C4AA8990B504BA7FB850658BA5C%40sol.transas.com

which reimplements stat() from scratch on top of GetFileAttributesEx(),
and thus doesn't require any assumptions at all about what's available
from the toolchain's <sys/stat.h>.

regards, tom lane

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Lynn Carol Johnson 2019-06-19 19:19:18 Re: BUG #15854: postgres wtih Docker: binding port fails with release greater than 9.6.13
Previous Message Tom Lane 2019-06-19 17:40:10 Re: BUG #15858: could not stat file - over 4GB

Browse pgsql-hackers by date

  From Date Subject
Next Message Robert Haas 2019-06-19 18:05:38 Re: POC: Cleaning up orphaned files using undo logs
Previous Message Pavel Stehule 2019-06-19 17:46:46 Re: idea: log_statement_sample_rate - bottom limit for sampling