Re: Temporary file access API

I’m a latecomer to the discussion, but as a word of introduction, I’m working with Stephen, and I have started looking over the temp file proposal with the idea of helping it move along.

I’ll start by summarizing the temp file proposal and its goals.

From a high level, the proposed code:

* Creates an fread/fwrite replacement (BufFileStream) for buffering data to a single file.
* Updates BufFile, which reads/writes a set of files, to use BufFileStream internally.
* Does not impact the normal (page cached) database I/O.
* Updates all the other places where fread/fwrite and read/write are used.
* Creates and removes transient files.
I see the following goals:

* Unify all the “other” file accesses into a single, consistent API.
* Integrate with VFDs.
* Integrate transient files with transactions and tablespaces.
* Create a consolidated framework where features like encryption and compression can be more easily added.
* Maintain good streaming performance.
Is this a fair description of the proposal?

For myself, I’d like to map out how features like compression and encryption would fit into the framework, more as a sanity check than anything else, and I’d like to look closer at some of the implementation details. But at the moment, I want to make sure I have the proper high-level view of the temp file proposal.

From: Robert Haas <robertmhaas(at)gmail(dot)com>
Date: Wednesday, September 21, 2022 at 11:54 AM
To: Antonin Houska <ah(at)cybertec(dot)at>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Peter Eisentraut <peter(dot)eisentraut(at)enterprisedb(dot)com>, Stephen Frost <sfrost(at)snowman(dot)net>
Subject: Re: Temporary file access API
On Mon, Aug 8, 2022 at 2:26 PM Antonin Houska <ah(at)cybertec(dot)at> wrote:
> > I don't think that (3) is a separate advantage from (1) and (2), so I
> > don't have anything separate to say about it.
>
> I thought that the uncontrollable buffer size is one of the things you
> complaint about in
>
> https://www.postgresql.org/message-id/CA+TgmoYGjN_f=FCErX49bzjhNG+GoctY+a+XhNRWCVvDY8U74w@mail.gmail.com

Well, I think that email is mostly complaining about there being no
buffering at all in a situation where it would be advantageous to do
some buffering. But that particular code I believe is gone now because
of the shared-memory stats collector, and when looking through the
patch set, I didn't see that any of the cases that it touched were
similar to that one.

--
Robert Haas
EDB: http://www.enterprisedb.com