Re: PostgreSQL on S3-backed Block Storage with Near-Local Performance

I built postgres (same version, 16.9) but --with-block-size=32 (I'd really love if this would be a initdb time flag!) and did some more testing:

synchronous_commit = off

postgres(at)zerofs:~$ pgbench -vvv -c 100 -j 40 -t 10000 bench
pgbench (16.9 (Ubuntu 16.10-1))
starting vacuum...end.
starting vacuum pgbench_accounts...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 100
number of threads: 40
maximum number of tries: 1
number of transactions per client: 10000
number of transactions actually processed: 1000000/1000000
number of failed transactions: 0 (0.000%)
latency average = 5.727 ms
initial connection time = 59.223 ms
tps = 17460.128835 (without initial connection time)

synchronous_commit = on

postgres(at)zerofs:/root$ pgbench -vvv -c 100 -j 40 -t 1000 bench
pgbench (16.9 (Ubuntu 16.10-1))
starting vacuum...end.
starting vacuum pgbench_accounts...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 100
number of threads: 40
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 100000/100000
number of failed transactions: 0 (0.000%)
latency average = 301.800 ms
initial connection time = 62.237 ms
tps = 331.345391 (without initial connection time)

=====================================

Then, using the same setup (same server, same postgres build), I create a ZeroFS NBD device with ext4 on top

/dev/nbd0 on /mnt_9p type ext4 (rw,relatime,stripe=32)

synchronous_commit = off

postgres(at)zerofs:/mnt_9p$ pgbench -vvv -c 100 -j 40 -t 10000 bench
pgbench (16.9 (Ubuntu 16.10-1))
starting vacuum...end.
starting vacuum pgbench_accounts...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 100
number of threads: 40
maximum number of tries: 1
number of transactions per client: 10000
number of transactions actually processed: 1000000/1000000
number of failed transactions: 0 (0.000%)
latency average = 3.615 ms
initial connection time = 45.653 ms
tps = 27665.373366 (without initial connection time)

synchronous_commit = on

postgres(at)zerofs:/root$ pgbench -vvv -c 100 -j 40 -t 1000 bench
pgbench (16.9 (Ubuntu 16.10-1))
starting vacuum...end.
starting vacuum pgbench_accounts...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 100
number of threads: 40
maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 100000/100000
number of failed transactions: 0 (0.000%)
latency average = 337.762 ms
initial connection time = 43.969 ms
tps = 296.066616 (without initial connection time)

Best,
Pierre

On Fri, Jul 25, 2025, at 11:25, Pierre Barre wrote:
> Hi,
>
> I went ahead and did that test.
>
> Here is the postgresql config I used for reference (note the wal
> options (recycle, init_zero) as well as full_page_writes = off, because
> ZeroFS cannot have torn writes by design).
>
> https://gist.github.com/Barre/8d68f0d00446389998a31f4e60f3276d
>
> Test was running on Azure with Standard D16ads v5 (16 vcpus, 64 GiB memory)
>
> This time, I didn't run ZFS with L2ARC, I just mounted ZeroFS with 9p.
>
> synchronous_commit = off
>
> postgres(at)zerofs:~$ pgbench -vvv -c 100 -j 40 -t 1000 bench
> pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
> starting vacuum...end.
> starting vacuum pgbench_accounts...end.
> transaction type: <builtin: TPC-B (sort of)>
> scaling factor: 50
> query mode: simple
> number of clients: 100
> number of threads: 40
> maximum number of tries: 1
> number of transactions per client: 1000
> number of transactions actually processed: 100000/100000
> number of failed transactions: 0 (0.000%)
> latency average = 6.239 ms
> initial connection time = 68.922 ms
> tps = 16026.940646 (without initial connection time)
>
>
> synchronous_commit = on
>
> postgres(at)zerofs:~$ pgbench -vvv -c 50 -j 15 -t 1000 bench
> pgbench (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1))
> starting vacuum...end.
> starting vacuum pgbench_accounts...end.
> transaction type: <builtin: TPC-B (sort of)>
> scaling factor: 50
> query mode: simple
> number of clients: 50
> number of threads: 15
> maximum number of tries: 1
> number of transactions per client: 1000
> number of transactions actually processed: 50000/50000
> number of failed transactions: 0 (0.000%)
> latency average = 197.723 ms
> initial connection time = 46.089 ms
> tps = 252.878721 (without initial connection time)
>
>
> Not great barebones with with synchronous_commit, but still usable!
>
> Best,
> Pierre
>
> On Fri, Jul 25, 2025, at 00:44, Pierre Barre wrote:
>>> This then begs the obvious question of how fast is this with
>>> synchronous_commit = on?
>>
>> Probably not awful, especially with commit_delay.
>>
>> I'll try that and report back.
>>
>> Best,
>> Pierre
>>
>> On Fri, Jul 25, 2025, at 00:03, Jeff Ross wrote:
>>> On 7/24/25 13:50, Pierre Barre wrote:
>>>
>>>> It’s not “safe” or “unsafe”, there’s mountains of valid workloads which don’t require synchronous_commit. Synchronous_commit don’t make your system automatically safe either, and if that’s a requirement, there’s many workarounds, as you suggested, it certainly doesn’t make the setup useless.
>>>>
>>>> Best,
>>>> Pierre
>>>>
>>>> On Thu, Jul 24, 2025, at 21:44, Nico Williams wrote:
>>>>> On Fri, Jul 18, 2025 at 12:57:39PM +0200, Pierre Barre wrote:
>>>>>> - Postgres configured accordingly memory-wise as well as with
>>>>>> synchronous_commit = off, wal_init_zero = off and wal_recycle = off.
>>>>> Bingo. That's why it's fast (synchronous_commit = off). It's also why
>>>>> it's not safe _unless_ you have a local, fast, persistent ZIL device
>>>>> (which I assume you don't).
>>>>>
>>>>> Nico
>>>>> --
>>> This then begs the obvious question of how fast is this with
>>> synchronous_commit = on?