Re: pgbench --continue-on-error: clarify TPS and failure reporting

> On Jun 18, 2026, at 13:36, Yugo Nagata <nagata(at)sraoss(dot)co(dot)jp> wrote:
>
> On Thu, 18 Jun 2026 11:24:04 +0800
> Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com> wrote:
>
>> Hi
>>
>> While testing “[0ab208fa5] pgbench: Add --continue-on-error option”, I got a question about the doc:
>> ```
>> <para>
>> This option is useful when your custom script may raise errors
>> such as unique constraint violations, but you want the benchmark
>> to continue and measure performance including those failures.
>> </para>
>> ```
>>
>> The phrase “measure performance including those failures” gives the impression that failed transactions might be counted in TPS, but they are not.
>
> I agree with your point. The original phrasing "including those failures" is
> indeed　ambiguous and could mislead users into thinking failed transactions are
> included in the TPS calculation.
>
>> In my test, all transactions failed, and TPS was reported as 0:
>> ```
>> % pgbench -d evantest -n -t 5 --continue-on-error --failures-detailed -f pgbench-coe.sql
>> pgbench (19beta1)
>> transaction type: pgbench-coe.sql
>> scaling factor: 1
>> query mode: simple
>> number of clients: 1
>> number of threads: 1
>> maximum number of tries: 1
>> number of transactions per client: 5
>> number of transactions actually processed: 0/5
>> number of failed transactions: 5 (100.000%)
>> number of serialization failures: 0 (0.000%)
>> number of deadlock failures: 0 (0.000%)
>> number of other failures: 5 (100.000%)
>> latency average = 0.389 ms (including failures)
>> initial connection time = 4.172 ms
>> tps = 0.000000 (without initial connection time)
>> ```
>>
>> If this behavior is intended, I wonder if we should clarify the TPS counting logic more explicitly. For example:
>> ```
>> <para>
>> This option is useful when your custom script may raise errors
>> such as unique constraint violations, but you want the benchmark
>> to continue despite individual statement failures. Failed transactions
>> are reported separately and included in latency statistics, but are not
>> counted as transactions actually processed for TPS.
>> </para>
>> ```
>
> Your proposed documentation change is much clearer and accurately reflects
> the actual behavior. I prefer this version as well.
>
> Regards,
> Yugo Nagata
>
> --
> Yugo Nagata <nagata(at)sraoss(dot)co(dot)jp>

Thanks for confirming the doc issue. The here comes a small doc patch.

I also noticed there is a later paragraph about “latency” that also needs an update, it’s included in the patch as well.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/