Re: pgbench --continue-on-error: clarify TPS and failure reporting

On Thu, 18 Jun 2026 11:24:04 +0800
Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com> wrote:

> Hi
>
> While testing “[0ab208fa5] pgbench: Add --continue-on-error option”, I got a question about the doc:
> ```
> <para>
> This option is useful when your custom script may raise errors
> such as unique constraint violations, but you want the benchmark
> to continue and measure performance including those failures.
> </para>
> ```
>
> The phrase “measure performance including those failures” gives the impression that failed transactions might be counted in TPS, but they are not.

I agree with your point. The original phrasing "including those failures" is
indeed　ambiguous and could mislead users into thinking failed transactions are
included in the TPS calculation.

> In my test, all transactions failed, and TPS was reported as 0:
> ```
> % pgbench -d evantest -n -t 5 --continue-on-error --failures-detailed -f pgbench-coe.sql
> pgbench (19beta1)
> transaction type: pgbench-coe.sql
> scaling factor: 1
> query mode: simple
> number of clients: 1
> number of threads: 1
> maximum number of tries: 1
> number of transactions per client: 5
> number of transactions actually processed: 0/5
> number of failed transactions: 5 (100.000%)
> number of serialization failures: 0 (0.000%)
> number of deadlock failures: 0 (0.000%)
> number of other failures: 5 (100.000%)
> latency average = 0.389 ms (including failures)
> initial connection time = 4.172 ms
> tps = 0.000000 (without initial connection time)
> ```
>
> If this behavior is intended, I wonder if we should clarify the TPS counting logic more explicitly. For example:
> ```
> <para>
> This option is useful when your custom script may raise errors
> such as unique constraint violations, but you want the benchmark
> to continue despite individual statement failures. Failed transactions
> are reported separately and included in latency statistics, but are not
> counted as transactions actually processed for TPS.
> </para>
> ```

Your proposed documentation change is much clearer and accurately reflects
the actual behavior. I prefer this version as well.

Regards,
Yugo Nagata

--
Yugo Nagata <nagata(at)sraoss(dot)co(dot)jp>