Re: Ragged latency log data in multi-threaded pgbench

Takahiro Itagaki wrote:
> 1. Use explicit locks. The lock primitive will be pthread_mutex for
> multi-threaded implementations or semaphore for multi-threaded ones.
> 2. Use per-thread log files.
> File names would be "pgbench_log.<main-process-id>.<thread-id>".
>

I'm concerned that the locking itself will turn into a new pgbench
bottleneck, just as we're clearing the point where it's not for the
first time in a while. And that sounds like it has its own potential
risks/complexity involved.

I could live with per-thread log files. I think my pgbench-tools is the
main consumer of these latency logs floating around right now, I just
pushed a 9.0 update to handle the multiple workers option today that
discovered this). It doesn't make any difference to what I'm doing how
many file I have to process. Just a few lines of extra shell code for
me to pull the rest into the import. That seems like the simplest
solution that's guaranteed to work, just push the problem onto the
client side instead where it's easier to deal with.

Unless someone feels strongly that these have to be interleaved into one
file, based on Andrew's suggestion that this is a hard problem to get
right and Tom's suggestion that this might even extend into the proper
threaded version too, I think a log file per worker is the easiest way
out of this.

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.us