Re: General purpose hashing func in pgbench

Hello Ildar,

>> so that different instances of hash function within one script would
>> have different seeds. Yes, that is a good idea, I can do that.
>>
> Added this feature in attached patch. But on a second thought this could
> be something that user won't expect. For example, they may want to run
> pgbench with two scripts:
> - the first one updates row by key that is a hashed random_zipfian value;
> - the second one reads row by key generated the same way
> (that is actually what YCSB workloads A and B do)
>
> It feels natural to write something like this:
> \set rnd random_zipfian(0, 1000000, 0.99)
> \set key abs(hash(:rnd)) % 1000
> in both scripts and expect that they both would have the same
> distribution. But they wouldn't. We could of course describe this
> implicit behaviour in documentation, but ISTM that shared seed would be
> more clear.

I think that it depends on the use case, that both can be useful, so there
should be a way to do either.

With "always different" default seed, distinct distributions are achieved
with:

-- DIFF distinct seeds inside and between runs
\set i1 abs(hash(:r1)) % 1000
\set j1 abs(hash(:r2)) % 1000

and the same distribution can be done with an explicit seed:

-- DIFF same seed inside and between runs
\set i1 abs(hash(:r1), 5432) % 1000
\set j1 abs(hash(:r2), 5432) % 1000

The drawback is that the same seed is used between runs in this case,
which is not desirable. This could be circumvented by adding the random
seed as an automatic variable and using it, eg:

-- DIFF same seed inside run, distinct between runs
\set i1 abs(hash(:r1), :random_seed + 5432) % 1000
\set j1 abs(hash(:r2), :random_seed + 2345) % 1000

Now with a shared hash_seed the same distribution is by default:

-- SHARED same underlying hash_seed inside run, distinct between runs
\set i1 abs(hash(:r1)) % 1000
\set j1 abs(hash(:r2)) % 1000

However some trick is needed now to get distinct seeds. With

-- SHARED distinct seed inside run, but same between runs
\set i1 abs(hash(:r1, 5432)) % 1000
\set j1 abs(hash(:r2, 2345)) % 1000

We are back to the same issue has the previous case because then the
distribution is the same from one run to the next, which is not desirable.
I found this workaround trick:

-- SHARED distinct seeds inside and between runs
\set i1 abs(hash(:r1, hash(5432))) % 1000
\set j1 abs(hash(:r2, hash(2345))) % 1000

Or with a new :hash_seed or :random_seed automatic variable, we could also
have:

-- SHARED distinct seeds inside and between runs
\set i1 abs(hash(:r1, :hash_seed + 5432)) % 1000
\set j1 abs(hash(:r2, :hash_seed + 2345)) % 1000

It provides controllable distinct seeds between runs but equal one between
if desired, by reusing the same value to be hashed as a seed.

I also agree with your argument that the user may reasonably expect that
hash(5432) == hash(5432) inside and between scripts, at least on the same
run, so would be surprised that it is not the case.

So I've changed my mind, I'm sorry for making you going back and forth on
the subject. I'm now okay with one shared 64 bit hash seed, with a clear
documentation about the fact, and an outline of the trick to achieve
distinct distributions inside a run if desired and why it would be
desirable to avoid correlations. Also, I think that providing the seed as
automatic variable (:hash_seed or :hseed or whatever) would make some
sense as well. Maybe this could be used as a way to fix the seed
explicitely, eg:

pgbench -D hash_seed=1234 ...

Would use this value instead of the random generated one. Also, with that
the default inserted second argument could be simply ":hash_seed", which
would simplify the executor which would not have to do check for an
optional second argument.

--
Fabien.