PSA: rr recorder/debugger works well with Postgres + Linux

Heikki mentioned the rr recorder/debugger during his recent pgConf.EU
talk. It was recommended as a general debugging tool. I can now second
that recommendation. See https://rr-project.org for general background
information. It's a framework that extends gdb, so it's largely
compatible with existing gdb workflows.

I had access to a proprietary reverse debugger called UndoDB some
years back, and missed having that capability. I found that rr works
rather well with Postgres when I looked into it today. It seems mature
in general. I do have some specific guidance on getting it working
with Postgres, though:

* rr has a problem instrumenting the sync_file_range() syscall, so you
may want to start recording using a recipe along these lines:

$ rr record /code/postgresql/public/install/bin/postgres -D
/code/postgresql/public/data --wal_writer_flush_after=0
--backend_flush_after=0 --bgwriter_flush_after=0
--checkpoint_flush_after=0

(It doesn't matter how Postgres was built, as long as
sync_file_range() actually isn't called, which this avoids.)

* When replaying with "rr replay", be sure to use the -f flag, not the
-p flag, since there is usually no exec() for the -p option to pick up
on within Postgres backends.

I've found a reverse debugger particularly useful when trying to
understand the optimizer's behavior. The rr "when" command shows the
current event, which can be used to understand what point in
optimization we're in when a break point is hit. Recording is also
very helpful because the values of pointers and other incidental
details don't change within a recording, allowing you to take notes
that include "stable" pointer values that remain usable days later.
Breaking down query planning into understandable parts becomes far
easier.

I never had much use for watchpoints in the past, but once I gained
the ability to "reverse-continue" that changed. I don't find that a
reverse debugger makes an enormous difference when I already more or
less understand the code that I'm debugging, but it does still make
the process a bit easier. rr is supposed to be particularly helpful
with non-deterministic or difficult to recreate bugs; hopefully I'll
be able to apply it in that way when I gain some more experience with
it.

--
Peter Geoghegan