Re: pg_terminate_backend() issues

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > Tom Lane wrote:
> >> I'm willing to enable a SIGTERM-based pg_terminate_backend for 8.4
> >> if there is some reasonable amount of testing done during this
> >> development cycle to try to expose any problems.
>
> > If someone can come up with an automated script to do this kind of
> > testing, I can commit a VM or three to running this 24/7 for a month,
> > easily... But I don't trust myself in coming up with a test-case that's
> > good enough :-P
>
> The closest thing I can think of to an automated test is to run repeated
> sets of the parallel regression tests, and each time SIGTERM a randomly
> chosen backend at a randomly chosen time. Then see if anything "funny"

Yep, that was my plan, plus running the parallel regression tests you
get the possibility of >2 backends.

> happens. The hard part here is distinguishing expected from unexpected
> regression outputs, especially in view of the fact that some of the
> tests depend on database contents set up by earlier tests.

Oh, that is a problem. I was going to get the typical errors, sort them
and put them in a file. Then when I run the test, I sort the log again
and use diff | grep '<' to see an differences. If there are cases that
generate new errors on a previous failure, I will have to add that
output too.

> I'm thinking that you could automatically discard the regression diff
> for the specific test that got SIGTERM'd, as long as it looked like
> the normal output up to the point where the "terminated by
> administrator" error appears. Then what you'd have is the potential for
> downstream failures due to things not being created, which *should* fall
> into a fairly stylized set of possible diffs. So get the script to
> throw away any diffs that exactly match ones seen previously. Run it
> for awhile, and then hand-validate the set of diffs that it's saved
> ... or if any of 'em look funny, report.

Yep, see above.

> One gotcha I can think of is that killing the prepared_xacts test
> can leave you with open 2PC transactions, which will interfere with
> starting the next cycle of the tests (you have to kill them before you
> can dropdb). But you could add a "rollback prepared" to the driver
> script to clean out any uncommitted prepared xact.

Good idea.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +