SSL and crash woes.

A couple of years back (2005) we were setting up replication for the first time (using slony) from
our production database server to a stand-by box sitting next to it and a remote box in a DR
site. We were running FreeBSD 5.X/6.X on all systems on Dell servers and postgres 7.4.X and then
8.0.X

Replication appeared to crash our production database...a lot. After looking at the core
dumps/stack traces at the time, we determined that we were crashing in the ssl layers....so we
disabled SSL (via pg_hba.conf and the slony conn settings) and haven't had an issue for the last
couple of years.

Stable as a rock.

Well....we just upgraded our hardware (Sun X4600s) and operating sytems (solaris 10) , postgres
versions (8.2.4), and slony (1.2.10). Rock solid.
However, our first indication of an issue was an issue with executing pg_dump from a remote backup
server. (see http://archives.postgresql.org/pgsql-general/2007-08/msg01347.php)
Local pg_dump's have no issue. So we changed our backup scheme to do local dumps and push the
files off the server to the backup location. Problem solved.

Then...replication woes again. With these fresh installs, we didn't think too much about the SSL
settings....and bing-bang...crash. Crash. Crash. Crash. Stopped replication. Problem goes
away. Start replication...crash crash. So we stopped replication.

We recompiled postgres with debug info on a test db box and loaded up the most recent database
dump. We then attempted a remote pg_dump from another local server. Crash. Took a look at the
core dump...

Core was generated by `/usr/local/pgsql/bin/postgres -D /testdb'.
Program terminated with signal 11, Segmentation fault.
#0 0xfee8ec23 in sk_value () from /usr/local/ssl/lib/libcrypto.so.0.9.8
(gdb) bt
#0 0xfee8ec23 in sk_value () from /usr/local/ssl/lib/libcrypto.so.0.9.8
#1 0xfef5b05b in ssl3_output_cert_chain () from /usr/local/ssl/lib/libssl.so.0.9.8
#2 0x00000000 in ?? ()

Hmmm...that looked familiar (from years ago)

So....we set up the connection to be 'hostnossl' in pg_hba.conf and tried again. Success.
Changed it back to 'hostssl'.....crash. Same place.

I am going to take the time and set up test environment for the replication as well, but I assume
I will experience the same thing. SSL means crash...no SSL means no crash.

Anyone have any thoughts?

____________________________________________________________________________________
Pinpoint customers who are looking for what you sell.
http://searchmarketing.yahoo.com/