BDR problem rejoining node

From: Tanner Kerr <tanner(at)itel(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: BDR problem rejoining node
Date: 2017-02-11 00:34:53
Message-ID: CAMRsaUqfSLwstcrQjr39B2bVZeHEp6qFrCLb69_y6NAKN5ryPg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I have two databases being replicated across three nodes with bdr. The
third node filled up and crashed. I removed this node from the group
successfully, but now I'm having trouble rejoining it. I'm able to re-join
the one database no problem. However, trying to do a bdr join on the other
causes the two active nodes to slow down considerably, and refuses to join.
This database copies the data after a few minutes, however, the join fails
and bdr.bdr_nodes on the third node shows only itself in the group with
status 'i'. The active nodes don't show anything different in the
bdr.bdr_nodes. All they can see is a new pg_replication_slot for that node.
The logs say "previous init failed, manual cleanup is required" Even though
there are no bdr_connections or broken bdr_nodes entries. Everytime I've
tried, I've removed the pg_replication slots, the bdr_connections, and the
bdr_nodes (bdr.remove_bdr_from_local_node()). I've also dropped the cluster
and recreated it again each time with:

pg_dropcluster 9.4 main --stop
/usr/bin/pg_createcluster --start 9.4 -d /var/lib/postgresql/9.4/main main

The database failing to copy is small only around 60M. The one being copied
successfully, is around 1600M.

The join command I've been using:

select bdr.bdr_group_join(local_node_name := 'NODENAMEOFREBUILTNODE',
node_external_dsn := 'host=IPOFREBUILTNODE port=5432 dbname=MYDB
connect_timeout=50 keepalives_idle=5 keepalives_interval=1', join_using_dsn
:= 'host=ACTIVENODEIP port=5432 dbname=MYDB connect_timeout=50
keepalives_idle=5 keepalives_interval=1');

I'm not sure why it works for one database and not the other.

The log from the failed join attemp is below:

2017-02-10 14:49:24 PST [6981-10] LOG: Creating replica with:
/usr/lib/postgresql/9.4/bin/bdr_initial_load --snapshot 00C58720-1 --source
"host=ACTIVEIP port=5432 dbname=MYDB connect_timeout=5
0 keepalives_idle=5 keepalives_interval=1" --target "host=REBUILDHOST
port=5432 dbname=MYDB connect_timeout=50 keepalives_idle=5
keepalives_interval=1" --tmp-directory "/tmp/postgres-bdr-00C58720-1
.6981", --pg-dump-path "/usr/lib/postgresql/9.4/bin/bdr_dump",
--pg-restore-path "/usr/lib/postgresql/9.4/bin/pg_restore"
Dumping remote database "connect_timeout=30 keepalives=1 keepalives_idle=20
keepalives_interval=20 keepalives_count=5 host=ACTIVEIP port=5432
dbname=MYDB connect_timeout=50 keepalives_idle=5
keepalives_interval=1 fallback_application_name='bdr
(6385577151748866557,1,16391,): init_replica dump'" with 1 concurrent
workers to "/tmp/postgres-bdr-00C58720-1.6981"
Restoring dump to local DB "host=REBUILDHOST port=5432 dbname=MYDB
connect_timeout=50 keepalives_idle=5 keepalives_interval=1
fallback_application_name='bdr (6385577151748866557,1,16391,): init_replica
restore' options='-c bdr.do_not_replicate=on -c
bdr.permit_unsafe_ddl_commands=on -c bdr.skip_ddl_replication=on -c
bdr.skip_ddl_locking=on -c session_replication_role=replica'" with 1
concurrent workers from "/tmp/postgres-bdr-00C58720-1.6981"
2017-02-10 14:57:16 PST [7271-1] postgres(at)MYDB NOTICE: extension
"btree_gist" already exists, skipping
2017-02-10 14:57:16 PST [7271-2] postgres(at)MYDB NOTICE: extension "bdr"
already exists, skipping
2017-02-10 14:57:16 PST [7271-3] postgres(at)MYDB NOTICE: extension "plpgsql"
already exists, skipping
2017-02-10 14:57:16 PST [7271-4] postgres(at)MYDB NOTICE: extension
"pgcrypto" already exists, skipping
2017-02-10 14:57:16 PST [7271-5] postgres(at)MYDB NOTICE: extension
"uuid-ossp" already exists, skipping

*alters tables and builds indexes*

2017-02-10 14:57:23 PST [6981-10] DEBUG: syncing bdr_nodes and
bdr_connections
2017-02-10 14:57:23 PST [6981-11] ERROR: BEGIN or table locking on remote
failed:
2017-02-10 14:57:23 PST [6884-11] LOG: worker process: bdr db: MYDB (PID
6981) exited with exit code 1
2017-02-10 14:57:28 PST [6884-12] LOG: starting background worker process
"bdr db: MYDB"
2017-02-10 14:57:28 PST [7274-1] NOTICE: version "1.0" of extension
"btree_gist" is already installed
2017-02-10 14:57:28 PST [7274-2] NOTICE: version "1.0.1.0" of extension
"bdr" is already installed
2017-02-10 14:57:28 PST [7274-3] DEBUG: per-db worker for node bdr
(6385577151748866557,1,16391,) starting
2017-02-10 14:57:28 PST [7274-4] DEBUG: init_replica init from remote
host=ACTIVEIP port=5432 dbname=MYDB connect_timeout=50 keepalives_idle=5
keepalives_interval=1
2017-02-10 14:57:28 PST [7274-5] ERROR: previous init failed, manual
cleanup is required
2017-02-10 14:57:28 PST [7274-6] DETAIL: Found bdr.bdr_nodes entry for bdr
(6385577151748866557,1,16391,) with state=i in remote bdr.bdr_nodes
2017-02-10 14:57:28 PST [7274-7] HINT: Remove all replication identifiers
and slots corresponding to this node from the init target node then drop
and recreate this database and try again

Tanner

Responses

Browse pgsql-general by date

  From Date Subject
Next Message David G. Johnston 2017-02-11 01:46:08 Re: Fwd: Query parameter types not recognized
Previous Message rob stone 2017-02-11 00:13:41 Re: Fwd: Query parameter types not recognized