Re: speed up a logical replica setup

On Wed, 6 Dec 2023 at 12:53, Euler Taveira <euler(at)eulerto(dot)com> wrote:
>
> On Thu, Nov 9, 2023, at 8:12 PM, Michael Paquier wrote:
>
> On Thu, Nov 09, 2023 at 03:41:53PM +0100, Peter Eisentraut wrote:
> > On 08.11.23 00:12, Michael Paquier wrote:
> >> - Should the subdirectory pg_basebackup be renamed into something more
> >> generic at this point? All these things are frontend tools that deal
> >> in some way with the replication protocol to do their work. Say
> >> a replication_tools?
> >
> > Seems like unnecessary churn. Nobody has complained about any of the other
> > tools in there.
>
> Not sure. We rename things across releases in the tree from time to
> time, and here that's straight-forward.
>
>
> Based on this discussion it seems we have a consensus that this tool should be
> in the pg_basebackup directory. (If/when we agree with the directory renaming,
> it could be done in a separate patch.) Besides this move, the v3 provides a dry
> run mode. It basically executes every routine but skip when should do
> modifications. It is an useful option to check if you will be able to run it
> without having issues with connectivity, permission, and existing objects
> (replication slots, publications, subscriptions). Tests were slightly improved.
> Messages were changed to *not* provide INFO messages by default and --verbose
> provides INFO messages and --verbose --verbose also provides DEBUG messages. I
> also refactored the connect_database() function into which the connection will
> always use the logical replication mode. A bug was fixed in the transient
> replication slot name. Ashutosh review [1] was included. The code was also indented.
>
> There are a few suggestions from Ashutosh [2] that I will reply in another
> email.
>
> I'm still planning to work on the following points:
>
> 1. improve the cleanup routine to point out leftover objects if there is any
> connection issue.
> 2. remove the physical replication slot if the standby is using one
> (primary_slot_name).
> 3. provide instructions to promote the logical replica into primary, I mean,
> stop the replication between the nodes and remove the replication setup
> (publications, subscriptions, replication slots). Or even include another
> action to do it. We could add both too.
>
> Point 1 should be done. Points 2 and 3 aren't essential but will provide a nice
> UI for users that would like to use it.

Few comments:
1) We should not allow specifying the same database name twice as we
will try to create the slots multiple times in the publisher, this can
be detected while parsing the options and error can be thrown:
+ case 'd':
+
simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ break;

+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
....
....
....
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i],
replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on
publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }

E.g.: pg_subscriber -d postgres -d postgres

2) 2023 should be changed to 2024
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */

3) Similarly here too:
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..9d20847dc2
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#

4) Similarly here too:
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..ce25608c68
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.

Regards,
Vignesh