Re: WIP/PoC for parallel backup

Hi Asif

Getting the following error on Parallel backup when --no-manifest option is
used.

[edb(at)localhost bin]$
[edb(at)localhost bin]$
[edb(at)localhost bin]$ ./pg_basebackup -v -j 5 -D
/home/edb/Desktop/backup/ --no-manifest
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_10223"
pg_basebackup: backup worker (0) created
pg_basebackup: backup worker (1) created
pg_basebackup: backup worker (2) created
pg_basebackup: backup worker (3) created
pg_basebackup: backup worker (4) created
pg_basebackup: write-ahead log end point: 0/2000100
pg_basebackup: error: could not get data for 'BUILD_MANIFEST': ERROR:
could not open file
"base/pgsql_tmp/pgsql_tmp_b4ef5ac0fd150b2a28caf626bbb1bef2.1": No such file
or directory
pg_basebackup: removing contents of data directory
"/home/edb/Desktop/backup/"
[edb(at)localhost bin]$

Thanks

On Tue, Apr 14, 2020 at 5:33 PM Asif Rehman <asifr(dot)rehman(at)gmail(dot)com> wrote:

>
>
> On Wed, Apr 8, 2020 at 6:53 PM Kashif Zeeshan <
> kashif(dot)zeeshan(at)enterprisedb(dot)com> wrote:
>
>>
>>
>> On Tue, Apr 7, 2020 at 9:44 PM Asif Rehman <asifr(dot)rehman(at)gmail(dot)com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks, Kashif and Rajkumar. I have fixed the reported issues.
>>>
>>> I have added the shared state as previously described. The new grammar
>>> changes
>>> are as follows:
>>>
>>> START_BACKUP [LABEL '<label>'] [FAST] [MAX_RATE %d]
>>> - This will generate a unique backupid using pg_strong_random(16)
>>> and hex-encoded
>>> it. which is then returned as the result set.
>>> - It will also create a shared state and add it to the hashtable.
>>> The hash table size is set
>>> to BACKUP_HASH_SIZE=10, but since hashtable can expand
>>> dynamically, I think it's
>>> sufficient initial size. max_wal_senders is not used, because it
>>> can be set to quite a
>>> large values.
>>>
>>> JOIN_BACKUP 'backup_id'
>>> - finds 'backup_id' in hashtable and attaches it to server process.
>>>
>>>
>>> SEND_FILE '(' 'FILE' ')' [NOVERIFY_CHECKSUMS]
>>> - renamed SEND_FILES to SEND_FILE
>>> - removed START_WAL_LOCATION from this because 'startptr' is now
>>> accessible through
>>> shared state.
>>>
>>> There is no change in other commands:
>>> STOP_BACKUP [NOWAIT]
>>> LIST_TABLESPACES [PROGRESS]
>>> LIST_FILES [TABLESPACE]
>>> LIST_WAL_FILES [START_WAL_LOCATION 'X/X'] [END_WAL_LOCATION 'X/X']
>>>
>>> The current patches (v11) have been rebased to the latest master. The
>>> backup manifest is enabled
>>> by default, so I have disabled it for parallel backup mode and have
>>> generated a warning so that
>>> user is aware of it and not expect it in the backup.
>>>
>>> Hi Asif
>>
>> I have verified the bug fixes, one bug is fixed and working now as
>> expected
>>
>> For the verification of the other bug fixes faced following issues,
>> please have a look.
>>
>>
>> 1) Following bug fixes mentioned below are generating segmentation fault.
>>
>> Please note for reference I have added a description only as steps were
>> given in previous emails of each bug I tried to verify the fix. Backtrace
>> is also added with each case which points to one bug for both the cases.
>>
>> a) The backup failed with errors "error: could not connect to server:
>> could not look up local user ID 1000: Too many open files" when the
>> max_wal_senders was set to 2000.
>>
>>
>> [edb(at)localhost bin]$ ./pg_basebackup -v -j 1990 -D
>> /home/edb/Desktop/backup/
>> pg_basebackup: warning: backup manifest is disabled in parallel backup
>> mode
>> pg_basebackup: initiating base backup, waiting for checkpoint to complete
>> pg_basebackup: checkpoint completed
>> pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
>> pg_basebackup: starting background WAL receiver
>> pg_basebackup: created temporary replication slot "pg_basebackup_9925"
>> pg_basebackup: backup worker (0) created
>> pg_basebackup: backup worker (1) created
>> pg_basebackup: backup worker (2) created
>> pg_basebackup: backup worker (3) created
>> ….
>> ….
>> pg_basebackup: backup worker (1014) created
>> pg_basebackup: backup worker (1015) created
>> pg_basebackup: backup worker (1016) created
>> pg_basebackup: backup worker (1017) created
>> pg_basebackup: error: could not connect to server: could not look up
>> local user ID 1000: Too many open files
>> Segmentation fault
>> [edb(at)localhost bin]$
>>
>>
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$ gdb pg_basebackup
>> /tmp/cores/core.pg_basebackup.13219.localhost.localdomain.1586349551
>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from
>> /home/edb/Communtiy_Parallel_backup/postgresql/inst/bin/pg_basebackup...done.
>> [New LWP 13219]
>> [New LWP 13222]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `./pg_basebackup -v -j 1990 -D
>> /home/edb/Desktop/backup/'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 pthread_join (threadid=0, thread_return=0x0) at pthread_join.c:47
>> 47 if (INVALID_NOT_TERMINATED_TD_P (pd))
>> (gdb) bt
>> #0 pthread_join (threadid=0, thread_return=0x0) at pthread_join.c:47
>> #1 0x000000000040904a in cleanup_workers () at pg_basebackup.c:2978
>> #2 0x0000000000403806 in disconnect_atexit () at pg_basebackup.c:332
>> #3 0x00007f2226f76a49 in __run_exit_handlers (status=1,
>> listp=0x7f22272f86c8 <__exit_funcs>, run_list_atexit=run_list_atexit(at)entry=true)
>> at exit.c:77
>> #4 0x00007f2226f76a95 in __GI_exit (status=<optimized out>) at exit.c:99
>> #5 0x0000000000408c54 in create_parallel_workers (backupinfo=0x952ca0)
>> at pg_basebackup.c:2811
>> #6 0x000000000040798f in BaseBackup () at pg_basebackup.c:2211
>> #7 0x0000000000408b4d in main (argc=6, argv=0x7ffe3dabc718) at
>> pg_basebackup.c:2765
>> (gdb)
>>
>>
>>
>>
>> b) When executing two backups at the same time, getting FATAL error due
>> to max_wal_senders and instead of exit Backup got completed.
>>
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$ ./pg_basebackup -v -j 8 -D
>> /home/edb/Desktop/backup1/
>> pg_basebackup: warning: backup manifest is disabled in parallel backup
>> mode
>> pg_basebackup: initiating base backup, waiting for checkpoint to complete
>> pg_basebackup: checkpoint completed
>> pg_basebackup: write-ahead log start point: 1/DA000028 on timeline 1
>> pg_basebackup: starting background WAL receiver
>> pg_basebackup: created temporary replication slot "pg_basebackup_17066"
>> pg_basebackup: backup worker (0) created
>> pg_basebackup: backup worker (1) created
>> pg_basebackup: backup worker (2) created
>> pg_basebackup: backup worker (3) created
>> pg_basebackup: backup worker (4) created
>> pg_basebackup: backup worker (5) created
>> pg_basebackup: backup worker (6) created
>> pg_basebackup: error: could not connect to server: FATAL: number of
>> requested standby connections exceeds max_wal_senders (currently 10)
>> Segmentation fault (core dumped)
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$ gdb pg_basebackup
>> /tmp/cores/core.pg_basebackup.17041.localhost.localdomain.1586353696
>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from
>> /home/edb/Communtiy_Parallel_backup/postgresql/inst/bin/pg_basebackup...done.
>> [New LWP 17041]
>> [New LWP 17067]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `./pg_basebackup -v -j 8 -D
>> /home/edb/Desktop/backup1/'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 pthread_join (threadid=0, thread_return=0x0) at pthread_join.c:47
>> 47 if (INVALID_NOT_TERMINATED_TD_P (pd))
>> (gdb) bt
>> #0 pthread_join (threadid=0, thread_return=0x0) at pthread_join.c:47
>> #1 0x000000000040904a in cleanup_workers () at pg_basebackup.c:2978
>> #2 0x0000000000403806 in disconnect_atexit () at pg_basebackup.c:332
>> #3 0x00007f051edc1a49 in __run_exit_handlers (status=1,
>> listp=0x7f051f1436c8 <__exit_funcs>, run_list_atexit=run_list_atexit(at)entry=true)
>> at exit.c:77
>> #4 0x00007f051edc1a95 in __GI_exit (status=<optimized out>) at exit.c:99
>> #5 0x0000000000408c54 in create_parallel_workers (backupinfo=0x1c6dca0)
>> at pg_basebackup.c:2811
>> #6 0x000000000040798f in BaseBackup () at pg_basebackup.c:2211
>> #7 0x0000000000408b4d in main (argc=6, argv=0x7ffdb76a6d68) at
>> pg_basebackup.c:2765
>> (gdb)
>>
>>
>>
>>
>> 2) The following bug is not fixed yet
>>
>> A similar case is when DB Server is shut down while the Parallel Backup
>> is in progress then the correct error is displayed but then the backup
>> folder is not cleaned and leaves a corrupt backup.
>>
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$ ./pg_basebackup -v -D /home/edb/Desktop/backup/ -j
>> 8
>> pg_basebackup: warning: backup manifest is disabled in parallel backup
>> mode
>> pg_basebackup: initiating base backup, waiting for checkpoint to complete
>> pg_basebackup: checkpoint completed
>> pg_basebackup: write-ahead log start point: 0/A0000028 on timeline 1
>> pg_basebackup: starting background WAL receiver
>> pg_basebackup: created temporary replication slot "pg_basebackup_16235"
>> pg_basebackup: backup worker (0) created
>> pg_basebackup: backup worker (1) created
>> pg_basebackup: backup worker (2) created
>> pg_basebackup: backup worker (3) created
>> pg_basebackup: backup worker (4) created
>> pg_basebackup: backup worker (5) created
>> pg_basebackup: backup worker (6) created
>> pg_basebackup: backup worker (7) created
>> pg_basebackup: error: could not read COPY data: server closed the
>> connection unexpectedly
>> This probably means the server terminated abnormally
>> before or while processing the request.
>> pg_basebackup: error: could not read COPY data: server closed the
>> connection unexpectedly
>> This probably means the server terminated abnormally
>> before or while processing the request.
>> pg_basebackup: removing contents of data directory
>> "/home/edb/Desktop/backup/"
>> pg_basebackup: error: could not read COPY data: server closed the
>> connection unexpectedly
>> This probably means the server terminated abnormally
>> before or while processing the request.
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$
>>
>>
>>
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$ ls /home/edb/Desktop/backup
>> base pg_hba.conf pg_logical pg_notify pg_serial
>> pg_stat pg_subtrans pg_twophase pg_xact postgresql.conf
>> pg_dynshmem pg_ident.conf pg_multixact pg_replslot pg_snapshots
>> pg_stat_tmp pg_tblspc PG_VERSION postgresql.auto.conf
>> [edb(at)localhost bin]$
>> [edb(at)localhost bin]$
>>
>>
>>
>>
>> Thanks
>> Kashif Zeeshan
>>
>>>
>>>
>>> On Tue, Apr 7, 2020 at 4:03 PM Kashif Zeeshan <
>>> kashif(dot)zeeshan(at)enterprisedb(dot)com> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Apr 3, 2020 at 3:01 PM Kashif Zeeshan <
>>>> kashif(dot)zeeshan(at)enterprisedb(dot)com> wrote:
>>>>
>>>>> Hi Asif
>>>>>
>>>>> When a non-existent slot is used with tablespace then correct error is
>>>>> displayed but then the backup folder is not cleaned and leaves a corrupt
>>>>> backup.
>>>>>
>>>>> Steps
>>>>> =======
>>>>>
>>>>> edb(at)localhost bin]$
>>>>> [edb(at)localhost bin]$ mkdir /home/edb/tbl1
>>>>> [edb(at)localhost bin]$ mkdir /home/edb/tbl_res
>>>>> [edb(at)localhost bin]$
>>>>> postgres=# create tablespace tbl1 location '/home/edb/tbl1';
>>>>> CREATE TABLESPACE
>>>>> postgres=#
>>>>> postgres=# create table t1 (a int) tablespace tbl1;
>>>>> CREATE TABLE
>>>>> postgres=# insert into t1 values(100);
>>>>> INSERT 0 1
>>>>> postgres=# insert into t1 values(200);
>>>>> INSERT 0 1
>>>>> postgres=# insert into t1 values(300);
>>>>> INSERT 0 1
>>>>> postgres=#
>>>>>
>>>>>
>>>>> [edb(at)localhost bin]$
>>>>> [edb(at)localhost bin]$ ./pg_basebackup -v -j 2 -D
>>>>> /home/edb/Desktop/backup/ -T /home/edb/tbl1=/home/edb/tbl_res -S test
>>>>> pg_basebackup: initiating base backup, waiting for checkpoint to
>>>>> complete
>>>>> pg_basebackup: checkpoint completed
>>>>> pg_basebackup: write-ahead log start point: 0/2E000028 on timeline 1
>>>>> pg_basebackup: starting background WAL receiver
>>>>> pg_basebackup: error: could not send replication command
>>>>> "START_REPLICATION": ERROR: replication slot "test" does not exist
>>>>> pg_basebackup: backup worker (0) created
>>>>> pg_basebackup: backup worker (1) created
>>>>> pg_basebackup: write-ahead log end point: 0/2E000100
>>>>> pg_basebackup: waiting for background process to finish streaming ...
>>>>> pg_basebackup: error: child thread exited with error 1
>>>>> [edb(at)localhost bin]$
>>>>>
>>>>> backup folder not cleaned
>>>>>
>>>>> [edb(at)localhost bin]$
>>>>> [edb(at)localhost bin]$
>>>>> [edb(at)localhost bin]$
>>>>> [edb(at)localhost bin]$ ls /home/edb/Desktop/backup
>>>>> backup_label global pg_dynshmem pg_ident.conf pg_multixact
>>>>> pg_replslot pg_snapshots pg_stat_tmp pg_tblspc PG_VERSION pg_xact
>>>>> postgresql.conf
>>>>> base pg_commit_ts pg_hba.conf pg_logical pg_notify
>>>>> pg_serial pg_stat pg_subtrans pg_twophase pg_wal
>>>>> postgresql.auto.conf
>>>>> [edb(at)localhost bin]$
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> If the same case is executed without the parallel backup patch then
>>>>> the backup folder is cleaned after the error is displayed.
>>>>>
>>>>> [edb(at)localhost bin]$ ./pg_basebackup -v -D /home/edb/Desktop/backup/
>>>>> -T /home/edb/tbl1=/home/edb/tbl_res -S test999
>>>>> pg_basebackup: initiating base backup, waiting for checkpoint to
>>>>> complete
>>>>> pg_basebackup: checkpoint completed
>>>>> pg_basebackup: write-ahead log start point: 0/2B000028 on timeline 1
>>>>> pg_basebackup: starting background WAL receiver
>>>>> pg_basebackup: error: could not send replication command
>>>>> "START_REPLICATION": ERROR: replication slot "test999" does not exist
>>>>> pg_basebackup: write-ahead log end point: 0/2B000100
>>>>> pg_basebackup: waiting for background process to finish streaming ...
>>>>> pg_basebackup: error: child process exited with exit code 1
>>>>> *pg_basebackup: removing data directory " /home/edb/Desktop/backup"*
>>>>> pg_basebackup: changes to tablespace directories will not be undone
>>>>>
>>>>
>>>>
>>>> Hi Asif
>>>>
>>>> A similar case is when DB Server is shut down while the Parallel Backup
>>>> is in progress then the correct error is displayed but then the backup
>>>> folder is not cleaned and leaves a corrupt backup. I think one bug fix will
>>>> solve all these cases where clean up is not done when parallel backup is
>>>> failed.
>>>>
>>>> [edb(at)localhost bin]$
>>>> [edb(at)localhost bin]$
>>>> [edb(at)localhost bin]$ ./pg_basebackup -v -D /home/edb/Desktop/backup/
>>>> -j 8
>>>> pg_basebackup: initiating base backup, waiting for checkpoint to
>>>> complete
>>>> pg_basebackup: checkpoint completed
>>>> pg_basebackup: write-ahead log start point: 0/C1000028 on timeline 1
>>>> pg_basebackup: starting background WAL receiver
>>>> pg_basebackup: created temporary replication slot "pg_basebackup_57337"
>>>> pg_basebackup: backup worker (0) created
>>>> pg_basebackup: backup worker (1) created
>>>> pg_basebackup: backup worker (2) created
>>>> pg_basebackup: backup worker (3) created
>>>> pg_basebackup: backup worker (4) created
>>>> pg_basebackup: backup worker (5) created
>>>> pg_basebackup: backup worker (6) created
>>>> pg_basebackup: backup worker (7) created
>>>> pg_basebackup: error: could not read COPY data: server closed the
>>>> connection unexpectedly
>>>> This probably means the server terminated abnormally
>>>> before or while processing the request.
>>>> pg_basebackup: error: could not read COPY data: server closed the
>>>> connection unexpectedly
>>>> This probably means the server terminated abnormally
>>>> before or while processing the request.
>>>> [edb(at)localhost bin]$
>>>> [edb(at)localhost bin]$
>>>>
>>>> Same case when executed on pg_basebackup without the Parallel backup
>>>> patch then proper clean up is done.
>>>>
>>>> [edb(at)localhost bin]$
>>>> [edb(at)localhost bin]$ ./pg_basebackup -v -D /home/edb/Desktop/backup/
>>>> pg_basebackup: initiating base backup, waiting for checkpoint to
>>>> complete
>>>> pg_basebackup: checkpoint completed
>>>> pg_basebackup: write-ahead log start point: 0/C5000028 on timeline 1
>>>> pg_basebackup: starting background WAL receiver
>>>> pg_basebackup: created temporary replication slot "pg_basebackup_5590"
>>>> pg_basebackup: error: could not read COPY data: server closed the
>>>> connection unexpectedly
>>>> This probably means the server terminated abnormally
>>>> before or while processing the request.
>>>> pg_basebackup: removing contents of data directory
>>>> "/home/edb/Desktop/backup/"
>>>> [edb(at)localhost bin]$
>>>>
>>>> Thanks
>>>>
>>>>
>>>>>
>>>>> On Fri, Apr 3, 2020 at 1:46 PM Asif Rehman <asifr(dot)rehman(at)gmail(dot)com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 2, 2020 at 8:45 PM Robert Haas <robertmhaas(at)gmail(dot)com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Apr 2, 2020 at 11:17 AM Asif Rehman <asifr(dot)rehman(at)gmail(dot)com>
>>>>>>> wrote:
>>>>>>> >> Why would you need to do that? As long as the process where
>>>>>>> >> STOP_BACKUP can do the check, that seems good enough.
>>>>>>> >
>>>>>>> > Yes, but the user will get the error only after the STOP_BACKUP,
>>>>>>> not while the backup is
>>>>>>> > in progress. So if the backup is a large one, early error
>>>>>>> detection would be much beneficial.
>>>>>>> > This is the current behavior of non-parallel backup as well.
>>>>>>>
>>>>>>> Because non-parallel backup does not feature early detection of this
>>>>>>> error, it is not necessary to make parallel backup do so. Indeed, it
>>>>>>> is undesirable. If you want to fix that problem, do it on a separate
>>>>>>> thread in a separate patch. A patch proposing to make parallel backup
>>>>>>> inconsistent in behavior with non-parallel backup will be rejected,
>>>>>>> at
>>>>>>> least if I have anything to say about it.
>>>>>>>
>>>>>>> TBH, fixing this doesn't seem like an urgent problem to me. The
>>>>>>> current situation is not great, but promotions ought to be relatively
>>>>>>> infrequent, so I'm not sure it's a huge problem in practice. It is
>>>>>>> also worth considering whether the right fix is to figure out how to
>>>>>>> make that case actually work, rather than just making it fail
>>>>>>> quicker.
>>>>>>> I don't currently understand the reason for the prohibition so I
>>>>>>> can't
>>>>>>> express an intelligent opinion on what the right answer is here, but
>>>>>>> it seems like it ought to be investigated before somebody goes and
>>>>>>> builds a bunch of infrastructure to make the error more timely.
>>>>>>>
>>>>>>
>>>>>> Non-parallel backup already does the early error checking. I only
>>>>>> intended
>>>>>>
>>>>>> to make parallel behave the same as non-parallel here. So, I agree
>>>>>> with
>>>>>>
>>>>>> you that the behavior of parallel backup should be consistent with the
>>>>>>
>>>>>> non-parallel one. Please see the code snippet below from
>>>>>>
>>>>>> basebackup.c:sendDir()
>>>>>>
>>>>>>
>>>>>> /*
>>>>>>>
>>>>>>> * Check if the postmaster has signaled us to exit, and abort with an
>>>>>>>
>>>>>>> * error in that case. The error handler further up will call
>>>>>>>
>>>>>>> * do_pg_abort_backup() for us. Also check that if the backup was
>>>>>>>
>>>>>>> * started while still in recovery, the server wasn't promoted.
>>>>>>>
>>>>>>> * do_pg_stop_backup() will check that too, but it's better to stop
>>>>>>>
>>>>>>> * the backup early than continue to the end and fail there.
>>>>>>>
>>>>>>> */
>>>>>>>
>>>>>>> CHECK_FOR_INTERRUPTS();
>>>>>>>
>>>>>>> *if* (RecoveryInProgress() != backup_started_in_recovery)
>>>>>>>
>>>>>>> ereport(ERROR,
>>>>>>>
>>>>>>> (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
>>>>>>>
>>>>>>> errmsg("the standby was promoted during online backup"),
>>>>>>>
>>>>>>> errhint("This means that the backup being taken is corrupt "
>>>>>>>
>>>>>>> "and should not be used. "
>>>>>>>
>>>>>>> "Try taking another online backup.")));
>>>>>>>
>>>>>>>
>>>>>>> > Okay, then I will add the shared state. And since we are adding
>>>>>>> the shared state, we can use
>>>>>>> > that for throttling, progress-reporting and standby early error
>>>>>>> checking.
>>>>>>>
>>>>>>> Please propose a grammar here for all the new replication commands
>>>>>>> you
>>>>>>> plan to add before going and implement everything. That will make it
>>>>>>> easier to hash out the design without forcing you to keep changing
>>>>>>> the
>>>>>>> code. Your design should include a sketch of how several sets of
>>>>>>> coordinating backends taking several concurrent parallel backups will
>>>>>>> end up with one shared state per parallel backup.
>>>>>>>
>>>>>>> > There are two possible options:
>>>>>>> >
>>>>>>> > (1) Server may generate a unique ID i.e. BackupID=<unique_string>
>>>>>>> OR
>>>>>>> > (2) (Preferred Option) Use the WAL start location as the BackupID.
>>>>>>> >
>>>>>>> > This BackupID should be given back as a response to start backup
>>>>>>> command. All client workers
>>>>>>> > must append this ID to all parallel backup replication commands.
>>>>>>> So that we can use this identifier
>>>>>>> > to search for that particular backup. Does that sound good?
>>>>>>>
>>>>>>> Using the WAL start location as the backup ID seems like it might be
>>>>>>> problematic -- could a single checkpoint not end up as the start
>>>>>>> location for multiple backups started at the same time? Whether
>>>>>>> that's
>>>>>>> possible now or not, it seems unwise to hard-wire that assumption
>>>>>>> into
>>>>>>> the wire protocol.
>>>>>>>
>>>>>>> I was thinking that perhaps the client should generate a unique
>>>>>>> backup
>>>>>>> ID, e.g. leader does:
>>>>>>>
>>>>>>> START_BACKUP unique_backup_id [options]...
>>>>>>>
>>>>>>> And then others do:
>>>>>>>
>>>>>>> JOIN_BACKUP unique_backup_id
>>>>>>>
>>>>>>> My thought is that you will have a number of shared memory structure
>>>>>>> equal to max_wal_senders, each one large enough to hold the shared
>>>>>>> state for one backup. The shared state will include
>>>>>>> char[NAMEDATALEN-or-something] which will be used to hold the backup
>>>>>>> ID. START_BACKUP would allocate one and copy the name into it;
>>>>>>> JOIN_BACKUP would search for one by name.
>>>>>>>
>>>>>>> If you want to generate the name on the server side, then I suppose
>>>>>>> START_BACKUP would return a result set that includes the backup ID,
>>>>>>> and clients would have to specify that same backup ID when invoking
>>>>>>> JOIN_BACKUP. The rest would stay the same. I am not sure which way is
>>>>>>> better. Either way, the backup ID should be something long and hard
>>>>>>> to
>>>>>>> guess, not e.g. the leader processes' PID. I think we should generate
>>>>>>> it using pg_strong_random, say 8 or 16 bytes, and then hex-encode the
>>>>>>> result to get a string. That way there's almost no risk of two backup
>>>>>>> IDs colliding accidentally, and even if we somehow had a malicious
>>>>>>> user trying to screw up somebody else's parallel backup by choosing a
>>>>>>> colliding backup ID, it would be pretty hard to have any success. A
>>>>>>> user with enough access to do that sort of thing can probably cause a
>>>>>>> lot worse problems anyway, but it seems pretty easy to guard against
>>>>>>> intentional collisions robustly here, so I think we should.
>>>>>>>
>>>>>>>
>>>>>> Okay so If we are to add another replication command ‘JOIN_BACKUP
>>>>>> unique_backup_id’
>>>>>> to make workers find the relevant shared state. There won't be any
>>>>>> need for changing
>>>>>> the grammar for any other command. The START_BACKUP can return the
>>>>>> unique_backup_id
>>>>>> in the result set.
>>>>>>
>>>>>> I am thinking of the following struct for shared state:
>>>>>>
>>>>>>> *typedef* *struct*
>>>>>>>
>>>>>>> {
>>>>>>>
>>>>>>> *char* backupid[NAMEDATALEN];
>>>>>>>
>>>>>>> XLogRecPtr startptr;
>>>>>>>
>>>>>>>
>>>>>>> slock_t lock;
>>>>>>>
>>>>>>> int64 throttling_counter;
>>>>>>>
>>>>>>> *bool* backup_started_in_recovery;
>>>>>>>
>>>>>>> } BackupSharedState;
>>>>>>>
>>>>>>>
>>>>>> The shared state structure entries would be maintained by a shared
>>>>>> hash table.
>>>>>> There will be one structure per parallel backup. Since a single
>>>>>> parallel backup
>>>>>> can engage more than one wal sender, so I think max_wal_senders might
>>>>>> be a little
>>>>>> too much; perhaps max_wal_senders/2 since there will be at least 2
>>>>>> connections
>>>>>> per parallel backup? Alternatively, we can set a new GUC that defines
>>>>>> the maximum
>>>>>> number of for concurrent parallel backups i.e.
>>>>>> ‘max_concurent_backups_allowed = 10’
>>>>>> perhaps, or we can make it user-configurable.
>>>>>>
>>>>>> The key would be “backupid=hex_encode(pg_random_strong(16))”
>>>>>>
>>>>>> Checking for Standby Promotion:
>>>>>> At the START_BACKUP command, we initialize
>>>>>> BackupSharedState.backup_started_in_recovery
>>>>>> and keep checking it whenever send_file () is called to send a new
>>>>>> file.
>>>>>>
>>>>>> Throttling:
>>>>>> BackupSharedState.throttling_counter - The throttling logic remains
>>>>>> the same
>>>>>> as for non-parallel backup with the exception that multiple threads
>>>>>> will now be
>>>>>> updating it. So in parallel backup, this will represent the overall
>>>>>> bytes that
>>>>>> have been transferred. So the workers would sleep if they have
>>>>>> exceeded the
>>>>>> limit. Hence, the shared state carries a lock to safely update the
>>>>>> throttling
>>>>>> value atomically.
>>>>>>
>>>>>> Progress Reporting:
>>>>>> Although I think we should add progress-reporting for parallel backup
>>>>>> as a
>>>>>> separate patch. The relevant entries for progress-reporting such as
>>>>>> ‘backup_total’ and ‘backup_streamed’ would be then added to this
>>>>>> structure
>>>>>> as well.
>>>>>>
>>>>>>
>>>>>> Grammar:
>>>>>> There is a change in the resultset being returned for START_BACKUP
>>>>>> command;
>>>>>> unique_backup_id is added. Additionally, JOIN_BACKUP replication
>>>>>> command is
>>>>>> added. SEND_FILES has been renamed to SEND_FILE. There are no other
>>>>>> changes
>>>>>> to the grammar.
>>>>>>
>>>>>> START_BACKUP [LABEL '<label>'] [FAST]
>>>>>> - returns startptr, tli, backup_label, unique_backup_id
>>>>>> STOP_BACKUP [NOWAIT]
>>>>>> - returns startptr, tli, backup_label
>>>>>> JOIN_BACKUP ‘unique_backup_id’
>>>>>> - attaches a shared state identified by ‘unique_backup_id’ to a
>>>>>> backend process.
>>>>>>
>>>>>> LIST_TABLESPACES [PROGRESS]
>>>>>> LIST_FILES [TABLESPACE]
>>>>>> LIST_WAL_FILES [START_WAL_LOCATION 'X/X'] [END_WAL_LOCATION 'X/X']
>>>>>> SEND_FILE '(' FILE ')' [NOVERIFY_CHECKSUMS]
>>>>>>
>>>>>>
>>
>
> Hi,
>
> rebased and updated to the current master (8128b0c1). v13 is attached.
>
> - Fixes the above reported issues.
>
> - Added progress-reporting support for parallel:
> For this, 'backup_streamed' is moved to a shared structure (BackupState) as
> pg_atomic_uint64 variable. The worker processes will keep incrementing this
> variable.
>
> While files are being transferred from server to client. The main process
> remains
> in an idle state. So after each increment, the worker process will signal
> master to
> update the stats in pg_stat_progress_basebackup view.
>
> The 'tablespace_streamed' column is not updated and will remain empty.
> This is
> because multiple workers may be copying files from different tablespaces.
>
>
> - Added backup manifest:
> The backend workers maintain their own manifest file which contains a list
> of files
> that are being transferred by the work. Once all backup files are
> transferred, the
> workers will create a temp file as
> ('pg_tempdir/temp_file_prefix_backupid.workerid')
> to write the content of the manifest file from BufFile. The workers won’t
> add the
> header, nor the WAL information in their manifest. These two will be added
> by the
> main process while merging all worker manifest files.
>
> The main process will read these individual files and concatenate them
> into a single file
> which is then sent back to the client.
>
> The manifest file is created when the following command is received:
>
>> BUILD_MANIFEST 'backupid'
>
>
> This is a new replication command. It is sent when pg_basebackup has
> copied all the
> $PGDATA files including WAL files.
>
>
>
> --
> Asif Rehman
> Highgo Software (Canada/China/Pakistan)
> URL : www.highgo.ca
>
>

--
Regards
====================================
Kashif Zeeshan
Lead Quality Assurance Engineer / Manager

EnterpriseDB Corporation
The Enterprise Postgres Company