Re: Add index scan progress to pg_stat_progress_vacuum

> I'm still unsure the current design of 0001 patch is better than other
> approaches we’ve discussed. Even users who don't use parallel vacuum
> are forced to allocate shared memory for index vacuum progress, with
> GetMaxBackends() entries from the beginning. Also, it’s likely to
> extend the progress tracking feature for other parallel operations in
> the future but I think the current design is not extensible. If we
> want to do that, we will end up creating similar things for each of
> them or re-creating index vacuum progress tracking feature while
> creating a common infra. It might not be a problem as of now but I'm
> concerned that introducing a feature that is not extensible and forces
> users to allocate additional shmem might be a blocker in the future.
> Looking at the precedent example, When we introduce the progress
> tracking feature, we implemented it in an extensible way. On the other
> hand, others in this thread seem to agree with this approach, so I'd
> like to leave it to committers.

Thanks for the review!

I think you make strong arguments as to why we need to take a different approach now than later.

Flaws with current patch set:

1. GetMaxBackends() is a really heavy-handed overallocation of a shared memory serving a very specific purpose.
2. Going with the approach of a vacuum specific hash breaks the design of progress which is meant to be extensible.
3. Even if we go with this current approach as an interim solution, it will be a real pain in the future.

With that said, v7 introduces the new infrastructure. 0001 includes the new infrastructure and 0002 takes advantage of this.

This approach is the following:

1. Introduces a new API called pgstat_progress_update_param_parallel along with some others support functions. This new infrastructure is in backend_progress.c

2. There is still a shared memory involved, but the size is capped to " max_worker_processes" which is the max to how many parallel workers can be doing work at any given time. The shared memory hash includes a st_progress_param array just like the Backend Status array.

typedef struct ProgressParallelEntry
{
pid_t leader_pid;
int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
} ProgressParallelEntry;

3. The progress update function is "pgstat_progress_update_param_parallel" and will aggregate totals reported for a specific progress parameter

For example , it can be called lie below. In the case below, PROGRESS_VACUUM_INDEXES_COMPLETED is incremented by 1 in the shared memory entry shared by the workers and leader.

case PARALLEL_INDVAC_STATUS_NEED_BULKDELETE:
istat_res = vac_bulkdel_one_index(&ivinfo, istat, pvs->dead_items);
pgstat_progress_update_param_parallel(pvs->shared->leader_pid, PROGRESS_VACUUM_INDEXES_COMPLETED, 1); <<-----
break;

4. pg_stat_get_progress_info will call a function called pgstat_progress_set_parallel which will set the parameter value to the total from the shared memory hash.

I believe this approach gives proper infrastructure for future use-cases of workers reporting progress -and- does not do the heavy-handed shared memory allocation.

--
Sami Imseih
Amazon Web Services