From 21ebf04efe2f19794161d460302d3c8491dd6732 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Tue, 16 Jun 2026 10:51:25 +0300 Subject: [PATCH 5/8] Extended vacuum statistics: buffer/WAL/timing machinery and total buffer counters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce the resource-usage sampling machinery for extended vacuum statistics, the database-wide aggregate, and the first counters that depend on the sampling. This commit introduces the resource-usage sampling that the buffer, WAL and timing metrics rely on. Around the processing of each relation, vacuum snapshots WAL, buffer and timing usage and records the difference, so every sampled metric reflects only that vacuum's work. Index processing is sampled on its own — including in parallel workers — and its usage is kept out of the parent table's figures, so heap and index work are accounted separately. The PGSTAT_KIND_VACUUM_DB stats kind is added here as well: it aggregates the common (buffer/WAL/timing) counters of every relation vacuumed in the database. pgstat_accumulate_common() -- which sums those counters -- is introduced together with it, since it is only meaningful once the sampling exists. This commit also creates the two remaining views, pg_stat_vacuum_indexes and pg_stat_vacuum_database, alongside pg_stat_vacuum_tables. They are introduced here because they expose the common (buffer/WAL/timing) counters produced by the sampling machinery; like the table view, they then grow column by column in the following commits as each metric category is added. The per-index view also exposes the index-specific pages_deleted and tuples_deleted counters from the start. This commit exposes the shared-buffer access counters in all three views, with documentation and regression coverage: total_blks_read shared buffer blocks missed (read from disk) total_blks_hit shared buffer blocks found in the buffer cache (hits) total_blks_dirtied shared buffer blocks dirtied by this vacuum total_blks_written shared buffer blocks written out These counters are common to tables, indexes and the database aggregate (where the first two are named db_blks_read/db_blks_hit); total_blks_dirtied counts only the pages dirtied by this vacuum. --- doc/src/sgml/system-views.sgml | 245 +++++++++++ src/backend/access/heap/vacuumlazy.c | 426 +++++++++++++++---- src/backend/catalog/system_views.sql | 28 +- src/backend/commands/dbcommands.c | 1 + src/backend/commands/vacuum.c | 4 + src/backend/commands/vacuumparallel.c | 1 + src/backend/utils/activity/pgstat.c | 15 + src/backend/utils/activity/pgstat_database.c | 9 + src/backend/utils/activity/pgstat_vacuum.c | 106 ++++- src/backend/utils/adt/pgstatfuncs.c | 53 ++- src/include/catalog/pg_proc.dat | 21 +- src/include/commands/vacuum.h | 29 ++ src/include/pgstat.h | 16 + src/include/utils/pgstat_internal.h | 7 + src/include/utils/pgstat_kind.h | 3 +- src/test/regress/expected/rules.out | 25 +- src/test/regress/expected/vacuum_stats.out | 45 +- src/test/regress/sql/vacuum_stats.sql | 31 +- 18 files changed, 945 insertions(+), 120 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 704d9e85eb..088b532e5c 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -5916,6 +5916,251 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx Number of heap pages newly marked both all-visible and all-frozen in the visibility map by the vacuum. + + + total_blks_read bigint + + + Number of shared buffer blocks missed (read from disk) by the vacuum. + + + + + total_blks_hit bigint + + + Number of shared buffer blocks found in the buffer cache (hits) by the vacuum. + + + + + total_blks_dirtied bigint + + + Number of shared buffer blocks dirtied by the vacuum. + + + + + total_blks_written bigint + + + Number of shared buffer blocks written out by the vacuum. + + + + + + + + + <structname>pg_stat_vacuum_indexes</structname> + + + pg_stat_vacuum_indexes + + + + The pg_stat_vacuum_indexes view will contain one row + for each index in the current database, showing extended statistics about + the activity of the most recent VACUUM on that index. + These statistics are accumulated only while + track_vacuum_statistics is enabled. + + + + <structname>pg_stat_vacuum_indexes</structname> Columns + + + + + Column Type + + + Description + + + + + + + + relid oid + + + OID of the table the index is on. + + + + + indexrelid oid + + + OID of the index. + + + + + schemaname name + + + Name of the schema that the table is in. + + + + + relname name + + + Name of the table. + + + + + indexrelname name + + + Name of the index. + + + + + pages_deleted bigint + + + Number of index pages deleted (made reusable) by the vacuum. + + + + + tuples_deleted bigint + + + Number of index entries removed by the vacuum. + + + + + total_blks_read bigint + + + Number of shared buffer blocks read while vacuuming the index. + + + + + total_blks_hit bigint + + + Number of shared buffer block hits while vacuuming the index. + + + + + total_blks_dirtied bigint + + + Number of shared buffer blocks dirtied while vacuuming the index. + + + + + total_blks_written bigint + + + Number of shared buffer blocks written out while vacuuming the index. + + + + +
+ + + + <structname>pg_stat_vacuum_database</structname> + + + pg_stat_vacuum_database + + + + The pg_stat_vacuum_database view will contain one row + for each database, showing extended statistics aggregated over all + VACUUM activity in that database. These statistics are + accumulated only while track_vacuum_statistics is enabled. + + + + <structname>pg_stat_vacuum_database</structname> Columns + + + + + Column Type + + + Description + + + + + + + + dboid oid + + + OID of the database. + + + + + dbname name + + + Name of the database. + + + + + errors integer + + + Number of vacuum operations in this database that failed with an error. + + + + + db_blks_read bigint + + + Number of shared buffer blocks read by vacuum operations in this database. + + + + + db_blks_hit bigint + + + Number of shared buffer block hits by vacuum operations in this database. + + + + + total_blks_dirtied bigint + + + Number of shared buffer blocks dirtied by vacuum operations in this database. + + + + + total_blks_written bigint + + + Number of shared buffer blocks written out by vacuum operations in this database. + +
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 91b6d9a9d9..9c524909fe 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -411,6 +411,25 @@ typedef struct LVRelState * been permanently disabled. */ BlockNumber eager_scan_remaining_fails; + + int32 wraparound_failsafe_count; /* number of emergency vacuums to + * prevent anti-wraparound + * shutdown */ + + PgStat_VacuumRelationCounts extVacReportIdx; + + /* + * Per-index accumulated extended vacuum statistics. VACUUM may process an + * index several times (a bulkdelete pass per index scan plus a final + * cleanup pass); we sum those passes here and report them to the + * cumulative stats system exactly once per index at the end of the vacuum. + * Only used by the non-parallel path -- parallel index vacuuming keeps the + * equivalent running totals in the DSM and reports from + * parallel_vacuum_end(). Sized vacrel->nindexes; NULL if there are no + * indexes. + */ + PgStat_VacuumRelationCounts *extVacIdxReports; + bool *extVacIdxTouched; /* was the matching index processed? */ } LVRelState; @@ -422,7 +441,6 @@ typedef struct LVSavedErrInfo VacErrPhase phase; } LVSavedErrInfo; - /* non-export function prototypes */ static void lazy_scan_heap(LVRelState *vacrel); static void heap_vacuum_eager_scan_setup(LVRelState *vacrel, @@ -452,12 +470,12 @@ static void lazy_cleanup_all_indexes(LVRelState *vacrel); static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat, double reltuples, - LVRelState *vacrel); + LVRelState *vacrel, int idx); static IndexBulkDeleteResult *lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat, double reltuples, bool estimated_count, - LVRelState *vacrel); + LVRelState *vacrel, int idx); static bool should_attempt_truncation(LVRelState *vacrel); static void lazy_truncate_heap(LVRelState *vacrel); static BlockNumber count_nondeletable_pages(LVRelState *vacrel, @@ -484,9 +502,223 @@ static void update_vacuum_error_info(LVRelState *vacrel, OffsetNumber offnum); static void restore_vacuum_error_info(LVRelState *vacrel, const LVSavedErrInfo *saved_vacrel); -static void accumulate_heap_vacuum_statistics(LVRelState *vacrel, - PgStat_VacuumRelationCounts *extVacStats); +/* ---------- + * extvac_stats_start() - + * + * Save cut-off values of extended vacuum counters before start of a relation + * processing. + * ---------- + */ +static void +extvac_stats_start(Relation rel, LVExtStatCounters * counters) +{ + TimestampTz starttime; + + if (!pgstat_track_vacuum_statistics) + return; + + memset(counters, 0, sizeof(LVExtStatCounters)); + + starttime = GetCurrentTimestamp(); + + counters->starttime = starttime; + counters->walusage = pgWalUsage; + counters->bufusage = pgBufferUsage; + counters->VacuumDelayTime = VacuumDelayTime; + counters->blocks_fetched = 0; + counters->blocks_hit = 0; + + if (!rel->pgstat_info || !pgstat_track_counts) + + /* + * if something goes wrong or user doesn't want to track a database + * activity - just suppress it. + */ + return; + + counters->blocks_fetched = rel->pgstat_info->counts.blocks_fetched; + counters->blocks_hit = rel->pgstat_info->counts.blocks_hit; +} + +/* ---------- + * extvac_stats_end() - + * + * Called to finish an extended vacuum statistic gathering and form a report. + * ---------- + */ +static void +extvac_stats_end(Relation rel, LVExtStatCounters * counters, + PgStat_CommonCounts * report) +{ + BufferUsage bufusage; + + if (!pgstat_track_vacuum_statistics) + return; + + memset(report, 0, sizeof(PgStat_CommonCounts)); + + /* Calculate diffs of global stat parameters on buffer usage. */ + memset(&bufusage, 0, sizeof(BufferUsage)); + BufferUsageAccumDiff(&bufusage, &pgBufferUsage, &counters->bufusage); + + /* + * Fill additional statistics on a vacuum processing operation. + */ + report->total_blks_read += bufusage.local_blks_read + bufusage.shared_blks_read; + report->total_blks_hit += bufusage.local_blks_hit + bufusage.shared_blks_hit; + report->total_blks_dirtied += bufusage.local_blks_dirtied + bufusage.shared_blks_dirtied; + report->total_blks_written += bufusage.shared_blks_written; +} + +void +extvac_stats_start_idx(Relation rel, IndexBulkDeleteResult *stats, + LVExtStatCountersIdx * counters) +{ + if (!pgstat_track_vacuum_statistics) + return; + + /* Set initial values for common heap and index statistics */ + extvac_stats_start(rel, &counters->common); + counters->pages_deleted = counters->tuples_removed = 0; + + if (stats != NULL) + { + /* + * XXX: Why do we need this code here? If it is needed, I feel lack of + * comments, describing the reason. + */ + counters->tuples_removed = stats->tuples_removed; + counters->pages_deleted = stats->pages_deleted; + } +} + +void +extvac_stats_end_idx(Relation rel, IndexBulkDeleteResult *stats, + LVExtStatCountersIdx * counters, PgStat_VacuumRelationCounts * report) +{ + if (!pgstat_track_vacuum_statistics) + return; + + memset(report, 0, sizeof(PgStat_VacuumRelationCounts)); + + extvac_stats_end(rel, &counters->common, &report->common); + + report->type = PGSTAT_EXTVAC_INDEX; + + if (stats != NULL) + { + /* + * if something goes wrong or an user doesn't want to track a database + * activity - just suppress it. + */ + + /* Fill index-specific extended stats fields */ + report->common.tuples_deleted = + stats->tuples_removed - counters->tuples_removed; + report->index.pages_deleted = + stats->pages_deleted - counters->pages_deleted; + } +} + +/* Accumulate vacuum statistics for heap. + * + * Because of complexity of vacuum processing: it switch procesing between + * the heap relation to index relations and visa versa, we need to store + * gathered statistics information for heap relations several times before + * the vacuum starts processing the indexes again. + * + * It is necessary to gather correct statistics information for heap and indexes + * otherwice the index statistics information would be added to his parent heap + * statistics information and it would be difficult to analyze it later. + * + * We can't subtract union vacuum statistics information for index from the heap relations + * because of total and delay time time statistics collecting during parallel vacuum + * procudure. +*/ +static void +accumulate_heap_vacuum_statistics(LVRelState *vacrel, PgStat_VacuumRelationCounts * extVacStats) +{ + if (!pgstat_track_vacuum_statistics) + return; + + /* Fill heap-specific extended stats fields */ + extVacStats->type = PGSTAT_EXTVAC_TABLE; + extVacStats->table.pages_scanned = vacrel->scanned_pages; + extVacStats->table.pages_removed = vacrel->removed_pages; + extVacStats->table.vm_new_frozen_pages = vacrel->new_all_frozen_pages; + extVacStats->table.vm_new_visible_pages = vacrel->new_all_visible_pages; + extVacStats->table.vm_new_visible_frozen_pages = vacrel->new_all_visible_all_frozen_pages; + extVacStats->common.tuples_deleted = vacrel->tuples_deleted; + extVacStats->table.tuples_frozen = vacrel->tuples_frozen; + extVacStats->table.recently_dead_tuples = vacrel->recently_dead_tuples; + extVacStats->table.missed_dead_tuples = vacrel->missed_dead_tuples; + extVacStats->table.missed_dead_pages = vacrel->missed_dead_pages; + + extVacStats->common.total_blks_dirtied -= vacrel->extVacReportIdx.common.total_blks_dirtied; + extVacStats->common.total_blks_hit -= vacrel->extVacReportIdx.common.total_blks_hit; + extVacStats->common.total_blks_read -= vacrel->extVacReportIdx.common.total_blks_read; + extVacStats->common.total_blks_written -= vacrel->extVacReportIdx.common.total_blks_written; +} + +static void +accumulate_idxs_vacuum_statistics(LVRelState *vacrel, PgStat_VacuumRelationCounts * extVacIdxStats) +{ + /* Fill heap-specific extended stats fields */ + vacrel->extVacReportIdx.common.total_blks_dirtied += extVacIdxStats->common.total_blks_dirtied; + vacrel->extVacReportIdx.common.total_blks_hit += extVacIdxStats->common.total_blks_hit; + vacrel->extVacReportIdx.common.total_blks_read += extVacIdxStats->common.total_blks_read; + vacrel->extVacReportIdx.common.total_blks_written += extVacIdxStats->common.total_blks_written; +} + +/* + * Accumulate one index extended-vacuum report into a per-index running total. + * + * VACUUM may touch an index more than once: a bulkdelete pass for every index + * scan, plus a final cleanup pass. Rather than reporting to the cumulative + * stats system on every pass, callers sum the passes here and report the + * totals exactly once per index at the end of the vacuum. Shared between the + * non-parallel path (accumulating in vacrel) and the parallel path + * (accumulating in the DSM), hence not static. + */ +void +extvac_accumulate_idx_report(PgStat_VacuumRelationCounts * dst, + const PgStat_VacuumRelationCounts * src) +{ + dst->type = PGSTAT_EXTVAC_INDEX; + + dst->common.total_blks_read += src->common.total_blks_read; + dst->common.total_blks_hit += src->common.total_blks_hit; + dst->common.total_blks_dirtied += src->common.total_blks_dirtied; + dst->common.total_blks_written += src->common.total_blks_written; + dst->common.tuples_deleted += src->common.tuples_deleted; + + dst->index.pages_deleted += src->index.pages_deleted; +} + +/* + * Report the accumulated per-index extended vacuum statistics, one report per + * index. Used by the non-parallel path only; the parallel path reports its + * DSM-resident totals from parallel_vacuum_end(). + */ +static void +report_index_vacuum_extstats(LVRelState *vacrel) +{ + if (!pgstat_track_vacuum_statistics) + return; + + for (int idx = 0; idx < vacrel->nindexes; idx++) + { + Relation indrel = vacrel->indrels[idx]; + + if (!vacrel->extVacIdxTouched[idx]) + continue; + + pgstat_report_vacuum_extstats(RelationGetRelid(indrel), + indrel->rd_rel->relisshared, + &vacrel->extVacIdxReports[idx]); + } +} /* @@ -612,69 +844,6 @@ heap_vacuum_eager_scan_setup(LVRelState *vacrel, const VacuumParams *params) first_region_ratio; } -/* - * Fill the extended vacuum statistics report for a heap relation with the - * counters that are derived directly from the LVRelState. Buffer, WAL and - * timing counters require sampling resource usage around the vacuum and are - * gathered separately; they are not touched here. - */ -static void -accumulate_heap_vacuum_statistics(LVRelState *vacrel, - PgStat_VacuumRelationCounts *extVacStats) -{ - if (!pgstat_track_vacuum_statistics) - return; - - extVacStats->type = PGSTAT_EXTVAC_TABLE; - extVacStats->table.pages_scanned = vacrel->scanned_pages; - extVacStats->table.pages_removed = vacrel->removed_pages; - extVacStats->common.tuples_deleted = vacrel->tuples_deleted; - extVacStats->table.tuples_frozen = vacrel->tuples_frozen; - extVacStats->table.recently_dead_tuples = vacrel->recently_dead_tuples; - extVacStats->table.missed_dead_pages = vacrel->missed_dead_pages; - extVacStats->table.missed_dead_tuples = vacrel->missed_dead_tuples; - extVacStats->table.vm_new_frozen_pages = vacrel->new_all_frozen_pages; - extVacStats->table.vm_new_visible_pages = vacrel->new_all_visible_pages; - extVacStats->table.vm_new_visible_frozen_pages = vacrel->new_all_visible_all_frozen_pages; -} - -/* - * Report the per-index extended vacuum statistics, one report per index. - * - * The per-index counters (pages_deleted and the number of removed index - * entries) are derived directly from each index's final IndexBulkDeleteResult, - * which already holds the totals accumulated across all bulkdelete and cleanup - * passes -- so no per-pass sampling is needed here. Used by the non-parallel - * path only; the parallel path reports its DSM-resident results from - * parallel_vacuum_end(). - */ -static void -report_index_vacuum_extstats(LVRelState *vacrel) -{ - if (!pgstat_track_vacuum_statistics) - return; - - for (int idx = 0; idx < vacrel->nindexes; idx++) - { - Relation indrel = vacrel->indrels[idx]; - IndexBulkDeleteResult *istat = vacrel->indstats[idx]; - PgStat_VacuumRelationCounts report; - - /* Skip indexes that this vacuum did not process */ - if (istat == NULL) - continue; - - memset(&report, 0, sizeof(report)); - report.type = PGSTAT_EXTVAC_INDEX; - report.common.tuples_deleted = istat->tuples_removed; - report.index.pages_deleted = istat->pages_deleted; - - pgstat_report_vacuum_extstats(RelationGetRelid(indrel), - indrel->rd_rel->relisshared, - &report); - } -} - /* * heap_vacuum_rel() -- perform VACUUM for one heap relation * @@ -701,7 +870,6 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, new_rel_allvisible, new_rel_allfrozen; PGRUsage ru0; - TimestampTz starttime = 0; PgStat_Counter startreadtime = 0, startwritetime = 0; WalUsage startwalusage = pgWalUsage; @@ -709,9 +877,10 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, ErrorContextCallback errcallback; char **indnames = NULL; Size dead_items_max_bytes = 0; + LVExtStatCounters extVacCounters; PgStat_VacuumRelationCounts extVacReport; - /* Initialize the extended vacuum statistics report */ + /* Initialize vacuum statistics */ memset(&extVacReport, 0, sizeof(PgStat_VacuumRelationCounts)); verbose = (params->options & VACOPT_VERBOSE) != 0; @@ -727,8 +896,7 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, } } - /* Used for instrumentation and stats report */ - starttime = GetCurrentTimestamp(); + extvac_stats_start(rel, &extVacCounters); pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM, RelationGetRelid(rel)); @@ -756,8 +924,8 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, vacrel = palloc0_object(LVRelState); vacrel->dbname = get_database_name(MyDatabaseId); vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel)); - vacrel->reloid = RelationGetRelid(rel); vacrel->relname = pstrdup(RelationGetRelationName(rel)); + vacrel->reloid = RelationGetRelid(rel); vacrel->indname = NULL; vacrel->phase = VACUUM_ERRCB_PHASE_UNKNOWN; vacrel->verbose = verbose; @@ -766,12 +934,26 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, errcallback.previous = error_context_stack; error_context_stack = &errcallback; + memset(&vacrel->extVacReportIdx, 0, sizeof(PgStat_VacuumRelationCounts)); + memset(&extVacReport.common, 0, sizeof(PgStat_CommonCounts)); + /* Set up high level stuff about rel and its indexes */ vacrel->rel = rel; vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes, &vacrel->indrels); vacrel->bstrategy = bstrategy; + /* + * Allocate per-index accumulators for extended vacuum statistics. Index + * passes add into these and the totals are reported once per index at the + * end of the vacuum (see report_index_vacuum_extstats()). + */ + if (vacrel->nindexes > 0) + { + vacrel->extVacIdxReports = + palloc0_array(PgStat_VacuumRelationCounts, vacrel->nindexes); + vacrel->extVacIdxTouched = palloc0_array(bool, vacrel->nindexes); + } if (instrument && vacrel->nindexes > 0) { /* Copy index names used by instrumentation (not error reporting) */ @@ -874,6 +1056,7 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs); vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel); vacrel->vistest = GlobalVisTestFor(rel); + vacrel->wraparound_failsafe_count = 0; /* Initialize state used to track oldest extant XID/MXID */ vacrel->NewRelfrozenXid = vacrel->cutoffs.OldestXmin; @@ -1057,14 +1240,19 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, * soon in cases where the failsafe prevented significant amounts of heap * vacuuming. */ + + /* + * Make generic extended vacuum stats report and fill heap-specific + * extended stats fields. + */ + extvac_stats_end(vacrel->rel, &extVacCounters, &extVacReport.common); accumulate_heap_vacuum_statistics(vacrel, &extVacReport); - pgstat_report_vacuum_extstats(vacrel->reloid, rel->rd_rel->relisshared, - &extVacReport); + pgstat_report_vacuum_extstats(vacrel->reloid, rel->rd_rel->relisshared, &extVacReport); pgstat_report_vacuum(rel, Max(vacrel->new_live_tuples, 0), vacrel->recently_dead_tuples + vacrel->missed_dead_tuples, - starttime); + extVacCounters.starttime); pgstat_progress_end_command(); if (instrument) @@ -1072,7 +1260,7 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, TimestampTz endtime = GetCurrentTimestamp(); if (verbose || params->log_vacuum_min_duration == 0 || - TimestampDifferenceExceeds(starttime, endtime, + TimestampDifferenceExceeds(extVacCounters.starttime, endtime, params->log_vacuum_min_duration)) { long secs_dur; @@ -1088,7 +1276,7 @@ heap_vacuum_rel(Relation rel, const VacuumParams *params, int64 total_blks_read; int64 total_blks_dirtied; - TimestampDifference(starttime, endtime, &secs_dur, &usecs_dur); + TimestampDifference(extVacCounters.starttime, endtime, &secs_dur, &usecs_dur); memset(&walusage, 0, sizeof(WalUsage)); WalUsageAccumDiff(&walusage, &pgWalUsage, &startwalusage); memset(&bufferusage, 0, sizeof(BufferUsage)); @@ -1698,7 +1886,8 @@ lazy_scan_heap(LVRelState *vacrel) * Report the per-index extended vacuum statistics accumulated over all * bulkdelete and cleanup passes, exactly once per index. The parallel * path reports its DSM-resident totals from parallel_vacuum_end() instead, - * so only do it here when index vacuuming ran in the leader. + * so only do it here when index vacuuming ran in the leader. pvs is still + * alive at this point (it is torn down later in dead_items_cleanup()). */ if (vacrel->nindexes > 0 && !ParallelVacuumIsActive(vacrel)) report_index_vacuum_extstats(vacrel); @@ -2619,7 +2808,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel) vacrel->indstats[idx] = lazy_vacuum_one_index(indrel, istat, old_live_tuples, - vacrel); + vacrel, idx); /* Report the number of indexes vacuumed */ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_PROCESSED, @@ -2635,11 +2824,21 @@ lazy_vacuum_all_indexes(LVRelState *vacrel) } else { + LVExtStatCounters counters; + PgStat_VacuumRelationCounts extVacReport; + + memset(&extVacReport.common, 0, sizeof(PgStat_CommonCounts)); + + extvac_stats_start(vacrel->rel, &counters); + /* Outsource everything to parallel variant */ parallel_vacuum_bulkdel_all_indexes(vacrel->pvs, old_live_tuples, vacrel->num_index_scans, &(vacrel->worker_usage.vacuum)); + extvac_stats_end(vacrel->rel, &counters, &extVacReport.common); + accumulate_idxs_vacuum_statistics(vacrel, &extVacReport); + /* * Do a postcheck to consider applying wraparound failsafe now. Note * that parallel VACUUM only gets the precheck and this postcheck. @@ -2987,6 +3186,7 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel) int64 progress_val[3] = {0, 0, PROGRESS_VACUUM_MODE_FAILSAFE}; VacuumFailsafeActive = true; + vacrel->wraparound_failsafe_count++; /* * Abandon use of a buffer access strategy to allow use of all of @@ -3060,7 +3260,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel) vacrel->indstats[idx] = lazy_cleanup_one_index(indrel, istat, reltuples, - estimated_count, vacrel); + estimated_count, vacrel, idx); /* Report the number of indexes cleaned up */ pgstat_progress_update_param(PROGRESS_VACUUM_INDEXES_PROCESSED, @@ -3069,11 +3269,21 @@ lazy_cleanup_all_indexes(LVRelState *vacrel) } else { + LVExtStatCounters counters; + PgStat_VacuumRelationCounts extVacReport; + + memset(&extVacReport.common, 0, sizeof(PgStat_CommonCounts)); + + extvac_stats_start(vacrel->rel, &counters); + /* Outsource everything to parallel variant */ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, reltuples, vacrel->num_index_scans, estimated_count, &(vacrel->worker_usage.cleanup)); + + extvac_stats_end(vacrel->rel, &counters, &extVacReport.common); + accumulate_idxs_vacuum_statistics(vacrel, &extVacReport); } /* Reset the progress counters */ @@ -3095,10 +3305,22 @@ lazy_cleanup_all_indexes(LVRelState *vacrel) */ static IndexBulkDeleteResult * lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat, - double reltuples, LVRelState *vacrel) + double reltuples, LVRelState *vacrel, int idx) { IndexVacuumInfo ivinfo; LVSavedErrInfo saved_err_info; + LVExtStatCountersIdx extVacCounters; + PgStat_VacuumRelationCounts extVacReport; + + /* + * Zero the report up front: extvac_stats_end_idx() leaves it untouched + * when statistics tracking is disabled, but the accumulators below read it + * unconditionally. + */ + memset(&extVacReport, 0, sizeof(PgStat_VacuumRelationCounts)); + + /* Set initial statistics values to gather vacuum statistics for the index */ + extvac_stats_start_idx(indrel, istat, &extVacCounters); ivinfo.index = indrel; ivinfo.heaprel = vacrel->rel; @@ -3125,6 +3347,19 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat, istat = vac_bulkdel_one_index(&ivinfo, istat, vacrel->dead_items, vacrel->dead_items_info); + /* Make extended vacuum stats report for index */ + extvac_stats_end_idx(indrel, istat, &extVacCounters, &extVacReport); + + /* This index's time must be excluded from the parent heap's totals */ + accumulate_idxs_vacuum_statistics(vacrel, &extVacReport); + + /* + * Accumulate this pass into the index's running totals. They are reported + * to the cumulative stats system once per index at the end of the vacuum. + */ + extvac_accumulate_idx_report(&vacrel->extVacIdxReports[idx], &extVacReport); + vacrel->extVacIdxTouched[idx] = true; + /* Revert to the previous phase information for error traceback */ restore_vacuum_error_info(vacrel, &saved_err_info); pfree(vacrel->indname); @@ -3145,10 +3380,22 @@ lazy_vacuum_one_index(Relation indrel, IndexBulkDeleteResult *istat, static IndexBulkDeleteResult * lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat, double reltuples, bool estimated_count, - LVRelState *vacrel) + LVRelState *vacrel, int idx) { IndexVacuumInfo ivinfo; LVSavedErrInfo saved_err_info; + LVExtStatCountersIdx extVacCounters; + PgStat_VacuumRelationCounts extVacReport; + + /* + * Zero the report up front: extvac_stats_end_idx() leaves it untouched + * when statistics tracking is disabled, but the accumulators below read it + * unconditionally. + */ + memset(&extVacReport, 0, sizeof(PgStat_VacuumRelationCounts)); + + /* Set initial statistics values to gather vacuum statistics for the index */ + extvac_stats_start_idx(indrel, istat, &extVacCounters); ivinfo.index = indrel; ivinfo.heaprel = vacrel->rel; @@ -3174,6 +3421,19 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat, istat = vac_cleanup_one_index(&ivinfo, istat); + /* Make extended vacuum stats report for index */ + extvac_stats_end_idx(indrel, istat, &extVacCounters, &extVacReport); + + /* This index's time must be excluded from the parent heap's totals */ + accumulate_idxs_vacuum_statistics(vacrel, &extVacReport); + + /* + * Accumulate this pass into the index's running totals. They are reported + * to the cumulative stats system once per index at the end of the vacuum. + */ + extvac_accumulate_idx_report(&vacrel->extVacIdxReports[idx], &extVacReport); + vacrel->extVacIdxTouched[idx] = true; + /* Revert to the previous phase information for error traceback */ restore_vacuum_error_info(vacrel, &saved_err_info); pfree(vacrel->indname); diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 21fa841f5a..b4fd8f90c6 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1575,7 +1575,11 @@ CREATE VIEW pg_stat_vacuum_tables AS S.missed_dead_tuples AS missed_dead_tuples, S.vm_new_frozen_pages AS vm_new_frozen_pages, S.vm_new_visible_pages AS vm_new_visible_pages, - S.vm_new_visible_frozen_pages AS vm_new_visible_frozen_pages + S.vm_new_visible_frozen_pages AS vm_new_visible_frozen_pages, + S.total_blks_read AS total_blks_read, + S.total_blks_hit AS total_blks_hit, + S.total_blks_dirtied AS total_blks_dirtied, + S.total_blks_written AS total_blks_written FROM pg_class C JOIN pg_namespace N ON N.oid = C.relnamespace, @@ -1591,7 +1595,12 @@ CREATE VIEW pg_stat_vacuum_indexes AS I.relname AS indexrelname, S.pages_deleted AS pages_deleted, - S.tuples_deleted AS tuples_deleted + S.tuples_deleted AS tuples_deleted, + + S.total_blks_read AS total_blks_read, + S.total_blks_hit AS total_blks_hit, + S.total_blks_dirtied AS total_blks_dirtied, + S.total_blks_written AS total_blks_written FROM pg_class C JOIN pg_index X ON C.oid = X.indrelid JOIN @@ -1599,3 +1608,18 @@ CREATE VIEW pg_stat_vacuum_indexes AS LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace), LATERAL pg_stat_get_vacuum_indexes(I.oid) S WHERE C.relkind IN ('r', 't', 'm'); + +CREATE VIEW pg_stat_vacuum_database AS + SELECT + D.oid AS dboid, + D.datname AS dbname, + + S.errors AS errors, + + S.db_blks_read AS db_blks_read, + S.db_blks_hit AS db_blks_hit, + S.total_blks_dirtied AS total_blks_dirtied, + S.total_blks_written AS total_blks_written + FROM + pg_database D, + LATERAL pg_stat_get_vacuum_database(D.oid) S; diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index f0819d15ab..49c3b40c35 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -1830,6 +1830,7 @@ dropdb(const char *dbname, bool missing_ok, bool force) * Tell the cumulative stats system to forget it immediately, too. */ pgstat_drop_database(db_id); + pgstat_drop_vacuum_database(db_id); /* * Except for the deletion of the catalog row, subsequent actions are not diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index a4abb29cf6..aacecee58a 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -118,6 +118,9 @@ pg_atomic_uint32 *VacuumSharedCostBalance = NULL; pg_atomic_uint32 *VacuumActiveNWorkers = NULL; int VacuumCostBalanceLocal = 0; +/* Cumulative storage to report total vacuum delay time. */ +double VacuumDelayTime = 0; /* msec. */ + /* non-export function prototypes */ static List *expand_vacuum_rel(VacuumRelation *vrel, MemoryContext vac_context, int options); @@ -2566,6 +2569,7 @@ vacuum_delay_point(bool is_analyze) exit(1); VacuumCostBalance = 0; + VacuumDelayTime += msec; /* * Balance and update limit values for autovacuum workers. We must do diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c index 7725c4ecc1..6c229564e0 100644 --- a/src/backend/commands/vacuumparallel.c +++ b/src/backend/commands/vacuumparallel.c @@ -1303,6 +1303,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc) VacuumUpdateCosts(); VacuumCostBalance = 0; + VacuumDelayTime = 0; VacuumCostBalanceLocal = 0; VacuumSharedCostBalance = &(shared->cost_balance); VacuumActiveNWorkers = &(shared->active_nworkers); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 4e950672e2..3741e4e54a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -500,6 +500,21 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .reset_all_cb = pgstat_wal_reset_all_cb, .snapshot_cb = pgstat_wal_snapshot_cb, }, + [PGSTAT_KIND_VACUUM_DB] = { + .name = "vacuum statistics", + + .fixed_amount = false, + .write_to_file = true, + /* so pg_stat_database entries can be seen in all databases */ + .accessed_across_databases = true, + + .shared_size = sizeof(PgStatShared_VacuumDB), + .shared_data_off = offsetof(PgStatShared_VacuumDB, stats), + .shared_data_len = sizeof(((PgStatShared_VacuumDB *) 0)->stats), + .pending_size = sizeof(PgStat_VacuumDBCounts), + + .flush_pending_cb = pgstat_vacuum_db_flush_cb, + }, [PGSTAT_KIND_VACUUM_RELATION] = { .name = "vacuum statistics", diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 7f3bc01659..cdff619200 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -46,6 +46,15 @@ pgstat_drop_database(Oid databaseid) pgstat_drop_transactional(PGSTAT_KIND_DATABASE, databaseid, InvalidOid); } +/* + * Remove entry for the database being dropped. + */ +void +pgstat_drop_vacuum_database(Oid databaseid) +{ + pgstat_drop_transactional(PGSTAT_KIND_VACUUM_DB, databaseid, InvalidOid); +} + /* * Called from autovacuum.c to report startup of an autovacuum process. * We are called before InitPostgres is done, so can't rely on MyDatabaseId; diff --git a/src/backend/utils/activity/pgstat_vacuum.c b/src/backend/utils/activity/pgstat_vacuum.c index 4f6cbfd540..21678bf646 100644 --- a/src/backend/utils/activity/pgstat_vacuum.c +++ b/src/backend/utils/activity/pgstat_vacuum.c @@ -7,7 +7,7 @@ * kept separate from pgstat_relation.c and pgstat_database.c to reduce the * memory footprint of the regular relation and database statistics: vacuum * metrics require significantly more space per relation, so they live in their - * own PGSTAT_KIND_VACUUM_RELATION stats kind. + * own PGSTAT_KIND_VACUUM_RELATION and PGSTAT_KIND_VACUUM_DB stats kinds. * * Copyright (c) 2001-2026, PostgreSQL Global Development Group * @@ -22,15 +22,32 @@ #include "utils/memutils.h" #include "utils/pgstat_internal.h" +/* ---------- + * GUC parameters + * ---------- + */ +bool pgstat_track_vacuum_statistics_for_relations = false; + +#define ACCUMULATE_FIELD(field) (dst->field += src->field) #define ACCUMULATE_SUBFIELD(substruct, field) (dst->substruct.field += src->substruct.field) /* - * Accumulate the per-table extended vacuum counters collected so far. - * - * Only the counters derived directly from the vacuum's own bookkeeping are - * summed here. The buffer, WAL and timing counters (and the per-index - * counters) are accumulated by additional code added together with the - * helpers that gather them. + * Accumulate the counters that are common to heap relations, indexes and + * databases. + */ +static void +pgstat_accumulate_common(PgStat_CommonCounts *dst, const PgStat_CommonCounts *src) +{ + ACCUMULATE_FIELD(total_blks_read); + ACCUMULATE_FIELD(total_blks_hit); + ACCUMULATE_FIELD(total_blks_dirtied); + ACCUMULATE_FIELD(total_blks_written); + + ACCUMULATE_FIELD(tuples_deleted); +} + +/* + * Accumulate per-relation (heap or index) extended vacuum counters. */ static void pgstat_accumulate_extvac_stats_relations(PgStat_VacuumRelationCounts *dst, @@ -46,19 +63,19 @@ pgstat_accumulate_extvac_stats_relations(PgStat_VacuumRelationCounts *dst, src->type != PGSTAT_EXTVAC_DB && src->type == dst->type); - ACCUMULATE_SUBFIELD(common, tuples_deleted); + pgstat_accumulate_common(&dst->common, &src->common); if (dst->type == PGSTAT_EXTVAC_TABLE) { ACCUMULATE_SUBFIELD(table, pages_scanned); ACCUMULATE_SUBFIELD(table, pages_removed); - ACCUMULATE_SUBFIELD(table, tuples_frozen); - ACCUMULATE_SUBFIELD(table, recently_dead_tuples); - ACCUMULATE_SUBFIELD(table, missed_dead_pages); - ACCUMULATE_SUBFIELD(table, missed_dead_tuples); - ACCUMULATE_SUBFIELD(table, vm_new_frozen_pages); + ACCUMULATE_SUBFIELD(table, vm_new_frozen_pages); ACCUMULATE_SUBFIELD(table, vm_new_visible_pages); ACCUMULATE_SUBFIELD(table, vm_new_visible_frozen_pages); + ACCUMULATE_SUBFIELD(table, missed_dead_pages); + ACCUMULATE_SUBFIELD(table, tuples_frozen); + ACCUMULATE_SUBFIELD(table, recently_dead_tuples); + ACCUMULATE_SUBFIELD(table, missed_dead_tuples); } else if (dst->type == PGSTAT_EXTVAC_INDEX) { @@ -67,8 +84,22 @@ pgstat_accumulate_extvac_stats_relations(PgStat_VacuumRelationCounts *dst, } /* - * Report that the relation was just vacuumed, accumulating its extended - * statistics into the per-relation entry. + * Accumulate per-database extended vacuum counters. + */ +static void +pgstat_accumulate_extvac_stats_db(PgStat_VacuumDBCounts *dst, + PgStat_VacuumDBCounts *src) +{ + if (!pgstat_track_vacuum_statistics) + return; + + pgstat_accumulate_common(&dst->common, &src->common); + dst->errors += src->errors; +} + +/* + * Report that the relation was just vacuumed, accumulating both its own + * extended statistics and the database-wide aggregate. */ void pgstat_report_vacuum_extstats(Oid tableoid, bool shared, @@ -76,16 +107,25 @@ pgstat_report_vacuum_extstats(Oid tableoid, bool shared, { PgStat_EntryRef *entry_ref; PgStatShared_VacuumRelation *shtabentry; + PgStatShared_VacuumDB *shdbentry; Oid dboid = (shared ? InvalidOid : MyDatabaseId); if (!pgstat_track_vacuum_statistics) return; + /* Per-relation extended vacuum statistics */ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_VACUUM_RELATION, dboid, tableoid, false); shtabentry = (PgStatShared_VacuumRelation *) entry_ref->shared_stats; pgstat_accumulate_extvac_stats_relations(&shtabentry->stats, params); pgstat_unlock_entry(entry_ref); + + /* Database-wide aggregate of the same work */ + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_VACUUM_DB, + dboid, InvalidOid, false); + shdbentry = (PgStatShared_VacuumDB *) entry_ref->shared_stats; + pgstat_accumulate_common(&shdbentry->stats.common, ¶ms->common); + pgstat_unlock_entry(entry_ref); } /* @@ -119,6 +159,31 @@ pgstat_vacuum_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) return true; } +/* + * Flush out pending per-database extended vacuum stats for the entry. + */ +bool +pgstat_vacuum_db_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +{ + PgStatShared_VacuumDB *sharedent; + PgStat_VacuumDBCounts *pendingent; + + pendingent = (PgStat_VacuumDBCounts *) entry_ref->pending; + sharedent = (PgStatShared_VacuumDB *) entry_ref->shared_stats; + + if (pg_memory_is_all_zeros(pendingent, sizeof(PgStat_VacuumDBCounts))) + return true; + + if (!pgstat_lock_entry(entry_ref, nowait)) + return false; + + pgstat_accumulate_extvac_stats_db(&sharedent->stats, pendingent); + + pgstat_unlock_entry(entry_ref); + + return true; +} + /* * Support function for the SQL-callable pgstat* functions. Returns the vacuum * collected statistics for one relation or NULL. @@ -129,3 +194,14 @@ pgstat_fetch_stat_vacuum_tabentry(Oid relid, Oid dbid) return (PgStat_VacuumRelationCounts *) pgstat_fetch_entry(PGSTAT_KIND_VACUUM_RELATION, dbid, relid, NULL); } + +/* + * Support function for the SQL-callable pgstat* functions. Returns the vacuum + * collected statistics for one database or NULL. + */ +PgStat_VacuumDBCounts * +pgstat_fetch_stat_vacuum_dbentry(Oid dbid) +{ + return (PgStat_VacuumDBCounts *) + pgstat_fetch_entry(PGSTAT_KIND_VACUUM_DB, dbid, InvalidOid, NULL); +} diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 3acf0a7391..fc32b20850 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -2374,7 +2374,7 @@ pg_stat_have_stats(PG_FUNCTION_ARGS) Datum pg_stat_get_vacuum_tables(PG_FUNCTION_ARGS) { -#define PG_STAT_GET_VACUUM_TABLES_STATS_COLS 11 +#define PG_STAT_GET_VACUUM_TABLES_STATS_COLS 15 Oid relid = PG_GETARG_OID(0); PgStat_VacuumRelationCounts *extvacuum; @@ -2410,6 +2410,10 @@ pg_stat_get_vacuum_tables(PG_FUNCTION_ARGS) values[i++] = Int64GetDatum(extvacuum->table.vm_new_frozen_pages); values[i++] = Int64GetDatum(extvacuum->table.vm_new_visible_pages); values[i++] = Int64GetDatum(extvacuum->table.vm_new_visible_frozen_pages); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_read); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_hit); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_dirtied); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_written); Assert(i == PG_STAT_GET_VACUUM_TABLES_STATS_COLS); @@ -2423,7 +2427,7 @@ pg_stat_get_vacuum_tables(PG_FUNCTION_ARGS) Datum pg_stat_get_vacuum_indexes(PG_FUNCTION_ARGS) { -#define PG_STAT_GET_VACUUM_INDEX_STATS_COLS 3 +#define PG_STAT_GET_VACUUM_INDEX_STATS_COLS 7 Oid relid = PG_GETARG_OID(0); PgStat_VacuumRelationCounts *extvacuum; @@ -2451,8 +2455,53 @@ pg_stat_get_vacuum_indexes(PG_FUNCTION_ARGS) values[i++] = Int64GetDatum(extvacuum->index.pages_deleted); values[i++] = Int64GetDatum(extvacuum->common.tuples_deleted); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_read); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_hit); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_dirtied); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_written); + Assert(i == PG_STAT_GET_VACUUM_INDEX_STATS_COLS); /* Returns the record as Datum */ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); } + +/* + * Get the extended vacuum statistics for a database. + */ +Datum +pg_stat_get_vacuum_database(PG_FUNCTION_ARGS) +{ +#define PG_STAT_GET_VACUUM_DATABASE_STATS_COLS 6 + + Oid dbid = PG_GETARG_OID(0); + PgStat_VacuumDBCounts *extvacuum; + TupleDesc tupdesc; + Datum values[PG_STAT_GET_VACUUM_DATABASE_STATS_COLS] = {0}; + bool nulls[PG_STAT_GET_VACUUM_DATABASE_STATS_COLS] = {0}; + int i = 0; + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + extvacuum = OidIsValid(dbid) ? pgstat_fetch_stat_vacuum_dbentry(dbid) : NULL; + if (!extvacuum) + { + InitMaterializedSRF(fcinfo, 0); + PG_RETURN_VOID(); + } + + values[i++] = ObjectIdGetDatum(dbid); + + values[i++] = Int32GetDatum(extvacuum->errors); + + values[i++] = Int64GetDatum(extvacuum->common.total_blks_read); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_hit); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_dirtied); + values[i++] = Int64GetDatum(extvacuum->common.total_blks_written); + + Assert(i == PG_STAT_GET_VACUUM_DATABASE_STATS_COLS); + + /* Returns the record as Datum */ + PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); +} diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7d87b03239..edf3cc8b62 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12643,9 +12643,9 @@ proname => 'pg_stat_get_vacuum_tables', prorows => 1000, provolatile => 's', prorettype => 'record', proisstrict => 'f', proretset => 't', proargtypes => 'oid', - proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', - proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{reloid,relid,pages_scanned,pages_removed,tuples_deleted,tuples_frozen,recently_dead_tuples,missed_dead_pages,missed_dead_tuples,vm_new_frozen_pages,vm_new_visible_pages,vm_new_visible_frozen_pages}', + proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{reloid,relid,pages_scanned,pages_removed,tuples_deleted,tuples_frozen,recently_dead_tuples,missed_dead_pages,missed_dead_tuples,vm_new_frozen_pages,vm_new_visible_pages,vm_new_visible_frozen_pages,total_blks_read,total_blks_hit,total_blks_dirtied,total_blks_written}', prosrc => 'pg_stat_get_vacuum_tables' } # oid8 related functions @@ -12718,8 +12718,17 @@ proname => 'pg_stat_get_vacuum_indexes', prorows => 1000, provolatile => 's', prorettype => 'record', proisstrict => 'f', proretset => 't', proargtypes => 'oid', - proallargtypes => '{oid,oid,int8,int8}', - proargmodes => '{i,o,o,o}', - proargnames => '{reloid,relid,pages_deleted,tuples_deleted}', + proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{reloid,relid,pages_deleted,tuples_deleted,total_blks_read,total_blks_hit,total_blks_dirtied,total_blks_written}', prosrc => 'pg_stat_get_vacuum_indexes' }, +{ oid => '8005', + descr => 'pg_stat_get_vacuum_database returns vacuum stats values for database', + proname => 'pg_stat_get_vacuum_database', prorows => 1000, provolatile => 's', prorettype => 'record', proisstrict => 'f', + proretset => 't', + proargtypes => 'oid', + proallargtypes => '{oid,oid,int4,int8,int8,int8,int8}', + proargmodes => '{i,o,o,o,o,o,o}', + proargnames => '{dbid,dboid,errors,db_blks_read,db_blks_hit,total_blks_dirtied,total_blks_written}', + prosrc => 'pg_stat_get_vacuum_database' }, ] diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h index 956d9cea36..f0d08e1a64 100644 --- a/src/include/commands/vacuum.h +++ b/src/include/commands/vacuum.h @@ -21,9 +21,11 @@ #include "catalog/pg_class.h" #include "catalog/pg_statistic.h" #include "catalog/pg_type.h" +#include "executor/instrument.h" #include "parser/parse_node.h" #include "storage/buf.h" #include "utils/relcache.h" +#include "pgstat.h" /* * Flags for amparallelvacuumoptions to control the participation of bulkdelete @@ -321,6 +323,26 @@ typedef struct PVWorkerUsage PVWorkerStats cleanup; } PVWorkerUsage; +/* + * Counters and usage data for extended stats tracking. + */ +typedef struct LVExtStatCounters +{ + TimestampTz starttime; + WalUsage walusage; + BufferUsage bufusage; + double VacuumDelayTime; + PgStat_Counter blocks_fetched; + PgStat_Counter blocks_hit; +} LVExtStatCounters; + +typedef struct LVExtStatCountersIdx +{ + LVExtStatCounters common; + int64 pages_deleted; + int64 tuples_removed; +} LVExtStatCountersIdx; + /* GUC parameters */ extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */ extern PGDLLIMPORT int vacuum_freeze_min_age; @@ -353,6 +375,7 @@ extern PGDLLIMPORT double vacuum_max_eager_freeze_failure_rate; extern PGDLLIMPORT pg_atomic_uint32 *VacuumSharedCostBalance; extern PGDLLIMPORT pg_atomic_uint32 *VacuumActiveNWorkers; extern PGDLLIMPORT int VacuumCostBalanceLocal; +extern PGDLLIMPORT double VacuumDelayTime; extern PGDLLIMPORT bool VacuumFailsafeActive; extern PGDLLIMPORT double vacuum_cost_delay; @@ -439,4 +462,10 @@ extern double anl_random_fract(void); extern double anl_init_selection_state(int n); extern double anl_get_next_S(double t, int n, double *stateptr); +extern void extvac_stats_start_idx(Relation rel, IndexBulkDeleteResult *stats, + LVExtStatCountersIdx * counters); +extern void extvac_stats_end_idx(Relation rel, IndexBulkDeleteResult *stats, + LVExtStatCountersIdx * counters, PgStat_VacuumRelationCounts * report); +extern void extvac_accumulate_idx_report(PgStat_VacuumRelationCounts * dst, + const PgStat_VacuumRelationCounts * src); #endif /* VACUUM_H */ diff --git a/src/include/pgstat.h b/src/include/pgstat.h index f2bdef1463..6a9c9800b7 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -175,6 +175,12 @@ typedef struct PgStat_TableCounts typedef struct PgStat_CommonCounts { + /* blocks */ + int64 total_blks_read; + int64 total_blks_hit; + int64 total_blks_dirtied; + int64 total_blks_written; + /* tuples */ int64 tuples_deleted; } PgStat_CommonCounts; @@ -244,6 +250,13 @@ typedef struct PgStat_VacuumRelationStatus PgStat_VacuumRelationCounts counts; /* event counts to be sent */ } PgStat_VacuumRelationStatus; +typedef struct PgStat_VacuumDBCounts +{ + Oid dbjid; + PgStat_CommonCounts common; + int32 errors; +} PgStat_VacuumDBCounts; + /* ---------- * PgStat_TableStatus Per-table status within a backend * @@ -924,11 +937,13 @@ extern int pgstat_get_transactional_drops(bool isCommit, struct xl_xact_stats_it extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo); +extern void pgstat_drop_vacuum_database(Oid databaseid); extern void pgstat_vacuum_relation_delete_pending_cb(Oid relid); extern void pgstat_report_vacuum_extstats(Oid tableoid, bool shared, PgStat_VacuumRelationCounts * params); extern PgStat_VacuumRelationCounts * pgstat_fetch_stat_vacuum_tabentry(Oid relid, Oid dbid); +extern PgStat_VacuumDBCounts * pgstat_fetch_stat_vacuum_dbentry(Oid dbid); /* * Functions in pgstat_wal.c @@ -947,6 +962,7 @@ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; extern PGDLLIMPORT int pgstat_fetch_consistency; extern PGDLLIMPORT bool pgstat_track_vacuum_statistics; +extern PGDLLIMPORT bool pgstat_track_vacuum_statistics_for_relations; /* * Variables in pgstat_bgwriter.c diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 46a127af2b..a234c797e0 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -507,6 +507,12 @@ typedef struct PgStatShared_Relation PgStat_StatTabEntry stats; } PgStatShared_Relation; +typedef struct PgStatShared_VacuumDB +{ + PgStatShared_Common header; + PgStat_VacuumDBCounts stats; +} PgStatShared_VacuumDB; + typedef struct PgStatShared_VacuumRelation { PgStatShared_Common header; @@ -695,6 +701,7 @@ extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *may_free); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +bool pgstat_vacuum_db_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); extern bool pgstat_vacuum_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h index f92149066c..52d9367fbe 100644 --- a/src/include/utils/pgstat_kind.h +++ b/src/include/utils/pgstat_kind.h @@ -40,9 +40,10 @@ #define PGSTAT_KIND_SLRU 12 #define PGSTAT_KIND_WAL 13 #define PGSTAT_KIND_VACUUM_RELATION 14 +#define PGSTAT_KIND_VACUUM_DB 15 #define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE -#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VACUUM_RELATION +#define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_VACUUM_DB #define PGSTAT_KIND_BUILTIN_SIZE (PGSTAT_KIND_BUILTIN_MAX + 1) /* Custom stats kinds */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index c30ff6c72f..7b67fe72a6 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2421,18 +2421,31 @@ pg_stat_user_tables| SELECT relid, frozen_page_marks_cleared FROM pg_stat_all_tables WHERE ((schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (schemaname !~ '^pg_toast'::text)); +pg_stat_vacuum_database| SELECT d.oid AS dboid, + d.datname AS dbname, + s.errors, + s.db_blks_read, + s.db_blks_hit, + s.total_blks_dirtied, + s.total_blks_written + FROM pg_database d, + LATERAL pg_stat_get_vacuum_database(d.oid) s(dboid, errors, db_blks_read, db_blks_hit, total_blks_dirtied, total_blks_written); pg_stat_vacuum_indexes| SELECT c.oid AS relid, i.oid AS indexrelid, n.nspname AS schemaname, c.relname, i.relname AS indexrelname, s.pages_deleted, - s.tuples_deleted + s.tuples_deleted, + s.total_blks_read, + s.total_blks_hit, + s.total_blks_dirtied, + s.total_blks_written FROM (((pg_class c JOIN pg_index x ON ((c.oid = x.indrelid))) JOIN pg_class i ON ((i.oid = x.indexrelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))), - LATERAL pg_stat_get_vacuum_indexes(i.oid) s(relid, pages_deleted, tuples_deleted) + LATERAL pg_stat_get_vacuum_indexes(i.oid) s(relid, pages_deleted, tuples_deleted, total_blks_read, total_blks_hit, total_blks_dirtied, total_blks_written) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char", 'm'::"char"])); pg_stat_vacuum_tables| SELECT n.nspname AS schemaname, c.relname, @@ -2446,10 +2459,14 @@ pg_stat_vacuum_tables| SELECT n.nspname AS schemaname, s.missed_dead_tuples, s.vm_new_frozen_pages, s.vm_new_visible_pages, - s.vm_new_visible_frozen_pages + s.vm_new_visible_frozen_pages, + s.total_blks_read, + s.total_blks_hit, + s.total_blks_dirtied, + s.total_blks_written FROM (pg_class c JOIN pg_namespace n ON ((n.oid = c.relnamespace))), - LATERAL pg_stat_get_vacuum_tables(c.oid) s(relid, pages_scanned, pages_removed, tuples_deleted, tuples_frozen, recently_dead_tuples, missed_dead_pages, missed_dead_tuples, vm_new_frozen_pages, vm_new_visible_pages, vm_new_visible_frozen_pages) + LATERAL pg_stat_get_vacuum_tables(c.oid) s(relid, pages_scanned, pages_removed, tuples_deleted, tuples_frozen, recently_dead_tuples, missed_dead_pages, missed_dead_tuples, vm_new_frozen_pages, vm_new_visible_pages, vm_new_visible_frozen_pages, total_blks_read, total_blks_hit, total_blks_dirtied, total_blks_written) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char", 'm'::"char"])); pg_stat_wal| SELECT wal_records, wal_fpi, diff --git a/src/test/regress/expected/vacuum_stats.out b/src/test/regress/expected/vacuum_stats.out index 0b3354cfff..6c46239eb6 100644 --- a/src/test/regress/expected/vacuum_stats.out +++ b/src/test/regress/expected/vacuum_stats.out @@ -99,16 +99,37 @@ SELECT vm_new_visible_pages > 0 AS vm_new_visible_pages, (1 row) DROP TABLE vacstat_frz; +-- total buffer access counters. The vacuum always touches the table's pages +-- through the buffer cache (total_blks_hit > 0) and dirties some of them while +-- removing dead tuples (total_blks_dirtied > 0). total_blks_read and +-- total_blks_written depend on the buffer-cache and checkpoint state at run +-- time, so they are only checked for being non-negative. +SELECT total_blks_read >= 0 AS total_blks_read, + total_blks_hit > 0 AS total_blks_hit, + total_blks_dirtied > 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written + FROM pg_stat_vacuum_tables WHERE relname = 'vacstat_t'; + total_blks_read | total_blks_hit | total_blks_dirtied | total_blks_written +-----------------+----------------+--------------------+-------------------- + t | t | t | t +(1 row) + -- per-index view: the primary key index is processed by the same VACUUM. -- No btree leaf empties out (interleaved deletions), so pages_deleted = 0, --- while every index entry for a removed heap tuple is deleted. +-- while every index entry for a removed heap tuple is deleted. The index is +-- read through the buffer cache (total_blks_hit > 0); the read/written/dirtied +-- counters depend on run-time cache state. SELECT indexrelname, pages_deleted = 0 AS pages_deleted, - tuples_deleted = 500 AS tuples_deleted + tuples_deleted = 500 AS tuples_deleted, + total_blks_read >= 0 AS total_blks_read, + total_blks_hit > 0 AS total_blks_hit, + total_blks_dirtied >= 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written FROM pg_stat_vacuum_indexes WHERE relname = 'vacstat_t' ORDER BY indexrelname; - indexrelname | pages_deleted | tuples_deleted -----------------+---------------+---------------- - vacstat_t_pkey | t | t + indexrelname | pages_deleted | tuples_deleted | total_blks_read | total_blks_hit | total_blks_dirtied | total_blks_written +----------------+---------------+----------------+-----------------+----------------+--------------------+-------------------- + vacstat_t_pkey | t | t | t | t | t | t (1 row) -- index page-deletion path: deleting a contiguous key range empties whole @@ -130,3 +151,17 @@ SELECT indexrelname, (1 row) DROP TABLE vacstat_idxdel; +-- per-database aggregate view: no vacuum errors occurred in this database, and +-- the vacuums in this database touched pages through the buffer cache +-- (db_blks_hit > 0). +SELECT errors = 0 AS errors, + db_blks_read >= 0 AS db_blks_read, + db_blks_hit > 0 AS db_blks_hit, + total_blks_dirtied >= 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written + FROM pg_stat_vacuum_database WHERE dbname = current_database(); + errors | db_blks_read | db_blks_hit | total_blks_dirtied | total_blks_written +--------+--------------+-------------+--------------------+-------------------- + t | t | t | t | t +(1 row) + diff --git a/src/test/regress/sql/vacuum_stats.sql b/src/test/regress/sql/vacuum_stats.sql index db80ecf6a1..91079759ea 100644 --- a/src/test/regress/sql/vacuum_stats.sql +++ b/src/test/regress/sql/vacuum_stats.sql @@ -77,12 +77,29 @@ SELECT vm_new_visible_pages > 0 AS vm_new_visible_pages, FROM pg_stat_vacuum_tables WHERE relname = 'vacstat_frz'; DROP TABLE vacstat_frz; +-- total buffer access counters. The vacuum always touches the table's pages +-- through the buffer cache (total_blks_hit > 0) and dirties some of them while +-- removing dead tuples (total_blks_dirtied > 0). total_blks_read and +-- total_blks_written depend on the buffer-cache and checkpoint state at run +-- time, so they are only checked for being non-negative. +SELECT total_blks_read >= 0 AS total_blks_read, + total_blks_hit > 0 AS total_blks_hit, + total_blks_dirtied > 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written + FROM pg_stat_vacuum_tables WHERE relname = 'vacstat_t'; + -- per-index view: the primary key index is processed by the same VACUUM. -- No btree leaf empties out (interleaved deletions), so pages_deleted = 0, --- while every index entry for a removed heap tuple is deleted. +-- while every index entry for a removed heap tuple is deleted. The index is +-- read through the buffer cache (total_blks_hit > 0); the read/written/dirtied +-- counters depend on run-time cache state. SELECT indexrelname, pages_deleted = 0 AS pages_deleted, - tuples_deleted = 500 AS tuples_deleted + tuples_deleted = 500 AS tuples_deleted, + total_blks_read >= 0 AS total_blks_read, + total_blks_hit > 0 AS total_blks_hit, + total_blks_dirtied >= 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written FROM pg_stat_vacuum_indexes WHERE relname = 'vacstat_t' ORDER BY indexrelname; -- index page-deletion path: deleting a contiguous key range empties whole @@ -99,3 +116,13 @@ SELECT indexrelname, tuples_deleted = 9000 AS tuples_deleted FROM pg_stat_vacuum_indexes WHERE relname = 'vacstat_idxdel' ORDER BY indexrelname; DROP TABLE vacstat_idxdel; + +-- per-database aggregate view: no vacuum errors occurred in this database, and +-- the vacuums in this database touched pages through the buffer cache +-- (db_blks_hit > 0). +SELECT errors = 0 AS errors, + db_blks_read >= 0 AS db_blks_read, + db_blks_hit > 0 AS db_blks_hit, + total_blks_dirtied >= 0 AS total_blks_dirtied, + total_blks_written >= 0 AS total_blks_written + FROM pg_stat_vacuum_database WHERE dbname = current_database(); -- 2.39.5 (Apple Git-154)