Re: BUG #17449: Disk space not released

From: Giorgio Saviane <gsaviane(at)gmail(dot)com>
To: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
Cc: Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com>, PostgreSQL mailing lists <pgsql-bugs(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #17449: Disk space not released
Date: 2022-05-30 11:40:24
Message-ID: CAHs6c0dhj=zTjc9uCC9gTFt=gCtQ+xK3St-Up-WnT=GA+Ke1Lw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

>>>
>>> > Despite any attempt
>>> > of full vacuum the discrepancy remained the same. I suspect that Postgres
>>> > started leaking disk space. I could see many 1Gb files with a timestamp of
>>> > two months back in time in the postgres data folder.
>
>
> If the database suffers a crash (or immediate shutdown) in the middle of something like VACUUM FULL or CLUSTER, it might leave orphaned in-process files such as the ones you describe behind and have no way to know to clean them up. The knowledge about what it was working on just before the crash was lost in the crash.
>
> Files not touched in 2 months and also not referenced in pg_class.relfilenode are almost certainly such orphaned files and could, with extreme nervousness, be cleaned up by hand. Especially if the human-readable log files support a crash having happened at that time.

That was not the case. The server has been running seamlessly since I
rebuilt the master.
>
>>>
>>> > Restarting the server did not have any effect, so I decided to pg_dump the
>>> > database and pg_restore the backup in a new instance. That worked, the new
>>> > database is now ~ 50 Gb and dropping the old one released that 500Gb of disk
>>> > space.
>>> > The database was under streaming replication and I noticed the postgres log
>>> > reporting many of these messages
>>> >
>>> > requested WAL segment 0000000100000000000000E3 has already been removed
>
>
> When did those start? Before you rebuilt the master? Was your replica using, or attempting to use, replication slots?

They show up after I rebuild the master and re-enable the replica. No,
the replica is not using any slot, but I got that it would help in
case of unstable networking between slave and master.

Kind regards

Giorgio

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Alvaro Herrera 2022-05-30 14:11:39 Re: BUG #17485: Records missing from Primary Key index when doing REINDEX INDEX CONCURRENTLY
Previous Message PG Bug reporting form 2022-05-30 11:21:35 BUG #17503: Package postgresql-jdbc-42.3.6-1.rhel8.noarch.rpm is not signed