Re: Slow shutdowns sometimes on RDS Postgres

On 09/13/2018 03:04 PM, Chris Williams wrote:
> Hi,
>
> I'm using AWS RDS Postgres (9.6.6) and have run into very slow
> shutdowns (10+ minutes) a few times when making database modifications
> (e.g. reboot, changing instance size, etc.).  Other times, it shuts
> down quickly (1 minute or so).  I have not been able to figure out why
> sometimes it takes a long time to shutdown.

This is probably something you would have to talk to Amazon about. AWS
RDS Postgres is a fork of PostgreSQL and not 100% compatible from an
administrative perspective.

JD

>
> When it happens, I see a bunch of lines in the postgres log like the
> following over and over (almost every second or two) during this 10
> minute shutdown period:
> 2018-09-12 06:37:01
> UTC:XXX.XXX.XXX.XXX(19712):my_user(at)my_db:[16495]:FATAL:
> 2018-09-12 06:37:01
> UTC:localhost(31368):rdsadmin(at)rdsadmin:[16488]:FATAL: the database
> system is shutting down
>
> Once I start seeing these messages, I start manually shutting down all
> of our applications that are connected to the db.  I'm not sure if
> shutting down the apps fixes it or if there's some timeout on the RDS
> side, but it seems like once I start doing this, the database finally
> shuts down.
>
> When it takes this long to shut down, it ends up causing a lot more
> downtime than I would like.  I've tried asking AWS's support why it
> takes so long to shutdown sometimes, but they basically just told me
> that's "how it works" and that I should try to shut down all of my
> connections ahead of time before making database modifications.
>
> We just have a few ruby on rails applications connected to the
> database, and don't really have any long running or heavy queries and
> the db is under very light load, so I don't understand why it takes so
> long to shutdown.  We do have a sizeable number of connections though
> (about 600) and there are two replicas connected to it.  I also tried
> setting idle_in_transaction_session_timeout to 300 seconds to see if
> that would help, but it made no difference.
>
> I was wondering if anyone else had seen this behavior on their RDS
> Postgres instances or had any suggestions on how I could shorten the
> shutdown time?
>
> Thanks,
> Chris
>

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
*** A fault and talent of mine is to tell it exactly how it is. ***
PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****