debugging intermittent slow updates under higher load

Hi All,

This is on postgres 9.4.16, same table as the last question I asked,
here's an abbreviated desc:

# \d alerts_alert
              Table "public.alerts_alert"
     Column      |           Type           | Modifiers
-----------------+--------------------------+-----------
 tags            | jsonb                    | not null
 id              | character varying(86)    | not null
...
Indexes:
    "alerts_alert_pkey" PRIMARY KEY, btree (id)

The table has around 1.5M rows which have been updated/inserted around
121M times, the distribution of updates to row in alerts_alert will be
quite uneven, from 1 insert up to 1 insert and 0.5M updates.

Under high load (200-300 inserts/updates per second) we see occasional
(~10 per hour) updates taking excessively long times (2-10s). These
updates are always of the form:

UPDATE "alerts_alert" SET ...bunch of fields... WHERE
"alerts_alert"."id" = '...sha1 hash...';

Here's a sample explain:

https://explain.depesz.com/s/Fjq8

What could be causing this? What could we do to debug? What config
changes could we make to alleviate this?

cheers,

Chris