Could not read block in file

From: Don Seiler <don(at)seiler(dot)us>
To: pgsql-general(at)postgresql(dot)org
Subject: Could not read block in file
Date: 2018-01-18 16:11:46
Message-ID: CAHJZqBDa0x7BfL_xN_Mig-D=brtCxW2Bj-s7rXEKY-5gqSFExg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

This morning I got some errors from our pre-prod environment. This is
running Pg 9.6.6 on CentOS 7.3.

Just grepping out these errors (and obfuscating sensitive data):
2018-01-18 06:29:21 CST [11912]: [1570-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 39 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:29:42 CST [11959]: [517-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 40 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:30:09 CST [11912]: [1577-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 41 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:30:32 CST [11970]: [162-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 42 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:30:40 CST [11912]: [1583-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 43 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:31:04 CST [11970]: [168-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 44 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:31:31 CST [11959]: [595-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 45 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes

All of these errors occurred during attempts to insert into this liferay
table. For example:

2018-01-18 06:29:21 CST [11912]: [1568-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com LOG: execute S_14: INSERT INTO
quartz_FIRED_TRIGGERS (ENTRY_ID, TRIGGER_NAME, TRIGGER_GROUP, IS_VOLATILE,
INSTANCE_NAME, FIRED_TIME, STATE, JOB_NAME, JOB_GROUP, IS_STATEFUL,
REQUESTS_RECOVERY, PRIORITY) VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9,
$10, $11, $12)
2018-01-18 06:29:21 CST [11912]: [1569-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com DETAIL: parameters: $1 = 'FOO', $2 = 'Bar
CSVTrigger', $3 = 'DEFAULT', $4 = 'f', $5 = 'FOO', $6 = '123', $7 = 'BLAH',
$8 = NULL, $9 = NULL, $10 = 'f', $11 = 'f', $12 = '5'
2018-01-18 06:29:21 CST [11912]: [1570-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com ERROR: could not read block 39 in file
"pg_tblspc/16406/PG_9.6_201608131/16416/25690536": read only 0 of 8192 bytes
2018-01-18 06:29:21 CST [11912]: [1571-1]
db=abcprod,user=abcreporting2,app=PostgreSQL JDBC Driver,client=
abcchizapp1.mycompanyname.com STATEMENT: INSERT INTO quartz_FIRED_TRIGGERS
(ENTRY_ID, TRIGGER_NAME, TRIGGER_GROUP, IS_VOLATILE, INSTANCE_NAME,
FIRED_TIME, STATE, JOB_NAME, JOB_GROUP, IS_STATEFUL, REQUESTS_RECOVERY,
PRIORITY) VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)

These are the only occurrences of these errors. I've since been able to
query this table as well as take a pg_dump of the schemas on this
tablespace, with no errors either time. So I can't seem to duplicate this
problem. The app has since successfully performed many, many inserts into
that table since these errors.

I don't see any errors in /var/log/messages that would incidate any
filesystem issues either. Obviously the worry is for possible corruption.
Granted this is "only" pre-prod but there will be concerns from the
business side prior to upgrading our prod DB (currently 9.2.22).

What else can/should I check here to make sure there isn't something wrong
with this database cluster?

--
Don Seiler
www.seiler.us

Browse pgsql-general by date

  From Date Subject
Next Message Steve Atkins 2018-01-18 16:19:10 Re: Connection type
Previous Message Enrico Pirozzi 2018-01-18 16:06:57 Connection type