Re: Connection terminated by the server causes deadlock in jdbc client side connection

I think there maybe multiple bugs going on here and possibly not fixed...
First I'd like to say that even though we are dealing with TCP one must
treat it almost like UDP. TCP packets can be consumed by firewalls and
proxy servers with no response given to the sender. So with that in mind
to create a robust fault tolerant driver, timeout on all TCP
communications is a must.

1) So in the first case where an IOException is thrown, the original
code that is closing the connection via "close" cannot have a timeout
feature enabled and that will still exist in the code with the potential
to fail...

2) When an IOException is thrown as the origin of an exception this
should not be lost and should always be attached to the final exception.
I don't see this cascaded in the updated driver using the "abort" code...

Regards
Donald

On 08/10/2015 18:45, Leonardo Frittelli wrote:
> Steffen,
>
> I can't really comment on what is the behavior of the PG server, but
> at least in the scenario that I am having (replication taking priority
> over a local query in hot_standby) either the client does not appear
> to receive an RST or Java 1.7 is still trying to write to that socket
> even if the reading end is gone, or even worse: the server is not
> actually closing it but simply no longer reading (Perhaps this is
> indeed hinting at a bug in the server side too?)
>
> The stack trace does not give much detail either. This is the new
> trace I get after patching the driver to use abort() instead of close():
>
> ...
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.postgresql.util.PSQLException: FATAL: terminating
> connection due to conflict with recovery
> Detail: User query might have needed to see row versions that must
> be removed.
> Hint: In a moment you should be able to reconnect to the database
> and repeat your command.
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:562)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:420)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:305)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
> at
> org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:56)
> ... 42 more
>
> As you can see, the IO Exception which caused the PSQL Exception is
> not coming up in the stack trace.
>
> Just for your awareness, please note that we are now using a snapshot
> that Dave created for us with the change above mentioned (to use
> abort() instead of close()) and we no longer see the issue now.
>
> Regards,
>
> Leonardo
>
>
>
> On 8 October 2015 at 13:46, Steffen Heil (Mailinglisten)
> <lists(at)steffen-heil(dot)de <mailto:lists(at)steffen-heil(dot)de>> wrote:
>
> Hi
>
>
> > I was referring to the fact that the TCP connection is indeed
> left 'half
> > open'
> > with the jdbc client side still assuming it's 'properly' open
> and thus
> > trying to
> > send a graceful close command over the socket ("X" in v2 or "X4"
> in v3) . It
> > is
> > this action that indefinitely blocks the client thread, which is
> then no
> > longer
> > usable by the application.
>
> But it should not block indefinitely. Why would it?
> If the server closed the connection, it should reply with a RST
> and the client
> should detect that and throw an IOException( "Socket closed" ).
>
>
> > All of this said, I still think it is conceptually incorrect to
> attempt a
> > graceful
> > closure after an IO error in the client.
>
> Agreed. IOExceptions on blocking sockets usually indicate that you
> cannot do
> anything good with them anymore.
>
>
> Regards,
> Steffen
>
>
>