Re: JDBC prepared statement: a 32767 limit of arguments number

From: Vladislav Malyshkin <mal(at)gromco(dot)com>
To: "David G(dot) Johnston" <david(dot)g(dot)johnston(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "pgsql-jdbc(at)lists(dot)postgresql(dot)org" <pgsql-jdbc(at)lists(dot)postgresql(dot)org>
Subject: Re: JDBC prepared statement: a 32767 limit of arguments number
Date: 2022-03-09 20:09:47
Message-ID: b676fc7e-0c3a-1c57-0d73-2f02a7fc8b37@gromco.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-jdbc

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 09/03/2022 14.55, David G. Johnston
wrote:<br>
&gt;PostgreSQL would expect that you would use the COPY API in
this kind of situation.<br>
<br>
Yes, one can use <code><span class="memberNameLink"><a
href="https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyManager.html#copyIn-java.lang.String-org.postgresql.util.ByteStreamWriter-">copyIn</a> 
</span></code>method but data transformation to <code> <a
href="https://docs.oracle.com/javase/8/docs/api/java/io/InputStream.html?is-external=true"
title="class or interface in java.io">InputStream</a> </code>loses
data type information of the programming language. The beauty of
PreparedStatement is that the methods like setLong, setString
preserve programming language data types<br>
<br>
&gt; If PostgreSQL ever does end up releasing a new protocol
version I would say it is at least in the realm of possibility
that this limit would be increased.  But releasing a new protocol
version is hard to get consensus to do given the incompatibilities
doing so creates.  The curse of a mature and widely used
application.<br>
<br>
I understand. <br>
<br>
<br>
On 09/03/2022 14.55, David G. Johnston wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAKFQuwYeZpr7c6TqmSPznTAAygB6yQ=q2yFQfc+pMf=YHKs=5Q(at)mail(dot)gmail(dot)com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:Arial,Helvetica,sans-serif">On Wed, Mar
9, 2022 at 12:38 PM Vladislav Malyshkin &lt;<a
href="mailto:mal(at)gromco(dot)com" moz-do-not-send="true">mal(at)gromco(dot)com</a>&gt;
wrote:</span><br>
</div>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>On 09/03/2022 10.03, Tom Lane wrote:<br>
</div>
<blockquote type="cite"><br>
<pre>I concur with David's opinion that if you think you need more
parameters, you're doing it wrong. One idea to consider is
aggregating similar values into an array parameter.</pre>
</blockquote>
<br>
I disagree:</div>
</div>
</blockquote>
<div><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Ok.  If
PostgreSQL ever does end up releasing a new protocol version
I would say it is at least in the realm of possibility that
this limit would be increased.  But releasing a new protocol
version is hard to get consensus to do given the
incompatibilities doing so creates.  The curse of a mature
and widely used application.  It isn't like this limitation
has gone unnoticed until now; yet still we haven't released
a new protocol version in many years nor presently have
concrete plans to do so.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div> <br>
1. It is extremely convenient to insert multiple values
in a single SQL insert:<br>
<p><b>INSERT INTO table_name (f1,f2,f3) VALUES
(1,"text",2.4), (2,"text2",2.5),...</b></p>
Setting all values as JDBC parameters is the easiest way
to integrate java/scala and SQL. <br>
A single insert of 10000 records is several orders of
magnitudes <b>faster</b> than 10000 separate inserts,
not to mention transaction simplification.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">PostgreSQL
would expect that you would use the COPY API in this kind
of situation.</div>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div> <br>
2. For automatic scala&lt;-&gt;jdbc integration tools
such as <a href="https://github.com/mal19992/sqlps"
target="_blank" moz-do-not-send="true">https://github.com/mal19992/sqlps</a>
a number of  JDBC arguments can be generated by an
automatic transformation, the SQL and JDBC arguments are
autogenerated and can be a very large number. <br>
Suggested by David approach "like using a temp table and
a join instead of an IN operator." is extremely
inconvenient for automatic tools.<br>
<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">This is too
vague to comment upon or be convinced by.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">David J.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>

Attachment Content-Type Size
unknown_filename text/html 6.2 KB

In response to

Responses

Browse pgsql-jdbc by date

  From Date Subject
Next Message David G. Johnston 2022-03-09 20:18:37 Re: JDBC prepared statement: a 32767 limit of arguments number
Previous Message David G. Johnston 2022-03-09 19:55:18 Re: JDBC prepared statement: a 32767 limit of arguments number