Roadmap for FE/BE protocol redesign

This is an attempt to lay out a road map for updating the frontend/backend
protocol in 7.4. I don't at this point want to get into details on any
one of the TODO items, just get consensus that this is the set of tasks
to be tackled. Are there any areas that I've missed (that require
protocol changes)?

* Extend ERROR and NOTICE messages to carry multiple fields, including
(as appropriate) a spec-compliant error code, a textual position in the
original query string, information about the source-code location where
the error was reported, etc.

* Consider extending NOTIFY messages to allow a parameter to be carried.

* Fix COPY protocol to allow graceful error recovery (more graceful than
aborting the connection, at least) and to support COPY BINARY to/from
frontend.

* Redesign fastpath function call protocol to eliminate the problems cited
in the source code comments (src/backend/tcop/fastpath.c), and to
eliminate the security hole of accepting unchecked internal representation
from frontend. Also consider a fastpath for execution of PREPAREd queries.

* Re-institute type-specific send/receive conversion routines to allow
some modest amount of architecture independence for binary data. This'd
provide a place to check for bogus internal representation during fastpath
input and COPY BINARY IN, too, thus alleviating security concerns.

* Get rid of hardwired field sizes in StartupPacket --- use
variable-length null-terminated strings. Fixes problem with usernames
being limited to 32 characters, gets around unreasonable limitation on
PGOPTIONS length. Also can remove unused fields.

* Backend should pass its version number, database encoding, default
client encoding, and possibly other data (any ideas?) to frontend during
startup, to avoid need for explicit queries to get this info. We could
also consider eliminating SET commands sent by libpq in favor of adding
variable settings to startup packet's PGOPTIONS field. Ideally we could
get back to the point where a standard connection startup takes only one
packet in each direction.

* Backend's ReadyForQuery message (Z message) should carry an indication
of current transaction status (idle/in transaction/in aborted transaction)
so that frontend need not guess at state. Perhaps also indicate
autocommit status. (Is there anything else that frontends would Really
Like To Know?)

* XML support? If we do anything, I'd want some extensible solution to
allowing multiple query-result output formats from the backend, not an
XML-specific hack. For one thing, that would allow the actual appearance
of any XML support to happen later.

One of the $64 questions that has to be answered is how much work we're
willing to expend on backwards compatibility. The path of least
resistance would be to handle it the same way we've done protocol
revisions in the past: the backend will be able to handle both old and new
protocols (so it can talk to old clients) but libpq would be revised to
speak only the new protocol (so new/recompiled clients couldn't talk to
old backends). We've gotten away with this approach in the past, but the
last time was release 6.4. I fully expect to hear more complaints now.

One way to tamp down expectations of client backwards compatibility
would be to call the release 8.0 instead of 7.4 ;-)

Comments?

regards, tom lane