Re: pg should ignore u+200b zero width space

On 2020-11-03 15:52, Tom Lane wrote:
> The difficulty with doing anything in this space --- whether it be
> ignoring, throwing an error, or whatever --- is that it makes the
> lexer's behavior encoding-sensitive and potentially locale-sensitive.
> That's problematic for all sorts of reasons. One of the worst is
> that frontend programs such as psql and ecpg also have SQL lexers,
> and there'd be no way to keep their behavior in precise sync with
> the backend's. (They might not even be running in the same encoding,
> never mind locale.) It might even be possible to build security
> holes around that; recall the fun we've had with trying to lock
> down quoting rules in encodings where backslash can be part of a
> multibyte character :-(.

I think we could support whitespace properly if we're using Unicode, and
use ASCII whitespace for all other encodings. The combinations to deal
with are finite. But it is a tricky problem that needs to be approached
with care.