Re: libpq compression

On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov
<usernamedt(at)yandex-team(dot)ru> wrote:
> When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage:
>
> No compression - 1.2 GiB
>
> ZSTD
> zstd:1 - 1.4 GiB
> zstd:7 - 4.0 GiB
> zstd:13 - 17.7 GiB
> zstd:19 - 56.3 GiB
> zstd:20 - 109.8 GiB - did not succeed
> zstd:21, zstd:22 > 140 GiB
> Postgres process crashes (out of memory)

Good grief. So, suppose we add compression and support zstd. Then, can
unprivileged user capable of connecting to the database can negotiate
for zstd level 1 and then choose to actually send data compressed at
zstd level 22, crashing the server if it doesn't have a crapton of
memory? Honestly, I wouldn't blame somebody for filing a CVE if we
allowed that sort of thing to happen. I'm not sure what the solution
is, but we can't leave a way for a malicious client to consume 140GB
of memory on the server *per connection*. I assumed decompression
memory was going to measured in kB or MB, not GB. Honestly, even at
say L7, if you've got max_connections=100 and a user who wants to make
trouble, you have a really big problem.

Perhaps I'm being too pessimistic here, but man that's a lot of memory.

--
Robert Haas
EDB: http://www.enterprisedb.com