Re: libpq compression

On 12/22/20 6:56 PM, Robert Haas wrote:
> On Tue, Dec 22, 2020 at 6:24 AM Daniil Zakhlystov
> <usernamedt(at)yandex-team(dot)ru> wrote:
>> When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage:
>>
>> No compression - 1.2 GiB
>>
>> ZSTD
>> zstd:1 - 1.4 GiB
>> zstd:7 - 4.0 GiB
>> zstd:13 - 17.7 GiB
>> zstd:19 - 56.3 GiB
>> zstd:20 - 109.8 GiB - did not succeed
>> zstd:21, zstd:22 > 140 GiB
>> Postgres process crashes (out of memory)
>
> Good grief. So, suppose we add compression and support zstd. Then, can
> unprivileged user capable of connecting to the database can negotiate
> for zstd level 1 and then choose to actually send data compressed at
> zstd level 22, crashing the server if it doesn't have a crapton of
> memory? Honestly, I wouldn't blame somebody for filing a CVE if we
> allowed that sort of thing to happen. I'm not sure what the solution
> is, but we can't leave a way for a malicious client to consume 140GB
> of memory on the server *per connection*. I assumed decompression
> memory was going to measured in kB or MB, not GB. Honestly, even at
> say L7, if you've got max_connections=100 and a user who wants to make
> trouble, you have a really big problem.
>
> Perhaps I'm being too pessimistic here, but man that's a lot of memory.
>

Maybe I'm just confused, but my assumption was this means there's a
memory leak somewhere - that we're not resetting/freeing some piece of
memory, or so. Why would zstd need so much memory? It seems like a
pretty serious disadvantage, so how could it become so popular?

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company