invalid alloc size error possible in shm_mq

Hi,

in shm_mq_receive of shm_mq.c, a huge payload may trigger an unjustified
"invalid memory alloc request size" error due to the way the buffer size
is increased.

I understand that this fine piece of infrastructure may not have been
designed for huge payloads. I still argue it should better not fail
that way for payloads between half a gigabyte and MaxAllocSize. The
attached patch adds an appropriate assertion (documenting the upper
limit) and avoids the error by limiting the allocation size to
MaxAllocSize (to correct for such huge payloads below the limit).

We originally hit the error with Postgres 11, but the provided patch
applies to master and 9.5. And I suspect to everything in between as well.

The only minor caveat I see from this patch could be that `mqh_buffer`
would be of size MaxAllocSize, which is not a power of two. As far as I
can tell, that does not matter, though.

Please consider applying and back-porting the provided patch. Thank you.

Best Regards

--
Markus Wanner
Senior PostgreSQL Developer
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/