Re: MPTCP - multiplexing many TCP connections through one socket to get better bandwidth

On Thu, Sep 4, 2025 at 12:56 PM Jakub Wartak
<jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
>
> Hi -hackers,
>
> With the attached patch PostgreSQL could possibly gain built-in MPTCP
> support which would allow multiplexing (aggregating) multiple
> kernel-based TCP streams into one MPTCP socket. This allows bypassing
> any "chokepoints" on the network transparently for libpq, especially
> if having *multiple* TCP streams could achieve higher bandwidth than
> single one. One can think of transparent aggregation of bandwidth over
> multiple WAN links/tunnels and so. In short it works like this:
> libpq_client <--MPTCP--> client_kernel <==multiple TCP
> connections==> server_kernel <--MPTCP--> server_kernel
>
> Without much rework of PostgreSQL, this means accelerating any
> libpq-based use case. Most obvious beneficiaries could be any
> libpq-based heavy network transfers, especially in enterprise
> networks. Those come to my mind:
> - pg_basebackup (over e.g. WAN or multiple interfaces; but also one
> can think of using 2x 10GigE over LAN)
> - streaming replication or logical replication [years ago I've was
> able to use MPTCP with colleagues on production to bypass single TCP
> stream limitation of streaming replication]
> - COPY (both upload and download)
> - postgres_fdw/dblinks?
>
> MPTCP is IETF standard and included from Linux kernels from some time
> (realistically 5.16+?) and it's *enabled* by default in most modern
> distributions. One could use it with mptcpize (LD_PRELOAD wrapper to
> hijack socket()), but it's not elegant and would require altering
> systemd startup scripts (the same story like with NUMA: literally
> nobody hacking those to just include numactl --interleave there or
> with adjusting ulimits).
>
> The patch right now just assumes IPPROTO_MPTCP is there, so it is not
> portable, but not that many OSes support it at all -- I think #ifdef
> would be good enough for now. I dont have access to MacOS to develop
> this more there, nor I think it would add benefit there, but I may be
> wrong. So as such the proposed patch is trivial and Linux-only,
> although there is RFC8684[1][2]. I suspect it is way easier and
> simpler to support it , rather than try to solve the same problem for
> each of the listed use-cases.
>
> Simulation, basic-use and tests:
>
> 1. Strictly for demo purposes here, we need to ARTIFICIALLY limit
> outbound bandwidth for each new flow (TCP connection) to 10 Mbit/s
> using `tc` on the server where PostgreSQL is going to be running later
> on (this simulates some chokepoints, multiple WAN paths):
> DEV=enp0s31f6
> tc qdisc add dev $DEV root handle 1: htb
> tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit
> for i in `seq 1 9`; do
> tc class add dev $DEV parent 1:1 classid 1:$i htb rate 10mbit
> ceil 10mbit
> done
> # see tc-flow(8) for details, classify each flow with port into
> separate class (1:X)
> tc filter add dev $DEV parent 1: protocol ip prio 1 handle 1 flow
> hash keys src,dst,proto,proto-src,proto-dst divisor 8 baseclass 1
>
> 2. From client, verify that single TCP bandwidth is really limited:
> verify using iperf3 -P 1 -R -c <server> # if you really getting
> limited single-stream TCP connection instead of full
> verify using iperf3 -P 8 -R -c <server> # if you really getting
> more bandwidth than above
>
> 3. Check if MPTCP is enabled and configured on both sides
> uname -r # at least 5.10+ according [4] to get this balancing
> working, but 6.1+ LTS highly recommended (I've used 6.14.x)
> sysctl net.mptcp.enabled # should be 1 on both sides by default
> ip mptcp limits set subflows 8 add_addr_accepted 8 # but feel
> free to setup max limits
>
> 4. Configure MPTCP endpoints on the server (registers some dedicated
> listening ports for MPTCP use so that there's no need to use multiple
> IP aliases or PBR):
> ps uaxw | grep -i mptcpd # check if mptcp daemon (path manager is
> running or not), it is NOT required in this case
> ip addr ls # let's assume 10.0.1.240 is my main IP on eno1 device,
> no need to add new IPs thanks to below trick:
> ip mptcp endpoint show # to verify
> #ip mptcp endpoint flush # if necessary
> # below registers ports 5202..5205 as LISTENing by kernel and
> dedicated for MPTCP subflows
> ip mptcp endpoint add 10.0.1.240 dev eno1 port 5202 signal
> ip mptcp endpoint add 10.0.1.240 dev eno1 port 5203 signal
> ip mptcp endpoint add 10.0.1.240 dev eno1 port 5204 signal
> ip mptcp endpoint add 10.0.1.240 dev eno1 port 5205 signal
> ip mptcp endpoint show # to verify
>
> 5. Configure the client:
> ip addr ls # here I got 10.0.1.250
> ip mptcp endpoint show
> ip mptcp endpoint add 10.0.1.250 dev enp0s31f6 subflow fullmesh #
> not sure fullmesh is necessary, probably not
> ip mptcp limits set add_addr_accepted 8 subflows 8
>
> 6. Verify that MPTCP works, rerun tests with mptcpize, e.g.:
> on server: mptcpize run iperf3 -s
> on client: mptcpize run -d iperf3 -P 1 -R -c <server> # should get
> better bandwidth but using just 1 MPTCP connection
> on server run PostgreSQL with listen_mptcp='on'
> on server: ss -Mtlnp sport 5432 # mptcp should be displayed
> on client: run basebackup/psql/..
>
> Sample results for 82MB table copy, it's 3x:
> $ time PGMPTCP=0 /usr/pgsql19/bin/psql -h 10.0.1.240 -c '\copy
> pgbench_accounts TO '/dev/null';'
> COPY 500000
> real 0m42.123s
>
> $ time PGMPTCP=1 /usr/pgsql19/bin/psql -h 10.0.1.240 -c '\copy
> pgbench_accounts TO '/dev/null';'
> enabling MPTCP client
> COPY 500000
> real 0m14.416s
>
> Sample results for pgbench of DB created with: pgbench -i -s 5,
> ~1076MB total due to WALs
> $ time /usr/pgsql19/bin/pg_basebackup -h 10.0.1.240 -c fast -D /tmp/test -v
> pg_basebackup: initiating base backup, waiting for checkpoint to complete
> pg_basebackup: checkpoint completed
> [..]
> pg_basebackup: base backup completed
> real 1m26.786s
>
> With PGMPTCP=1 set, it gets ~3x
> $ time PGMPTCP=1 /usr/pgsql19/bin/pg_basebackup -h 10.0.1.240 -c
> fast -D /tmp/test -v
> enabling MPTCP client
> pg_basebackup: initiating base backup, waiting for checkpoint to complete
> [..]
> pg_basebackup: starting background WAL receiver
> enabling MPTCP client
> [..]
> pg_basebackup: base backup completed
> real 0m30.460s
>
> Because in the above case, we have advertised 4 IP addresses/port of
> server to the client, we got the bump on a single socket (note: flows
> end up being hashed into various HTB classes is random depending on
> ports used you can get usually 2x .. 4x here). Also as there are two
> independent application-based connections here in basebackup (transfer
> + WALs), both get multiplexed (each with 4 subflows). If I would add
> more ip mptcp ports (server-side), we could get even more juice of
> course there, but it assumes one has that many paths. Some more
> advanced setups including separate policy-based-routed (ip rule)
> things are possible, and stuff like keeping the TCP connection highly
> available 0 even across ISP/interface (WiFi?) outages - is possible.
> It works transparently with SSL/TLS too - tested. Of course it won't
> remove the single CPU limitation of the tools involved (that's
> completely different problem).
>
> If it sounds interesting I was thinking about adding to the patch
> something like contrib/mptcpinfo (pg_stat_mptcp view to mimic
> pg_stat_ssl). Also as for the patch there were some places where
> socket() is being created (libpq cancel packet), but there's no
> purpose of adding MPTCP there I think.
>
> It is important to mention there are two implementations of MPTCP on
> Linux, so when someone will be googling there's lots of conflicting
> information:
> 1) Earlier one, required kernel patching up to <= 5.6, had
> "ndiffports" multiplexer built-in which worked mostly out of the box.
> 2) Newer one [3], already merged one into kernel today, a little bit
> different does not come with built-in ndiffports path manager. In this
> newer one, as shown above some more manual steps (ip mptcp endpoints)
> may be required, but mptcpd daemon which is managing (sub)flows seems
> to be evolving as the usage of this protocol is rising. So I hope in
> future all of those mptcp commands would be probably optional.
>
> Thoughts?
>
> -Jakub Wartak.
>
> [1] - https://en.wikipedia.org/wiki/Multipath_TCP
> [2] - https://www.rfc-editor.org/rfc/rfc8684.html
> [3] - https://www.mptcp.dev/
> [4] - https://github.com/multipath-tcp/mptcp_net-next/wiki/#changelog

This is rebased (v2) and revived attempt at (the very simple) MPTCP patch. It
allows bypassing single TCP connection limit speed present on most (all?)
clouds, as they are pretty constrainted in terms of single TCP connection
bandwidth available:

Hardware type | Max aggr bw | 1x TCP LAN | 1x TCP MAN| 1x TCP WAN
Physical server | <= 200 GbE | 1 CPU | 1-10Gbps | 1-10Gbps?
VM (Xen, VMware, etc) | <= 100 GbE? | 1 CPU | 1-10Gbps | 1-10Gbps?
lowend AWS VM | 5-40 Gbps | 5 Gbps | 5 Gbps | <= 5 Gbps
lowend AWS VM Cluster Pl. | <= 200 Gbps | 10 Gbps | 5 Gbps | <= 5 Gbps
highend AWS VM | <= 200 Gbps | 5 Gbps | 5 Gbps | <= 5 Gbps
highend AWS VM+ENA Express| <= 200 Gbps | 25 Gbps | 25 Gbps | <= 5 Gbps
lowend Azure VM | 3(!)-40 Gbps| 3(!) Gbps | 1.5-3 Gbps| 1.5-3 Gbps
highend Azure VM+acc. net.| <= 200 Gbps | 10-12.5Gbps| 10 Gbps | 1.5-3 Gbps

"1 CPU" is of couse not bandwidth, but single CPU core constraint. MAN is just
kinda shortcut for zone/local DC to fit into 80 char limit with the table.

How to use the patch

server:
sysctl net.mptcp.enabled # should be 1 on both sides by default
ip addr ls # to get main IP and device of server
ip mptcp limits set subflows 8 add_addr_accepted 8
# add two addtional listening TCP ports (for subflows)
ip mptcp endpoint add <myIP> dev <mydev> >port 5202 signal # +open it on fw
ip mptcp endpoint add <myIP> dev <mydev> >port 5203 signal # +open it on fw
set listen_mptcp=on in postgresql.auto.conf

client:
sysctl net.mptcp.enabled # should be 1 on both sides by default
ip mptcp limits set subflows 8 add_addr_accepted 8
PGMPTCP=1 pg_basebackup ... (or psql/COPY/whatever over TCP/libpq)

For sample raw performance numbers how much this can help please see recent
pg_basebackup patch [1].

Sample MPTCP results:
iperf3 single TCP stream: 15-23Gps with ENA Express (max what AWS provides)
iperf3 multiple TCP streams (-P 8): 50Gbps (max aggr. bandwidth)
iperf3 single MPTCP stream(!): ~41Gbps
so if there's app that uses 1 socket (literally anything libpq related), you
can easily get ~41Gbps (or much more with this, while on the network supports
maximum X Gbps per single TCP connection speed - as per table, affects every
major cloud infrastructure provider).

To best of my knowledge bandwidth is limited only the by the MPTCP kernel-side
stream (subflow) reassembly on the CPU core where the app is running (Recieve
Steer Side is working fine for all of the subflows so they get different cores,
it's just the final reassembly part that needs to happen before gets it's data
using recv()/recvfrom()).

-J.

[1] - https://www.postgresql.org/message-id/CAKZiRmwwW-hDc3B6ERJB%2BpaX7RNSBcQLheq1KdsTf42cGuRvuA%40mail.gmail.com