Multipath TCP

Costin Raiciu

Christoph Paasch

University Politehnica of
Bucharest

Universit catholique de Louvain

Joint work with: Mark Handley, Damon Wischik, University

College London
Olivier Bonaventure, Sbastien Barr, Universit
catholique de Louvain
and many many others
Thanks to

Networks are becoming

multipath

Mobile devices have multiple wireless

connections

Networks are becoming

multipath

Networks are becoming

multipath

Networks are becoming

multipath

Datacenters have redundant

topologies

Networks are becoming

multipath

Client

Servers are multi-homed

How do we use these networks?

TCP.
Used by most applications,
offers byte-oriented reliable delivery,
adjusts load to network conditions

[Labovits et al Internet Interdomain traffic Sigcomm 20

TCP is single path

A TCP connection
Uses a single-path in the network
regardless of network topology
Is tied to the source and
destination addresses of the
endpoints

Mismatch between
network and transport
creates problems

Poor Performance for Mobile Users

3G celltower

Poor Performance for Mobile Users

3G celltower

Poor Performance for Mobile Users

3G celltower

Poor Performance for Mobile Users

3G celltower

Offload to WiFi

Poor Performance for Mobile Users

3G celltower

All ongoing TCP connections die

Collisions in datacenters

[Fares et al - A Scalable, Commodity Data Center Network Architecture -

Single-path TCP collisions reduce

throughput

[Raiciu et. Al Sigcomm 201

Multipath TCP

Multipath TCP (MPTCP) is an

evolution of TCP that can effectively use multiple paths
within a single transport connection

Supports unmodified applications

Works over todays networks

Standardized at the IETF (almost there)

Multipath TCP components

[Raiciu et. al NSDI 2012]

Connection setup
Sending data over multiple paths
Encoding control information
Dealing with (many) middleboxes
Congestion control
[Wischik et. al NSDI 2011

Multipath TCP components

[Raiciu et. al NSDI 2012]

Connection setup
Sending data over multiple paths
Encoding control information
Dealing with (many) middleboxes
Congestion control
[Wischik et. al NSDI 2011

MPTCP Connection
Management
SY N
LE
B
A
P
A
MP_ C
X

MPTCP Connection
Management
SYN/
MP_C ACK
APAB
LE
Y

MPTCP Connection
Management
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

MPTCP Connection
Management
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO

SYN
JOIN Y

FLO
WY

MPTCP Connection
Management
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO

CK
A
/
N
SY
X
JOIN

FLO
WY

MPTCP Connection
Management
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY
SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

TCP Packet Header

Bit 0

Bit 15 Bit 16

Source Port

Bit 31

Destination Port
Sequence Number
Acknowledgment Number

Header
Length

Reserved

Code bits

Checksum

20
Bytes

Receive Window
Urgent Pointer

Options
Data

0 - 40
Bytes

TCP Packet Header

Bit 0

Bit 15 Bit 16

Source Port

Bit 31

Destination Port
Sequence Number
Acknowledgment Number

Header
Length

Reserved

Code bits

Checksum

20
Bytes

Receive Window
Urgent Pointer

Options
Data

0 - 40
Bytes

Sequence Numbers
Packets go multiple paths.
Need sequence numbers to put them back in
sequence.
Need sequence numbers to infer loss on a single path.

Options:
One sequence space shared across all paths?
One sequence space per path, plus an extra one to
put data back in the correct order at the receiver?

Sequence Numbers
One sequence space per path is preferable.
Loss inference is more reliable.
Some firewalls/proxies expect to see all the
sequence numbers on a path.

Outer TCP header holds subflow sequence

numbers.
Where do we put the data sequence numbers?

MPTCP Packet Header

Bit 0

Bit 15 Bit 16

Subflow Source Port

Bit 31

Subflow Destination Port

Subflow Sequence Number

Subflow Acknowledgment Number
Header
Length

Reserved

Code bits

Checksum
Data sequence number

20
Bytes

Receive Window
Urgent Pointer

Options
Data

Data ACK

0 - 40
Bytes

MPTCP Operation
options

SEQ
1000

DSEQ
10000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY
SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
options

SEQ
1000

DSEQ
10000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY
SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
options

SEQ
1000

DSEQ
10000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
options

SEQ
1000

DSEQ
10000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
options

SEQ
1000

DSEQ
10000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation

ACK
2000

Data
ACK
11000

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY

options

SEQ
5000

DSEQ
11000

DATA

SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

MPTCP Operation
options

SEQ
2000

DSEQ
11000

DATA

SUBFLOW 1
CWND
Snd.SEQNO
Rcv.SEQNO
FLO
WY
SUBFLOW
2
CWND
Snd.SEQNO
Rcv.SEQNO

Multipath TCP

Congestion
Control

Packet switching pools circuits.

Multipath pools links
TCP controls how a link is shared.
How should a pool be shared?

Two circuits

A link

Two separate
links

42

A pool of links

Design goal 1:

43

Multipath TCP should be fair to regular

TCP at shared bottlenecks
A
multipath
TCP flow
with two
subflows
Regular
TCP

Tobe
befair,
fair,Multipath
MultipathTCP
TCPshould
shouldtake
takeas
asmuch
much
To
To be fair, Multipath TCP should take as much
capacityas
asTCP
TCPatataabottleneck
bottlenecklink,
link,no
nomatter
matter
capacity
capacity as TCP at a bottleneck link, no matter
howmany
manypaths
subflows
it is using.
how
it
is
using.
how many paths it is using.

Design goal 2:

44

MPTCP should use efficient paths

12Mb/
s
12Mb/
s

Each flow has a choice of a 1-hop12Mb/

and a 2-hop
s
To
be
fair,
Multipath
TCP
should
take
Topath.
be fair, Multipath TCP should take as
as much
much
capacity
capacity as
as TCP
TCP at
at aa bottleneck
bottleneck link,
link, no
no matter
matter
How
shouldpaths
split itsistraffic?
how
how many
many paths it
it is using.
using.

Design goal 2:

45

MPTCP should use efficient paths

12Mb/
s
12Mb/
s

8Mb/
s
8Mb/
s
8Mb/
s

12Mb/
s
To
as
To be
be fair,
fair, Multipath
Multipath TCP
TCP should
should take
take
as much
much

If
each flow TCP
split itsatraffic
1:1 ...
capacity
capacity as
as TCP at
at a bottleneck
bottleneck link,
link, no
no matter
matter
how
how many
many paths
paths it
it is
is using.
using.

Design goal 2:

46

MPTCP should use efficient paths

12Mb/
s

9Mb/s
12Mb/
s

9Mb/s
9Mb/s

12Mb/
s
To
as
To be
be fair,
fair, Multipath
Multipath TCP
TCP should
should take
take
as much
much

If
each flow TCP
split itsatraffic
2:1 ...
capacity
capacity as
as TCP at
at a bottleneck
bottleneck link,
link, no
no matter
matter
how
how many
many paths
paths it
it is
is using.
using.

Design goal 2:

47

MPTCP should use efficient paths

12Mb/
s
12Mb/
s

10Mb/
s
10Mb/
s
10Mb/
s

12Mb/
s
To
as
To be
be fair,
fair, Multipath
Multipath TCP
TCP should
should take
take
as much
much

capacity
If each flow
split a
its traffic 4:1 link,
...
capacity as
as TCP
TCP at
at a bottleneck
bottleneck link, no
no matter
matter
how
how many
many paths
paths it
it is
is using.
using.

Design goal 2:

48

MPTCP should use efficient paths

12Mb/
s
12Mb/
s

12Mb/
s
12Mb/
s
12Mb/
s

12Mb/
s
To
as
To be
be fair,
fair, Multipath
Multipath TCP
TCP should
should take
take
as much
much

capacity
If each flow
split a
its traffic :1link,
...
capacity as
as TCP
TCP at
at a bottleneck
bottleneck link, no
no matter
matter
how
how many
many paths
paths it
it is
is using.
using.

Design goal 3:

49

MPTCP should get at least as much as

TCP on the best path
wifi path:
high loss, small RTT

3G path:
low loss, high RTT

Design Goal 2 says to send all your traffic on

the
least
path,
in thistake
case
But
To
be
fair,
Multipath
should
as
much
To
be
fair,congested
Multipath TCP
TCP
should
take
as3G.
much
this has high
RTT,
it will give
capacity
as
at
aa bottleneck
link,
no
capacity
as TCP
TCP
at hence
bottleneck
link,low
no matter
matter
throughput.
how
many
how
many paths
paths it
it is
is using.
using.

How does TCP congestion control

work?
Maintain a congestion window
w.

Increase w for each ACK, by

1/w

50

How does MPTCP congestion control

work?
Maintain a congestion window wr, one
window for each path, where r R
ranges over the set of available paths.
Increase wr for each ACK on path r,
by

Decrease wr for each drop on path

r, by wr /2

51

How does MPTCP congestion control

work?
Maintain a congestion window wr, one
window for each path, where r R
ranges over the set of available paths.
Increase wr for each ACK on path r,
by

Goal 2

Decrease wr for each drop on path

r, by wr /2

52

How does MPTCP congestion control

work?
Maintain a congestion window wr, one
window for each path, where r R
ranges over the set of available paths.
Increase wr for each ACK on path r,
by

Goals 1&3

Decrease wr for each drop on path

r, by wr /2

53

How does MPTCP congestion control

work?
Maintain a congestion window wr, one
window for each path, where r R
ranges over the set of available paths.
Increase wr for each ACK on path r,
by

Decrease wr for each drop on path r,

by wr /2

54

Applications
of Multipath TCP

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

56

2 TCPs
@ 50Mb/s

100Mb/
s

100Mb/
s
4 TCPs
@
25Mb/s

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

57

2 TCPs
@ 33Mb/s
1 MPTCP
@ 33Mb/s
4 TCPs
@
25Mb/s

100Mb/
s

100Mb/
s

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

58

2 TCPs
@ 25Mb/s
2 MPTCPs
@ 25Mb/s

100Mb/
s

100Mb/
s

4 TCPs
@
25Mb/s

The total capacity, 200Mb/s, is

shared out evenly between all 8
flows.

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

59

2 TCPs
@ 22Mb/s
3 MPTCPs
@ 22Mb/s

100Mb/
s

100Mb/
s

4 TCPs
@
22Mb/s

The total capacity, 200Mb/s, is

shared out evenly between all 9
flows.
Its as if they were all sharing a
single 200Mb/s link. The two links

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

60

2 TCPs
@ 20Mb/s
4 MPTCPs
@ 20Mb/s

100Mb/
s

100Mb/
s

4 TCPs
@
20Mb/s

The total capacity, 200Mb/s, is

shared out evenly between all 10
flows.
Its as if they were all sharing a
single 200Mb/s link. The two links

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

61

5 TCPs
First 0,
then
10 MPTCPs

100Mb/
s

100Mb/
s

15 TCPs

We confirmed in experiments
that MPTCP nearly manages
to pool the capacity of the
two access links.
Setup: two 100Mb/s access

throughput per flow

[Mb/s]

time [min]

At a multihomed web server, MPTCP tries

to share the pooled access capacity fairly.

62

5 TCPs
First 0,
then
10 MPTCPs

100Mb/
s

100Mb/
s

15 TCPs

MPTCP makes a collection of links behave

like a single large pool of capacity
i.e. if the total capacity is C, and there are
n flows, each flow gets throughput C/n.

Multipath TCP can pool datacenter

networks

Instead of using one path for each

flow, use many random paths
Dont worry about collisions.
Just dont send (much) traffic on
colliding paths

Multipath TCP in data centers

Multipath TCP in data centers

MPTCP better utilizes the FatTree

network

MPTCP on EC2
Amazon EC2: infrastructure as a service
We can borrow virtual machines by the hour
These run in Amazon data centers worldwide
We can boot our own kernel

A few availability zones have multipath

topologies
2-8 paths available between hosts not on the same
machine or in the same rack
Available via ECMP

Amazon EC2 Experiment

40 medium CPU instances running
MPTCP
For 12 hours, we sequentially ran allto-all iperf cycling through:
TCP
MPTCP (2 and 4 subflows)

MPTCP improves performance on

EC2

Same
Rack

Implementing
Multipath TCP

in the Linux
Kernel

Linux Kernel MPTCP

About 10000 lines of code in the Linux Kernel

Initially started by Sbastien Barr
Now, 3 actively working on Linux Kernel MPTCP

Christoph Paasch

Fabien Duchne

Gregory Detal
Freely available at http://mptcp.info.ucl.ac.be

MPTCP-session creation

Application creates regular

TCP-sockets

MPTCP-session creation

The Kernel creates the

Meta-socket

MPTCP creating new subflows

The Kernel handles the

different MPTCP subflows

MPTCP Performance with apache

100 simultaneous HTTP-Requests, total of 100000

MPTCP Performance with apache

100 simultaneous HTTP-Requests, total of 100000

MPTCP Performance with apache

100 simultaneous HTTP-Requests, total of 100000

MPTCP on multicore
architectures

Flow-to-core affinity steers all packets from one

TCP-flow to the same core.
MPTCP has lots of L1/L2 cache-misses because the
individual subflows are steered to different CPUcores

MPTCP on multicore
architectures

MPTCP on multicore
architectures

Solution: Send all packets from the same MPTCPsession to the same CPU-core
Based on Receive-Flow-Steering implementation in
Linux (Author: Tom Herbert from Google)

MPTCP on multicore
architectures

Multipath TCP
on

Mobile Devices

MPTCP over WiFi/3G

TCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

MPTCP over WiFi/3G

WiFi to 3G handover with

Multipath TCP

A mobile node may lose its WiFi

connection.
Regular TCP will break!
Some applications support recovering
from a broken TCP (HTTP-Header Range)
Thanks to the REMOVE_ADDR-option,
MPTCP is able to handle this without the
need for application support.

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

WiFi to 3G handover with

Multipath TCP

Related Work
Multipath TCP has been proposed many
times before
First by Huitema (1995),CMT, pTCP, M-TCP,

You can solve mobility differently

At different layer: Mobile IP, HTTP range
At transport layer: Migrate TCP, SCTP

You can deal with datacenter collisions

differently
Hedera (Openflow + centralized scheduling)

Multipath topologies
need multipath
transport
Multipath TCP can be used
by unchanged applications
over todays networks
MPTCP moves traffic away from
congestion,
making a collection of links behave like a
single pooled resource

Backup Slides

Packet-level ECMP in
datacenters

How does MPTCP congestion control

work?

107

Maintain a congestion window

wr, one window for each
path, where r R ranges over
the set of available paths.
Design goals 1&3:
At any potential
bottleneck S that path
rlook
might
be in,
at the
best that
a single-path TCP
and
compare
to what Im
could
get,
getting.

Increase wr for each ACK on

path r, by

How does MPTCP congestion control

work?

108

Maintain a congestion window

wr, one window for each
path, where r R ranges over
the set of available paths.
Design goal 2:
We want to shift
traffic away from
congestion.
To achieve this, we
increase windows
in proportion to
their size.

Increase wr for each ACK on

path r, by