I

I
I Report No. 5943

I
I
I /

t Congestion Control Study

I Final Report

I S. Calm, S. Eisner, M Frishkopf, J. Robinson, and J. Wiggins

I
I
I I
I Contract 'No. DCA 200-83-C-0021
Task Order 3-23
I
July 1985
I
I Prepared for:
Defense Communications Agency·
I
I
I
I
I Report No. 5943 BBN Communications Corporation

I -Table of Contents

I
I 1 INTRODUCTION ......................................................................................................... 1

I 1.1 The Proposed Congestion Control Algorithm ............................................................

1.1.1 Origins of Network Congestion ...............................................................................
1.1.2 Goals of Congestion Control ...................................................................................
2
2
3

I 1..1.3 Overview of the Algorithm ......................................................................................

1.2 Congestion Control Design Issues ..............................................................................
1.2.1 Characterizing Resource Loadings ...........................................................................
4
6
6

I 1.2.2 Throttling Offered Traffic ........................................................................................

1.2.3 Interactions With Delay-Oriented Routing .............................................................
8
9

I 2 SIMULATION EXPERIMENTS ................................................................................

2.1 Introduction ..............................................................................................................
2.1.1 The Congestion Control Algorithm as Simulated .................................................
10
10
12

I 2.1.2 Terminology and Interpretation of Simulation Experiments ................................

2.2 Local Analysis of the' Throttliilg Algorithm .................................................... :........
14
15
2.2.1 Throttling Traffic Sources .................. :.................................................................... 16

I 2.2.1.1 Description of the Simulation Model.. ................................................................

2.2.1.2 Criteria for the Evaluation of Throttling Policies ..............................................
17
19
2.2.1.3 Evaluation of Basic Throttling Feedback Model ............................................... 21
I 2.2.1.4 Modifications to Throttling Feedback Model ..................... :...............................
2.2.1.5 Analysis of Throttling Simulation Experiments .................................................
29
35
2.2.2 ARPANET Simulations: Responsiveness .... :......................................................... 36
I 2.2.3 Delay and Smoothing Properties of the Throttler Queueing System ...................
2.2.3.1 Theoretical Considerations ..................................................................................
51
51
2.2.3.2 ARPANET Simulator Simulations ..................................................................... 55
I 2.2.3.3 Further Modeling of the Throttler Queueing System ........................................
2.2.3.3.1 A Simulation Model .........................................................................................
58
59
2.2.3.3.2 An Analytic Model ........................................................................................... 60
I 2.2.3.4 Conclusions on Throttler Modelling ...................................................................
2.3 Congestion Control and SPF Routing .....................................................................
64
66
2.3.1 Unequal Paths and Very Light Traffic .................................................................. 74
I 2.3.2 Unequal Paths and Light Traffic ...........................................................................
2.3.3 Unequal Paths and Moderate Traffic ....................................................................
75
75

I 2.3.4 Unequal Paths and Heavy Traffic ............... ~ .........................................................

2.3.5 ,Equal Paths and Light Traffic ................................................ ~ .............................
2.3.6 Equal Paths and Moderate Traffic ........................................................................
75
82
82

I 2.3.7 Equal Paths and Heavy Traffic ............... ,......................... ;.. :................................ 87

I -1-
 •
Report No. 5943 BBN Communications Corporation
•
2.3.8 Conclusions from Rou ting Experimen ts ................................................................ 87
2.4 Global Analysis of Congestion Control ........................................ ;........................... 91
2.4.1 No Contention For Congested Resource ............................................................... 93
•I
2.4.1.1 Congestion Control Off ....................................................................................... 94
2.4.1.2 Congestion Control On ............................ ~ .. :::-:-;.~:~:-:-:: ........................................... 98
2.4.2 Con ten tion for Congested Resources............... ........................ ............ ................ 103 I
•
2.4.2.1 Congestion Control Off ...................................................................................... 105
2.4.2.2 Congestion Control On ..................................................................................... 112
2.5 Concluding Remarks on Global Experiments ......................................................... 115

·3 MEASUREMENTS OF CONGESTION IN TEST AND OPERA-

TIONAL NETWORKS ...........................................................................................
3.1 Tools for Measurement Experimen ts ......................................................................
3.1.1 Test Network Setup .............................................................................................
3.1.2 Modification to Message Generator .....................................................................
117
119
120
120
•I
•
3.1.3 Enhanced Store-and-Forward Statistics Package ................................................ 123
3.1.4 . Statistics Processing Facilities ............................................................................. 128
3.1.4.1 Collective Processing ......................................................................................... 128
3.1.4.2 Individual Message Processing ..........................................................................
3.2' Experiments on, Test Network .................................................. ,. ~ ...........................
3.2.1 Offered Load vs. Traffic Pattern ........................... ~ ................................., ............
3.2.1.1 Purpose ......'.............. :.........................................................................................
3.2.1.2 Descri~tion of Experiments ...............................................................................
3.2.1.3 Results and Conclusions ....................................................................................
130
133
133
133
135
138
•I
3.2.2 Effect of Other Variables on Congestion Metrics ................................................ 146
3.3 Measurements of Real Traffic ..................................................................................
3.,3.1 Implications for Growth Control .........................................................................
147
153
I
3.4 BBNNET Congestion Experiment ......................................................................... 159
3.4.1 Experiment Procedures ........................................................................................
3.4.2 Results ..................................................................................................................
160
161
I
3.4.3 Summary ..,............................................................................................................
3.5 Summary and Recommendations ...........................................................................
162
163 I
4 COSTS AND IMPLICATIONS OF CONGESTION CONTROL ........................... 165
4.1 Evaluation of Cost in Link and Processor Bandwidth .......................................... 165
I·~.~.
~

4.2 Other Changes to Store and Forward .................................................................... 170

5 CONCLUSIONS ........................................................................................................ 176 I

5.1 Analysis of the Congestion Control Algorithm ...................................................... 176
5.1.1 The Choice of a Congestion Metric .....................................................................
5.1.2 ·Measurement Interval for Link Utilization ............................. ;............................
176
177
I
5.1.3 Congestion Control Responsiveness ..................................................................... 178

I
-11-
I
I Report No. 5943 BBN Communications Corporation

I 5.1.4 Evaluation of Throttling Feedback ...................................................................... 179

S.1.S Delay and Smoothing Properties of the Throttler ............................................... 180
I 5.1.6 Congestion Control and SPF Routing ....................................... ·..........................
5.1.7 Congestion Control Effect on Cross Traffic .........................................................
181
182
5.2 Benefits of Congestion Control ............................................................................... 183
I 5.3 DDN Performance Prior to Congestion ControL .................................................. 186

I 6

A
BIBLIOGRAPHY ...................................................................................................... 188

Background Loops as a Measure of IMP Idleness ................................................... 189

I B Effect of Other Variables on Congestion Metrics ..................................................... 191

I, B.1 Short vs. Long Packets ..........................................................................................

B.1.1 Description of Experiments ...................................................................................
B.1.2 Results and Conclusions ......................................................................................
191
192
193
B.2 Link Mismatch vs. Homogenous Link Speeds ........................... ................ ............ 195

•I B.2.1 Description of Experiments ............................................................................... ~.

B.2.2 Results and Conclusions ......................................................................................
B.3 Source vs. Spreading ~ongestion ............................................................................
B.3.1 Description of Experiments .......................................................................... :.......
B.3.2 Results and Conclusions ............................................................. :... ~ .............. ! .....
196
196
198
198
198

I C The ARPANET Simulator .......................................................................................

C.1 Overview .................................................................................................................
201
201
C.2 The Congestion Control Algorithm ....................................................................... 203
I C.3 Special Purpose Simulation Tools ..........................................................................
C.4 Statistical Analysis of Simulation Data .................................................................
207
208
C.S Configuration of the ARP ANET simulator .......................................................... 210
I D Simulator Command Scripts ..................................................................................... 214

I
I
I
I
I
I -iii-
t
I Report No. 5943 BBN Communications Corporation

I 1 INTRODUCTION

t This report describes work carried out during the final phase of the IMP Congestion Con-

I trol study. The goal for this phase was to conduct the experiments designed in the previous

t phase of the task and described in the Congestion Control Experiment Design Study, BBN

Report 5809 [1]. These experiments focused on testing and validating key assumptions of the

I congestion control algorithm proposed in the First Interim Report, BBN Report 5721 [2]. The

I experiments encompass (a) measurements of the behavior of the IMP under conditions of heavy

utilizations and (b) modelin~ of the behavior of proposed congestion control measures using
I simulation.

I Chapter 1 reviews the congestion control techniques investigated previously In the study,

a including some of the key assumptions and important unanswered questions from the first p.hase

I of the task. In Chapter 2, we present the results of many simulation experiments. Some of

these used special-purpose simulators to validate specific design aspects of the congestion control

I algorithm. The rest were carried out using the ARPANET simulator which we modified to

I simulate operation with congestion control installed.

I Chapter 3 presents results from a series of IMP measurement experiments carried out in

the BBNNET. These experiments demonstrated the exact behavior of the present IMP algo-

I rithms under conditions that led to congestion, and helped guide our design of the congestion

I control algorithm. In Chapter 4, we evaluate the ·cost of the proposed congestion control algo-

rithm, and discuss its implications for the rest of the IMP's store-and-forward software.
I
I -1-
 I
Report No. 5943 BBN Communications Corporation
•'I
Finally, Chapter 5 summarizes the conclusions reach~ during the experiments, and includes

recommendations for further work on the congestion control algorithm. I

Although we review the proposed congestion control scheme below, this report assumes
:1
that the reader is familiar with Report 5721. As the program of experiments is somewhat

different from that proposed in Report 5809, each experiment will be described herein, so that

reference to the latter report is unnecessary.

•I
I
1.1 The Proposed Congestion Control Algorithm I
1.1.1 Origins· of Network Congestion I
Network congestion results from heavy utilization of the resources of the.store-and-forward I
(S/F) portion of the network, which transports packets between source and destination IMPs.

These resources are the network trunk lines, node store-and-forward buffers, and node CPU.
I
Given a network configured properly for its design load, congestion arises when one or more net- I
work components must perform beyond the design load. The symptom of congestion is repeated
I
refusal by a node to accept traffic from a neighbor node. Retransmission of refused packets

wastes network trunk resources. From a host's point of view, congestion increases network I
delays and reduces network throughput.
I
Congestion at one node due to oversubscription of some resource can also cause heavy I
loads to spread to other nodes, causing long queues and increas~d resource utilization to develop
I
-2- I
t
I Report No. 5943 BBN Communications Corporation

r 10 them as well. Spreading congestion- is· particularly wasteful, because it can degrade perfor-

t mance for traffic flows not contributing directly to the original overload.

I
I 1.1.2 Goals of Congestion Control

I Congestion control aims to prevent the waste of network re,sources. To do this it must

reduce the traffic flows that are contributing to resource overutilization by restricting them at

I their sources. Ideally, the network would achieve the theoretical maximum network throughput

I for the offered flows. In practice, congestion control will rarely be optimal. At times,

throughput may be reduced by congestion. At other times .congestion control may reduce
I throughput excessively. However, on average, congestion controi promises to stabilize network

l response to overloads and permit heavier utilization of the network's resources with correspond-

ing increases in throughput.

I
The goal of a congestion control system is to recognize when congestion has occurred, or is
I about to occur, and act to limit its effects. In particular, an effective congestion control tech-

I nique should prevent overloads caused by spreading congestion.

I The congestion control technique proposed in the First Interim Report is a global algo-

I rithm. Each IMP measures the condition of its own store-and-forward resources, and pro-

pagates the resulting congestion information to all the IMPs in the network. IMPs restrict

I flows originating from their attached hosts according to the state- of the resources that those

I flows reqUire. In this way, traffic that requires oversu bscribecl resources is held aside at the

I -3-
 II
Report No. 5943 BBN Communications Corporation I
II
boundary and metered into the network, while traffic that is not using congested resources flows

unregulated into the network.

}I
1.·1
!
. .·.·.·.

1.1.3 Overview of the Algorithm III ..

Congestion control consists of four major components which run in every network node.
!I
The first monitors a node's own SjF resources, and assigns congestion levels to its CPU, buffers,

and each of its network links. The second component propagates these levels to all nodes in the
11
network. The third processes the database of congestion information at each node and assigns a

congestion state to each destination. The assigne~ state depends upon the congestion levels of

all resources on the path to the destination. The fourth component uses this information to
'I
compute a throttle rate for traffic entering the S/F network for each destination, depending upon

the congestion state of that destination.

I
The resources which congestion control monitors are utilization of network trunks, utiliza-

tion of the buffers assigned to store-and-forward, and utilization of node CPU. Each node

assigns to each of its outgoing links, and to itself, a congestion level based on the measured util- I
izations.
I
The algorithm proposed in Report 5721 reduces the utilization information measured at
I
each node to one of several congestion levels to be propagated to all other nodes in the network.

Analysis and simulation during this phase of our study demonstrat-ed that use of just a few lev-
I
els was an insufficient basis for throttling. Whereas the original·scheme broadcast a qualitative I
-4-
I
I Report No. 5943 BBN Communications Corporation

I measure of co~gestion for each resource, we revised the scheme to use a quantitat2've measure.

I Each node calculates locally, for each of its resources, a congestion level to be used directly by

I all nodes to throttle traffic flows sharing that resource. The level is the node's best estimate of

the rate at which each source should admit traffic that must use each resource. By commUnI-

I cating a traffic rate, the system can adapt to the capacity of each resource.

I The congestion level information is flooded through the network, much as routing informa-

I tion is today. Given a routing tree and the per-link and per-node congestion level, each IMP

assigns a traffic congestion state to every destination. This congestion state is merely the

I minimum congestion level of any of the links and nodes along the current routing path from the

I IMP-to the desti~ation. A straigfitforward addition to the" incremental SPF routin.g calculation

can keep this information up to date for all destinations.

I
I Given the congestion state for a particular destination, the throttling component computes

the throttle rate. The throttle rate controls the traffic flow from an IMP's attached hosts into

I the store-and-forward subnetwork for that destination. Throttling acts like a simple negative

I feedback system. As load increases, Ll\1Ps with heavily-loaded resources reduce the correspond-

ing congestion levels, causing all IMPs to throttle back on the traffic offered by connected hosts

I that must use those resources. To enforce throttling rates, the SjF portion of the IMP requires

I that traffic offered faster than the throttle rate wait. It is the responsibility of the end-to-end

module to queue packets required to wait until the subnetwork can accept them, and to inform

I attached hosts of the traffic reduction using appropriate network protocol means.

I
I -5-
Report No. 5943 BBN Communications Corporation

1.2 Congestion Control Design Issues

The First Interim Report identified areas that required further study prior to final design

of the congestion control algorithm. This report presents results from a program of experiments

aimed at proving the basic approach proposed for controlling congestion, and tuning parameters

of the design for effective operation in the C/30 IMP. The following subsections discuss the

areas in which experimentation offers guidance and solutions to unresolved issues associated

with the congestion control algorithm.

1.2.1 Characterizing Resource Loadings

The first portion of the congestion control algorithm must measure utilization of the

potentially oversubscribed resources needed for store-and-forward processing of network traffic.

The experiments investigate whether the quantities to be m~asured provide an effective basis

upon which to perform congestion control. Although we define congestion in Report 5721 as

the occurrence of packet refusals on IMP-IMP trunk lines, we seek other measures which might

anticipate imminent congestion, in the hopes that congestion control might react quickly enough

to avoid the refusals.

I
We investigated the behavior of several measures under a variety of traffic scenarIos In
I
order to understand how they might be used to drive a congestion control algorithm. We used

steady' state traffic to find the level of offered load where congestion starts, and monitored the
I
behavior of the various measures when the load on the link is near that value. We also

-6- I
 Report No. 5943 BBN Communications Corporation

modified the message generator to provide a load that fluctuates over time, so that the reaction

of the measures to changes in loading could be observed.

In addition to these controlled experiments, we investigated the behavior of real traffic in

the BBNNET. One experiment looked at the behavior of the measured delays when a critical

link was removed, which caused congestion to spread throughout one portion of the net.

Another experiment looked at cumulative statistics data from typical working days to charac-

terize the load offered in a live network. These investigations allowed us to evaluate our

assumptions about traffic patterns used in the modeling.

In t~e experiments proposed' in Report 5809, we also expressed an interest, in characterizing

loads on the IMP's CPU and on its store-and-forward buffer pool. In both these cases, we
~
V decided that experiments would not teach us anything at this time.

In the case of CPU utilization, any measure that we might propose would be highly depen-

dent on the current structure of operational IMP software, especially the old end-to-end

software. This software will be replaced before congestion control is implemented, so any

results we could glean in this area would soon be obsolete. Additionally, the new end-to-end

software structure will provide much better measures of CPU utilization than are available

under the curren t release of the IMP software.

Conversion of DDN IMPs to Cj30E processors reduces the likelihood of exhausting S;F

buffers' because the memory available in the C j30E allows substant"ially more IMP buffers. The

IMP can dedicate one buffer to SjF for each logical channel on' each trunk. Therefore, as long

-7-
Report No. 5943 BBN Communications Corporation
•
as lines are configured with only 16 cha~nels, the IMP will never refuse a packet due to the una- •
vailability of a SjF buffer, and the congestion control algorithm can safely ignore the state of

the SjF buffer pool. This situation changes if the larger C/30E memory is filled with more
•
•
software and data structures, or more channels are assigned to each link. However, the question

of S jF buffer utilization can be postponed to a later date.

•
1.2.2 Throttling Offered Traffic

The congestion control algorithm sets target throughput rates for traffic. It throttles flows
•,I
contributing to overloads, and increases the throughput allowed to those which pass through
,I······
1

underutilized components. These factors raise several issues in the congestion control design

that require analysis. I

One such issue is achieving fairness under conditions where hosts must be throttled. A fair I
algorithm prevents some users from grabbing large chunks of a resource while other hosts wish

to use the same resource. A suitable measure of fairness must be defined, and the algorithm
I
must be tested to see if it meets this measure. Another issue is the stability of the feedback I
process; the congestion control algorithm should not lead to destructive oscillations. These
I
might occur, for example, when traffic is severely throttled, leading to underutilization, and then

allowed to dramatically increase, leading to congestion. At the same time the algorithm must I
respond as rapidly as possible as conditions change. The responsiveness should be good both I ',..•8%

when loads are building up and when they have subsided.

-8-
I
I Report No. 5943 BBN Communications Corporation

I We investigated these questions in the context of both the original congestion control algo-
I rithm proposed in Report 5721 and the modified algorithm based on broadcasting congestion

I levels using simulations built specifically for this purpose.

I
1.2.3 Interactions With Delay-Oriented Routing

I As discussed in the interim report, congestion control and delay-oriented routing may work

I at cross purposes because the former focuses on the throughput performance of the network,

I while the latter attempts to minimize delay. We investigated the conditions under which this

tension leads to destructive interacti~ns between the two algorithms.

I
l
For example, the routing algorithm acts fairly rapidly in response to changing delay condi-

tions. In part, this leads to the benefit that increases in load cause the network to seek better

I alternate routes so as to dissipate the congestion, but it can also result in oscillations in routing

I paths. Once congestion control is in place, then perhaps routing may be able to respond in a

more leisurely fashion in maintaining paths of minimum delay.

I We conducted a number of experiments using the ARPANET simulator, enhanced with

I the modified congestion control algorithm, to address this question.

I
I
I
I -9-
Report No. 5943 BBN Communications Corporation

5 CONCLUSIONS

5.1 Analysis of the Congestion Control Algorithm

5.1.1 The Choice of a Congestion Metric

An important assumption of the original congestion control algorithm. proposed in Report

5i21 was that resource utilization would be the proper metric to drive the control mechanisms.

Duri~g the cou!se of these experiment!), we confirmed this assumption.

The IMP' measurement experiments documented the behavior of routing delay, channel

utiliz~tion, IMP·IMP protocol refusals, and li~k utilization under a variety of scenarios" We.

-
found that refusals may happen under conditions of low utilization. The observation that

refusals and low utilization coincide means that, although the link logical channels are some·

times all busy, there is lots of time during which the link is in fact idle. This is not congestion

at all, but rather an improper configuration. Even though all the logical channels are in use, •
the link -cannot be fully utilized. Since this situation occurs most severely with spreading conges-
•
tion, restricting the flows is undesirable, since some of them may in fact be cross traffic which is

•
•
not contributing to the source of congestion. That the link utilization measure fails to detect

congestion in this situation is a point in its favor. The correct response to this situation is to

increase the number of logical channels, not to restrict the traffic flows using the link.

Conversely, the other parameters appear to be poor indicators of chronic congestion, I ~'!
;·J·.

because they will react when there is bursty traffic but no congestion. In addition, the other
I
-176- I
I
I Report No. 5943 BBN Communications Corporation

I
indicators would show congestion under conditions of spreading, or when link parameters are set
I wrong.

I Link utilization is best measured in bits rather than packets. We found good agreement

I' between the link 'utilization and chronic congestion with this measure for a variety of packet

SIzes. Given the wide variance in the packet sizes seen in user traffic, a utilization measure com-
I puted from the measured packet rate will either fail to see congestion due to a series of long

I pack~~s, or report a link congested when it is merely seeing a moderate load due to short pack-

I ets.

I
5.1.2 Measurement In.terval for Link Utilization

I
Setting the' time interval between congestion level updates is constrained by practical con-

I siderations. In particular, the cost in processor and link ov~rhead, given assumptions about

I update length and processor speed, must not be too high or the improvement in network perfor-

mance due to congestion control will be offset by the resources lost to running the alg9rithm.
I
Practical considerations yield both an upper and a lower bound on the rate at which
I update packets should be flooded by the IMP. The lower bound used for routing, which has

I similar costs, is set to 10 seconds, while the upper bound is determined by the need to guarantee

that any IMP can recover its routing database before isolated network segments are rejoined.
I Although the congestion control database is less critical than the routing database, it must be

I updated no less frequently.

I -177-
 I
Report No. 5943 BBN Communications Corporation
I
I
The range of update intervals we are thus left to consider is between 10 and 60 seconds,

which is the same range allowed to routing. A brief examination of the temporal characteristics I
of real traffic indicates that utilization measures probably need to be averaged over an interval
I
of at least 30 seconds to be stable. Therefore, we recommend that the initial congestion updat-

ing interval be set at 30 seconds with a range between 10 and 50 seconds configurable for test
I
purposes. In order to determine the utilization averaging interval, further measurements of I
traffic in real networks should be undertaken.
I
5.1.3 Congestion Control Responsiveness
I
We examined the speed with which the simulated congestion control .algorith~ would
I
respond to changes in offered load. The load might change by either an .increase or ~ decrease I
in traffic using an S/F resource. We examined both scenarios.
I
With an effective threshold/decay scheme, we found that congestion control could respond
I
promptly to increases in offered load. The delay to restore acceptable line utilization was typi-

cally two throttling intervals, and to achieve target utilization, eight intervals. When offered I
load decreased, the congestion control algorithm was quite responsive, correcting throttling rates
I
within two throttling intervals.
I
I
I
-178- I
I
I Report No. 5943 BBN Communications Corporation

I
5.1.4 Evaluation of Throttling Feedback

I
The throttling feedback algorithm proposed in Report 5721 assumed that classification of
I resource loading into one of a few possible levels would suffice for driving offered load to a desir-

I able rate. During this phase of the task, we modeled the behavior of this feedback mechanism

to satisfy ourselves that it would operate effectively under realistic conditions.

I
I Our experiments indicated that the qualitative approach is grossly inadequate to meet the

goals of an effective congestion control algorithm. In configurations where many sources com-

I pete for a single link, control might fail altogether. In addition, the parameters that drove the

I computation of the next throttling rate could not be set· properlr in a network having a variety

of link speeds. For these reasons, we abandoned the original feedback model altogether.

I We designed and modeled a new throttling feedback method which performed far better

I than the original algorithm in every respect. Briefly, each resource con tin uously main tains a

I congestion level, which it modifies based on utilization measured during the previous measure-

ment interval. To the extent that the measured utilization fails to meet the target utilization

I for that resource, the IMP modifies· the corresponding congestion level, subject to an upper

I bound derived from the capacity of the resource. That is, given a congestion level L, measured

utilization M, and desired utilization D, the IMP computes the new value of the congestion level
I L to be Lx(D-7M), subject to the resource capacity. Although we only modeled this congestion

I level computation method for link resources, it ought to be applicable to buffer and CPU

resources when they are incorporated into the congestion control algorithm.
I
I -179-
 I
Report No. 594.3 BBN Communications Corporation I
I
This congestion level method was used in the ARPANET simulator, and once again, we

were able to show that it performed well under a variety of traffic scenarios. In both the I
special-purpose simulation and the ARPANET simulator, the averaging interval for measuring
I
utilization and the interval for the throttler were set equal to 10 seconds.
I
One problem we found with the formulation of the throttling algorithm related to the use

of a throttling averaging interval. Traffic arriving near the beginning of the metering interval is
I
delayed more than that arriving near the end. One could modify the throttling algorithm to
.. - I
remove this asymmetry, and eliminate the need for a throttling averaging interval. This

modified throttler could adapt instantly to new congestion levels. We did not have enough time
I
to model or simulate. its behavior, but we recommend that it. be examined in greater detail in I
the follow-on project. Executing the modified throttling algorithm in the IMP should be

cheaper than executing the original one proposed.

I
I
5.1.5 Delay and Smoothing Properties of the Throttler I
In queueing analysis of the throttler, we both observed and calculated that throttler queu- I
ing delays increase rapidly as throttler utilization approaches unity. Therefore, even when the
I
throttling rate exceeds the average offered load, we must beware of overthrottling to avoid

excessive queueing delays.

I
I
I
. -180- I
Report No. 5943 BBN Communications Corporation

The throttler appears to function effectively as a traffic smoother. In simulation experi-

ments the throttler eliminated most refusals, without worsening delay or throughput. The

applicability of this positive result to larger networks bears further investigation to verify that

the benefits due to smoothing traffic as it enters the network persist when packets must make

many hops to reach their destination.

5.1.6 :~ongestion Control and SPF Routing

In ARPANET simulator experiments, we investigated the behavior of routing under load

in a simple topology, and the effect of routing changes on the utilization metric. We showed

that high traffic causes routing to oscillate badly, and that utilization measurements taken du~-

ing this time are likely to be both deceptive and unpredictable indicators of congestion. Rout-

ing is particularly susceptible to oscillations when two paths to a destination are of equal length.

W~en we simulated both the congestion control and routing algorithms, we observed poor

congestion control responsiveness to heavy traffic. We offered a tentative hypothesis to explain

this observation based on the destructive interactions between routing and congestion control

diagrammed in Figure 3-24. We conclude that the frequency of routing updates should be

decreased to ensure congestion control effectiveness. This could be accomplished either by

increasing the routing delay threshold or by lengthening the routing averaging interval.

-181-
 I
Report No. 5943 BBN Communications Corporation
I
I
These measures may result in less effective, though passable, routing decisions. In particu-

lar, routing will still react to changing network topology. Finer tuning of the intervals for both I
algorithms will require further simulation and experimentation in real networks. This work
I
should form part of the Congestion Control Implementation task.
I
For the longer term, we should consider extensions to congestion control which take

account 'of routing behavior, in an effort to guarantee productive cooperation between the two
I
algorithms. We should also investigate altern~tive routing algorithms which might coexist I
naturally with congestion control. These are open issues and require further study.
I
5.1.7 Congestion Control Effect op. Cross Traffic
I
In addition to examining specific design choices for the congestion control algorithm, we
I
attempted to verify that the algorithm worked as intended in situations for which it was I
designed: We used the ARPANET simulator to model the effects of congestion control and
I
throttling on cross traffic. Cross traffic is traffic that might suffer increased delay due to the

refusals resulting from congestion of another link that it is not trying to use.
I
With congestion control disabled, we observed cross traffic delay as congesting traffic levels
I
were increased incrementally. When congesting traffic reached a certain level, cross traffic delay I
increased abruptly. From these experiments, it appears that the effects of network congestion

on cross traffic are sudden and arIse 10 a very narrow range of utilizations of the congested
I
resource near 90%. I
-182- I
 Report No. 5943 BBN Communications Corporation

Congestion control effectively prevented most refusals, resulting in uniformly low cross

traffic delay independent of congesting traffic level. However, the target link utilization of 80%

caused the throttling rate to be unnecessarily low. Further study is required to determine an

optimal target utilization, if possible. More ambitiously, we should extend the congestion con-

trol algorithm to determine dynamically the target utilization for each link, perhaps usmg a

feedb~k mechanism driven by some other metric, such as packet refusal rate.

To generate extreme network congestion, we configured a network where multiple sources

all tri~d to use a single 9.6 Kb/s link. In this configuration, oversubscription of the 9.6 Kb/s

link was rapidly alleviated by congestion contr~l. Cross traffic packet delay swiftly dropped to

values characteristic of an uncongested network.

The congestion control feedback mechanism appears to be effective in dynamically adjust-

ing congestion levels when link utilizations approach unity. In the small network simulated,

both end-to-end delay and throughput performance improved as a result of congestion control.

Further -work in this area should examine networks where routing may affect the ability of

congestion control, and examine larger, more complex networks.

I
5.2 Benefits of Congestion Control

I
In summary, the benefits to be realized from congestion control are the following:
I
• Less waste of network resources under overload. Congestion control will cause sources
I
I -183-
 I
Report No. 5943 BBN Communications Corporation I
to be throttled which are contributing to overutilization of busy network links. As a
I
result, fewer packets will be refused due to output queue limits (logical channel lim- I
its) for those links. This, in turn, will result in fewer retransmissions under heavy
I
load conditions, and less waste of network resources.

I
• Better coping with heterogeneous links. One particularly dangerous point for conges-

tion to build is where traffic from a higher speed link is routed along a lower speed I
....
link. Congestion control will both lower the likelihood that such overloading may
I
occur, by initially throttling any source to no more than the link capacity, and will

enable the network to meter traffic at its sources when several streams of traffic com-
I
bine to overload the link.
I
• Less delay for noncongesting traffic. Because fewer packets will be refused on links I
leading to overload points, there will be less added delay for flows that must use

these links but avoid the overload point. In other words, since the network is wast-
I
ing fewer resources, it can better serve traffic which should not be affected by I
resource overloads.
I
• Less delay variance under overload. Retransmissions due to link refusals are a source I
of long reported delays in the routing algorithm. Avoidance of refusals will also

reduce the variance of delay reported by the routing algorithm, which will lead to I
more stable routing patterns and more predictable delay through the subnet. This,
I
in turn, should allow better performance by the end-to-end protocol.
I
-184- I
I
I Report No. 5943 BBN Communications Corporation

I
• Allow routing to perform better. As a result of the delay variance being reduced, the
I routing algorithm should be less apt to send updates. Moreover, the necessity for

I routing to respond rapidly to perceived suboptimalities in the routing tree is

reduced, and thus it may be prudent to allow routing to run even less often than is
I now the case. This should reduce the likelihood of unproductive routing oscillations.

I • Allow more logical channels per link. Because the global congestion control algorithm

I proposed outperforms the current local algorithm (limited logical chaimels), it is no

I longer critical that an IMP begin refusing traffic as its queues begin to grow. If this

purpose of the configured number of logical channels is removed, then the number

I may be increased,-leading to better link performance even with short packets. This

I will be another way that waste of line resources will be avoided.

I • Allow larger end-to-end windows and uncontrolled packets. Since the network can

respond to overloads by the throttling mechanism at the sources of traffic, it need no

I longer rely on the end-to-end window size to artificially restrict the amount of traffic

I that may be allowed to enter the S/F subnetwork. Therefore, end-to-end protocol

windows can be set to reflect the actual needs of the end-to-end protocol, rather
I than having to be set conservatively to help limit the exposure of the network to

I congestion. This should permit greater throughput, especially in networks with

high-speed satellite links. Moreover, because the throttling mechanism limits all
I packets regardless of their source, datagram packets will also be throttled, and thus

use of datagrams should not place the network in as much jeopardy as is the case

-185-
Report No. 5943 BBN Communications Corporation

today.

5.3 DDN Performance Prior to Congestion Control

Prior to the deployment of congestion control software, the network will remain vulnerable

to congestion incidents. It will therefore have to continue to protect itself as it does today:

, . New end-to-end protocol windows will have to stay small. Because the network
•
currently relies on the end-to-end. protocol to limit the amount of traffic entering the

SjF network, the new end-to-end protocol will have to resp~ct the limits used in the

old end-to-end to avoid ·chronic overloads of network resources. These limits will

constrain throughput in some situations.

•
• There may be local congestion, especially where there are low-speed links fed by higher

speed links. We have already observed congestion incidents in the BBNNET where •
low-speed links are fed by high speed links. This will continue to occur until the
•
c6ngestion control algorithm is deployed. For some traffic distributions, increasing

•
•I
logical channels will help reduce the impact of these incidents.

• Routing and logical channel limits will have to be tuned to avoid congestion, and delay

may be higher than necessary. The IMP will have to continue to deal with congestion

as it has up to the present, by restricting the number of logical channels in use on I

high-speed links which feed lower speed links and by allowing routing to act quickly
I
-186- I
Report No. 5943 BBN Communications Corporation

when it inight find a less congested path through the network. These policies will

occasionally lead to waste.

After deploying congestion control software we can consider relaxing some of the current

conservative policies toward resource management in the IMP. Many of these originated when

the IMPs had only a few tens of buffers to use for meeting all demands placed upon them.

With vastly larger memories that are now becoming standard on the DDN, it is possible to allay

some, of the suboptimalities listed above. In particular, we believe that increasing the number

of log)cal channels will allow the IMP to ride out traffic bursts better than it now does, and

reduce the effects of spreading congestion.

-187-
Report No. 5943 BBN Communications Corporation

6 BIBLIOGRAPHY

[1] M. Frishkopf, J. Mayersohn, J. Robinson, J. Wiggins, Congestion Control Experi-

ment Design Notes, BBN Report No. 5809, December 1984.

[2] J. Robinson, S. Cohn, S. Eisner, K. Laube, J. Mayersohn, Congestion Control, First

Interim Study Report, BBN Report No. 5721, July 1984.

•
[3] E.C. Rosen, J. Mayersohn, P.J. Sevcik, G.J. Williams, R. Attar, ARPANET Routing

•
•
Algorithm Improvements, Volume. 1, BBN Report No. 4473, August 1980.

•
[4] J.M. Mcquillan, 1. Richer, E.C: Rosen, D.P. Bertsekas, ARPANET Routing Algo-

rithm Improvements, Second Semiannual Technical Report, BBN Report No. 3940,

October 1978.

•
J.F. Haverty, B.L. Hitson, J. Mayersohn, P.J. Sevcik, G.J. Williams, ARPANET
•
Routing Algorithm Improvements, Volume 2, BBN Report No. 4931, March 1982.

•
[6] D.E. Knuth, Seminumerical Algorithms, The Art of Computer Programming, Volume

2, Addison-Wesley, 1969. •I
[7] J.M. McQuillan, 1. Richer, and E.C. Rosen, ARPANET Routing Algorithm Improve-

ments, First Semiannual Technical Report, BBN Report No. 3803, April 1978. I
[8] L. Kleinrock and M. Gerla, "Flow Control: A Comparative Survey," IEEE Transac- I
tions on Communications, v. COM-28, no. 4, April 1980, p. 553.
I
-188- I