A Hop by Hop Rate-based Control Partho Systems P. Design of Hemant Analysis of Computing Group Computer University Scheme Mishra* and Department Congestion AT&T Science Maryland 1 Abstract sliding window per, the flow control mechanism service” fic, and congestion in switch the described in simulation [12]. for In study paper, comparing controls stable behavior and diverse formance for network of occupied network delay, better than here. These lar myths measured buffers, results about in terms present control of a hop- proach the per- popu- mechanisms. research was initiated oratories. This DARPA under work while the author was supported contract in part to University of Maryland. The views, opinions, contained in this report are those of the author(s) be interpreted as representing the official policies, or implied, of the Defense Advanced RL, or the U.S. Government, Permission granted direct to copy provided commercial title of the that copying specific ACM fee the copies otherwise, Research and at role in ensuring recent with all or part of this the ACM cessing and of the Association or to republish, material notice could these well rate-hazed as at these be utilized along seri- to play and an streams schedul- coet of implementing switchee power and in packet for traffic reservation additional flow approach designed of service ap- switch developments this scales The suggests, at each are being using that product. “hopby-hop” Recent in microprocessors this paper, and the control notice is given for Computing requires-a over each link. of the proposed is for is small. provide so that Moreover, packet a portion for signaling fixed and a sender. USA 12...$1.50 112 scheme mechanism window We have described we address 1The sljd,ng fee permission. 0-89791-526-7/92/0008101 power name quality processing of switches of the pro- and controlling Agency, or distributed copyright appear, controls to scale approach to consider video, However, expected is exercised [4, 5]. The advancee enormous the switches as compressed in range. stream. packet used adapting the amount [6, 13, 14]. different feasible active hopby-hop the and/or findings and should not either expressed Projects it Future traffic are not made and its data is by permission To copy COMM’92-8/92/MD, 01992 without advantage, publication Machinery. and/or that Labs UMIACS make ing mechanisms was at Bell Lab- by Rome F306O2-9O-C-CO1O of a traffic been and or reduce as hopby-hop control win- to the observed has bandwidth-delay As use a modified accurately, are not a fundamentally ously. such *This the network control. switching according to increase a network’s used modifications Both [14]. and to the gigabit approach the path is studied case against move the issue of flow con- of ingenuity rapidly into network the size of a sender’s adapted is characterized in this the with mechanisms congestion average schemes injected with scheme A lot congestion We pursue well conditions of the dynamically control nationwide Two of the most commonly mechanism conditions. networke flow control scheme dis- at the bottleneck, a convincing hopby-hop is throughput, control traffic flow a gigabit are the Jacobson-Karels the size of the window of results importantly, utilization of the end-to-end of of this being detecting traf- switch of traffic the end-to-end and the link that range More data packet performance a wide topologies. of the scheme, number “quality implementation by-hop flow control scheme to two end-to-end schemes. The results indicate that the proposed plays so- .com to re-examine control. protocols window network haa become we present the dow rate-hazed to provide at building [6] and the DECbit to TCP sliding pa- .att researchers flow control to the In this as well as bursty a multi-gigabit this move Due to the increasing A cost-effective scheme pa- for our search scheme. of hop-by-hop Hill trol and congestion end-to-end a hopby-hop architectures, inexpensive. proposed control for real-time the implementation in this similar as networks control. Laboratories efforts has led many is based on the the motivation aa an alternative, guarantees relatively other deteriorates haa been for congestion phistication and This we propose of TCP As we demonstrate of this schemes range. a new flow per, scheme mechanism. performance gigabit for control Bell Murray Center Introduction The current flow/congestion Research kanakia~research partho@cs,umd.edu The Kanakia Science and the associated the question is more window an implementation effective mechanism sizes to handle the costs in [12]. of whether in controlling was originally mismatch of rates In a hop-by-hop network designed with of a receiver traffic than an end-to-end control control mechanism. 3, we compare TCP Intuitively, hopby-hop advantages, tion particularly bandwidth pect delay a group and control subject the mechanism in its feedback control hopby-hop and accurate support mechanisms of a network, malicious control users. restrict well behaved The simulation these intuitive leads to Finally, traffic since overloads to presented controls the sending rates congestion to date in this ON/OFF controls type gations suggested control strategy feedback desired tion which to predict rate utilizes neighbors. a model service at using of the system. inaccuracies traffic of one or more a decentralized expensive several where The to implement traffic statistics with are used in the of this switches neighbors traffic. The mechanism also monitors schemes scheme the nearest network streams. traffic statistics cooperation kind in- were feasible, the feedback only is used by a switch more and complicated provided for schemes only marginal in performance. rectly. using equa- Neither The each is necessary mechanism bursts the of traffic can obviously concept form to work the information sending or a group of traffic aggregation rate to be most be extended group, to groups cor- carried identifier a single the of con- establishment for the mechanism such as a connection of the aggregate adapts or a group nor explicit uses only We consider switch connection rate reservation tifier. cept rates. at a single control short and implemented rates of either each packet, feedback in the model controller of a connection switches. Old The volving nections. circuits a control switch While from traffic exchanges mechanism. The for adjusted, Each packet Each switch on a control control sending a rate-based of individual based here, traffic. or investi- by the neighboring We have are inappropriate rates is arrived is used to correct future preliminary are dynamically provided service information schemes that windows we have developed service at a switch information The these Hence, in which or aggregates strategies Our [1, 2, 16]. that high speed networks. control section. its nearest regulating are node by node sliding work. directions. proposed rates for individual and periodically were hopby-hop of Controller strategy are computed improvements The parameters and future role in regulating per stream received here arguments. been proposed we the related of results control sending we chose users are protected results the of the play an active described delay round-trip we believe, hopby-hop switches advan- is the shorter of traffic. 4, we discuss a summary Description In the into control. An additional IIag time, 2 a hopby-hop to the end-to-end The reduced control the periphery to feedback-based choice. for to section, of traf- streams at a switch, values scheme In this trans- bursts traffic mechanisms cycle compared more responsive from group with to choose In Section with proposed scheme. ex- the end-to-end short bursty occurs is a natural time in a network. data conclude delay- of the end-to-end how the scheme. propaga- one would to or less than several aggregation tage of hopby-hop gets larger, To control aggregate traffic a large the performance another also examine have certain propagation of users to transmit comparable a network. fic one could with As the of a networh number duration through Since product. product a larger mechanisms in networks delay-bandwidth mission control and in idenand useful when aJthough the con- formed on other considerations.2 To evaluate first namic that the performance compare sliding window of TCP. The with do not present simulation the results also the performance study further control results control by-hop scheme resources packets, as compared the that We of the proposed an end-to-end The results to traffic at the bottleneck to the end-to-end is organized proposed control feeding Due to lack of space, we with that changes. better monitoring an with the hop As a result paths spreads not passing through aggregated connections. Section 113 grOUP and we prolonged of connections the term not congested “connection” explicitly mention ba- because con- in the }eed- to also get congested. the initial shall unstable mechanism, switch is simrequires on a per connection causes all switches due to the blocking 2 In the rest of the paper, In Section In switch switches to stop forward- such as ours that is potentially regulation at a particular [2, 16] do not A scheme of this kind of traffic in tree of the congested gestion are forced a scheme such a scheme in In these schemes, link to the link. than a “source-blind” gestion proposed connections. to a congested and controlling sis. However, scheme. strategy. between traffic pler to implement and loses fewer as follows. approaches ing any more traffic hopby-hop scheme using show hopby-hop distinguish the perfor- of our scheme is much better. faster rest of the paper 2, we describe show The namely but it with reacts mechanism, we dy- DECbit strategy. it utilizes The better. scheme, of an existing of a comparison the effectiveness by comparing identical that based flow control mance of our scheme is much there of the proposed its performance Con- traversing point. includes a group the of In the control formation mechanism consists stream. The culated from number of waiting ing service buffer the period and collects going upstream assume that here the feedback occupancy of a connection occupancies for by noting switch measures per out- periodically traffic via the source also has the ability to the feedback provided link. to adjust For a given the control We switch. primary ing rate gord of the controller of a connection downstream tempts switch. hop. The adjustment rates, waiting at each user-specified this tune delay hop, The parameter. sider switch. Let The rate when packets is computed stream At time switch. level, switch, Let tz+~ step sired buffer setpoint the service the connection rate and k time be the estimated and fij switch. occupancy step z is computed level. The factor, feedback messages damps Assume rate at a fixed T, such the number conx. only rate about of at the knowledge estimated first of service order auto- a, cent rolling are tracked. service rate how In this calculated the buffer from occupancies. after of the initial rate a few steps. This side-effect used; increasing this one. at its available start for of minimizing this a connection the total because while its existing connections to the new equilibrium that With bandwidth happens rate, (1) the sending indicates is less than gmduai and to Equation is to increaze sending a con- in a slow as the feedback packets as above When increases modification as long starts designed in the design. rate oscillations of queued gradually rates the controller A simple of buffers their with the sending fashion. (3) O<cr<l improvement up, has the important number a connection is decrease value. the de- respectively, rate We define rate starts change a connection at the service sending a minor that number of convergence results at a The modified design equation is for a gain X*—’ to the = 1. at time k* ‘“+k + T* l//3 I/k * Xj~j]_*k-l rhj, otherwise(4) as ways. connection value on the r.h.s is zero when is equal to the expected sending rate is chosen Otherwise, wit h the correction is the used we use the estimated The following previous oscillatory (1) rate. with actual One could term (1). in the service ent The first Equation m~ -1 suggested the buffer x* denote denote, at the downstream the rate accurate the changes time let steps earlier z + k. Let ~j Since in general, rates is not available, The characteristics discipline the Experimental z, at the down- receives incurred. traffic of as the frequency overhead on the equation, nection under consideration, estimated controls z. interval at time the /3, that buffer and let the affects service rn~, in Con- are received time under at time service filter thus the value to hereafter to the in the discrete for the link buffers, measured from at time time as the is revised as follows. step Z, the sender Zz_k, buffers, state delay For the connection of occupied desired 3.3 how switch. period as the riz~, is calculated closely is a against information are sent with be the number parameter, set point, and referred +(l-a)*tij-l, rate k ~ T is the propagation acturd ca- control depend as well rate, regressive of packets of a connection at a future of k determines updates ‘iJ=a*m~-l the sending that downstream any additional packets k feedback occupied a connection number feedback that occupancy rate, in Section feedback of the system l= denote sideration. at each the throughput rate receives at instants sender. packets to as a buffer We examine sending the evolution space, service losses for a connection. a switch downstream when the future at the also at- allows desired to trade-off maximum whenever rate The referred parameter or packet packets sources two The mj, downstream the send- observed period. of rate procedure of waiting its sending available. rate adaptation of waiting increase becomes rate a reservoir reservoir to quickly pacity to the service The to maintain is to match level Zz+k is estimated the choice between service input The link, T, the period rates in by the adjacent occupancy setpoint. (2) to each that buffer the buffer dur- A switch sending the equal in the of packets for each connection is provided instant is cal- the change measurements. occupancies so as to have in- each traffic at a switch and the net inflow information nearest response rate between This buffer packets buffer link. proposed of the occupancy to match the sending the buffer level. In this the estimated rate is increased future setpoint case, the service during may number then If the buffers preferable. control be done of shared the same output buffer are shared available Let for a setpoint among of the buffer and C denote of differ- are reserved a static as follows. buffers link in a number of buffers its lifetime is acceptable. This setpoint number an adaptive sharing 114 a buffer tive connections, total or decreased choose If a fixed all ac- setpoint B denote is the to all connections the output link’s rate. Then, buffer for a connection setpoint whose current rate n1 is Ca, the is H2 z“ = * B (C”/C) (5) ms 2MWS The behavior with of the controller five different gain, ~, the rise rate, setpoint, pact x*. The tuning on the performance expense on the design for next-generation about parameters is discussed propriate of a traffic a, the and their in Section control im- 3.3. factor networks. the implementation factor, T, and the buffer was also a critical liberations tails period, of these FI=-6=EI here is controlled the correction 6, the control implementation discuss described parameters: The Connl in our de- mechanism While ap- we omit Figure de- here due to lack of space, we in [12] how our scheme is efficiently Conn2 .i”k 1: Simulation ..ili..ea-ai implemented. BMAX I I I configuration 1 .100 I I I I I 0.0 — Mm’! (M 1 0.8 — 3 Results In this section propose, trol 0.7 — we compare referred scheme the HBH the lhop-by-hop to hereafter implemented scheme with as HBH, in TCP with [6]. the end-to-end scheme that the flow Next, o.e — we con- 0,6 — we compare 04 . scheme using the feed0.3 — back controls E2E. The based on Equations latter comparison the effectiveness controls. provides of hop-by-hop Finally, the parameters HBH to hereafter better controls we discuss of the 2-5, referred various insights as O.* about 0.1 vis a vis end-to-end trade-offs scheme, The in choosing results 00 a reported Tm19 (* here were obtained a network through simulator performance simulator that studies discrete-event The directly from TCP other Figure Performance The trol dynamic the 4.3-Tahoe in TCP, due Unix BSD size. The in low-speed Host verses 1 switch other have a propagation longer delay 1 Gbit/s), much path In the TCP the Figure 1 shows the bottleneck packet to show stable flow control loss is the only event that link flow controlled for this link’s All traffic of higher tra- capacity three with strongly on the amount dependent TCP and HBH the bottleneck. The at the bottleneck, The setpoint amount average end-to-end Figures 2 and 3 we show end-to-end delay HBH. link The taking Figure trolled leads to a reduction maximum the bottleneck utilization running is measured The throughput similar to that 1, is 100 to be half so that to that is measured delay packet. available, the of TCP. In and the under over at available is chosen 3 is comparable a connection utilization of buffers of HBH is available in Configuration of buffering delay for of buffering number overflow of TCP TCP and 4 ms intervals for each successfully curve of the link which we omit utilization curve. speed (say connection the detection buffer losses is buffer the performance to a connection, here has a profile a for packet As a result transmitted is one-way, behavior. scheme, reason and the end-to-end links Using primary of the maximum of the net- traffic, size is 500 bytes. and a bottleneck in a TCP The utilization at the bottleneck. packets. The connection are 200 Mbps. of 10 ms. con- effectiveness to as the direct The links delay 3 and results longer 3, referred and 2 links. The Host networks. for flow [6], was chosen and used for the comparison. 1 to Host is 10 Mbps. towards popularity TCP with mechanism by Jacobson to the work configuration from adjustment suggested comparison the scheme Comparison window 2: Bottleneck code used in the (Tahoe) release. 3.1 “e.> using has also been used in several [17, 18, 19]. was taken simulations 3The of a packet in the window 115 2 shows that the bottleneck end-to-end when link delay the point at which at which the receiver is fully host receives connection utilized we measure the source host the is the transmits the packet. is HBH within time con- 4-5 roundduration a packet from to the point C.01.v (in .-) BMAX improves . lW am but to-end em somewhat the large once a connection start delay up time are not reaches steady and the oscillations state in the end- reduced. S.SO In [12], we explain SW connection running the poor under behavior, TCP. The observed here, of a performance observed 4.s0 here is worse than 4.03 The S.XJ figuration difference, 3M seen in many we believe, with vironment, that a large than bandwidth-delay that studies is due to our having the performance is far worse recent product. of the TCP [17, 19]. used a conIn this en- flow control scheme in low speed networks. 2.s0 aal nllm Figure 3.2 Is The superior Hop 3: End to end delays with TCP and HBH mostly times. In contrast, 100 round-trip over, in this phase, 190 packets whereas After tleneck remains with link HBH; only for the TCP the control almost the bottleneck capacity and in packet trast, under when traffic waiting HBH served. occur performed the the number the bandwidth-delay the TCP scheme mance of the connection. amplitude with deteriorates HBH version of buffers product, However, compared is not varying at to the oscillations embedded is used then If the uses packet The question this control by the comparison be justifiably to that in end- by hopby-hop addressed address argued that of control question, the pol- we compare of the end to end scheme, 2-5 are used based loss as to the oscillations is due to the choice Equations do not to adjust on the feedback the sender’s received from the experiments is node. switches be high even number of for sevob- network tion, the end to end propagation direct traffic, data to send. cross-traffic, of a HBH window Unix link at 3000 ms. and stabilizes The values ,L3, are 0.4 and TCP, ad- (Reno) utilization ran experiments traffic We study with source, with rates direct to voice/video but traffic. be subject of con- to changes rise rate, a, control The traffic using source identical uncontrolled which to a different round and by gain, T, is 4 time. We a constant and infinite controls share trip the trip cross may in the 8, is picked period, provided traffic infinite the behavior factor, The The 2000 ms per end-to-end cross-traffic controlled on the The O with in one hop round a Poisson traffic on at time reacts correction 0.45, respectively. sponds capacity 116 of the mul- for the direct up at time rate among In this configura- 30 ms, respectively. switches again. to five updates rate source delay 1, comes cross-traffic of the bottleneck ms leading smaller with modified in the 4.3 BSD In there controlled are of much observed time. perfor. the interactions 1 as it comes up, stablizes, to be 1/4 patterns, The of hops, by adding to be studied. is 60 ms and connection and disappears nection of significantly. controllers and the cross traffic these the number S1, S2 and S3, allows the performance the used in 4. Increasing hopby-hop is but configuration tiple bottleneck the bottleneck may of HBH shown in Figure In con- the behavior at the affected it to dkectly at the source The of go down comparison cross-traffic these oscillations cross-traffic. algorithm of TCP, will considerably, is not is TCP switches loss of throughput. in performance bottleneck of length come up. summarize since where TCP Since policy. algorithm leading to compared scheme shows of the fixed in the end to end delay or without justment scheme of randomly are oscillations of the nature connections HBH-TCP We briefly less than the presence because TCP where rate occur HBH are provided control In order E2E, 3, shows the in queue utilization the icy. at the bottleneck. scenarios. When existing advantages the performance under-utilization new connections the bottleneck changes We have oscillations when occa- is not the oscillations the HBH in significant losses when packets eral other The can result with end-to-end in Figure because In contrast, drops TCP what difference thus and possible over end-to-end the bot- of of the control the control feedback delay about loses performance feedback, to-end controlled The These of cross-traffic, no oscillations. the bottleneck with shown of TCP. mechanism. state comparison. connection, even in the absence by HBH stable the utilization start-up up in this characteristic Better? for use in an environment an implicit More- loses about by a connection flow control shown oscillations controlled utilized slow and erratic problem delay fully utilized. connection reaches provide it takes about is fully controlled a connection TCP scheme, the link a connection with The the TCP before a TCP no packets. sionally. with times Hop due to the nature is designed trip by 0“ 000.) data as that traffic corre- the service set of controls. .Ms. x- / I 1 I I I I L -G--ouLll ‘M’ ‘ s 0Tf12 M n I HI ‘w MS “ R2WMSP’%4 MIS 2(OWS M S5 lhns Ml’ Kills mo.od- lhsls Cmnl Figure 4: Simulation configuration 2 ,* Figure 600.00 . 500.00 . tin ..) 6: Sending rate at the source scale of this figure is much smaller than the time scale used in the figures showing the performance 400.00 300.00 . because both the E2E more quickly than TCP connection scheme This start scheme. The start-up configuration would is so up much time of a — 200.00 . 100.00 . :’ J ‘: ,,r.. . ...... At 0 m- , instant are used adjustiug delay 5: Buffer ditional occupancy the performance value chosen there are buffered the E2E node. scheme for Connection 1 buffer a connection the 50 and the E2E with of buffers a buffer state, experiments, these than the pipe when the entire HBH buffers with at the bottleneck in its path is the a buffer with setpoint buffers The of as the E2E values link capacity scheme 5, we show at the bottleneck switch 5, for connection a constant of the evolution the output point, rate source to 40% the 1. The smaller 1 link utilization and it that bottleneck from comes up at 2000 capacity. delay rate. in Due to fewer ad- Note switch ms with that the shape 4 to ing HBH, a rate almost time 117 The shown we show by to zero. scheme Yo as In this when by the buffer sending the source sending as quickly time On the other show curves occupancy rate closely curves. at the source; rate stabilizes as it does using E2E. The does occupancy happens with the cross-traffic delay the link delay. buffer both in adjust- occupancy that scheme compared depends delay we do not experiment and the HBH more capacity. capacity as the take in Figure is slower as the buffer long of avail- to react and the end-to-end end-to-end the actual is able as the For brevity, 100 As shown 200 buffers. additional rises as quickly at becomes to quickly the additional as well of 50 at the instant appeared. occupancy is generated of the rate. capacity 50 buffers in using The number of the value a connection with with setpoint remains does not fall more 200 buffers the throughput for the E2E of buffer cross-traffic buffer with scheme utilization utilization, is used. link additional much scheme. bandwidth. the E2E scheme than ing the sender’s with for connection of the available the HBH point In Figure service cycle time, allow rapidly on the used in these are much is 1500 packets to the is independent When packets scheme The HBH required chosen. to the E2E scheme of 200 uses the same number setpoint bottleneck setpoint of 200. For the configuration buffer up due to the reduced 5, the HBH hand, 4 switches queue size, which in in the a setpoint only rate comes bottleneck are needed for the HBH setpoint advantage scheme, of packets with scheme of 200. on the the number scheme at the bottleneck setpoint In the are queued HBH a buffer setpoint. depends at eachl hop for any connection; packets for both of tlhe scheme buffer packets in steady for scheme the Hence, network same for cross-traffic in the feedback able, the buffered Clearly, be very long. 4000 of additional Figure when up at the the input the smaller I 3om the buffers w I 2000 O. -) I lCOO this I’m IT 4 ,.(—” ............. /.,,,,.O) y a 0.90 with L.- .-:-..:.-..-.l\ equal and the HBH the TCP of TCP. only a buffer set has just dis- follows the In Figure clearly, 6 us- to the new value ,mak.t. hdlwldu.1 md ‘@’4- 120 told buffw .cufmrml” ml th. I I I I bWut8d4 I I I XcO i I 4UX! ‘4 I 1 1 mm mm Omold I I Imtl I 0 I m The Figure 7: Buffer occupancies with Figure E2E(200) Pm=ik.t . I 9: Buffer Mtvld.1 - with m Ill. —pmo!n I I tns) occupancies tow .rld [h I HBH(50) bowuld I I I .wm I I 4000 I -1I 1s0. Km. t I xmOl- .lJ I 1 Figure 8: Buffer If the cross traffic a Poisson cross traffic Tbn. different, because of the randomness with generates stream it at ively although then with In steady-state, buffers, tive occupied at a switch, connections transient Figures traffic the scheme. and behavior 7, 8 and is controlled end-to-end 4 we controls, here ampli- the total number on the number of namic bufTer setpoint algorithm in the next set of experiments. suggested remain traffic occurs static. in Section active in use in in with HBH(dyn) rate of the original Section and is much figures The 2, is the to decide how E2E connection dy- pens because 2 is explored are different 118 as the number of occupied and shared buffer adjusted respectively. scheme occupancies levels It is more than the end to end delays and the second (cross) when for the HBH is clear E2E Figures evolves successful for both the all the described value. occupancy from connection these at keeping connections scheme. of the two at a available among scheme, setpoint bottleneck of con- buffers, of buffers adaptive are dynamically scheme, the desired the the the HBH the sum of the buffer around to be limited we use the setpoints that then number Since the number is likely 11 show the buffer because is static the total connections, 10 and is. With setpoint also increases. at a switch the the rise rate of the cross connection setpoints the decrease increases, switch, the cross- of the cross connection buffer nections of ac- We study of buffers overshoot than occupancies a big overshoot. If the buffer scheme than setpoints.4 number causing (WI m.) fluctuates The the HBH 10: Buffer much higher to are not qual- set of experiments, a large delay that according occupancy the same way the direct and as a result ~sme buffer In this the feedback-control smaller their with depends for the total 9. packets in the cross traffic. is smaller Figure HBH(200) the results the buffer tude of the fluctuations the E2E moo ICrJO m,) occupancies stream I I o mmm(1” This at or hap- connections incremes its ..-* . me total him” bldludu.a -pmwln ●t U19 b0w9t0011 Throughput 9.36 pk/ms 9.19 pk/ms 10.57 pk/ms 10.26 pk/ms 10.57 pk/ms 10.57 pk/ms schemes y - . - . - . - . - . lW.W . HBH(50, 100) E2E(50, 100) HBH(200, 400) E2E(200, 400) HBH(500, 1000) \ \ \ Wld_ow & -“1. / E2E(500, Entries I Imo ‘Imm Figure rate faster 11: Buffer than Lossl I 3000 1 mm (In ml.) with connection is the percentage the total large number increases connections tion in buffer come if buffers study of buffers up, E2E(dyn) decreases the effect controlled we did not set any limits available to a connection. occupancy that occur when packet In the next set of experiments, by HBH and bursty E2E on seconds, where of ‘i?oo ms. amount ment delivers a burst of buffering that traffic new available. is 12000 milliseconds buffer of 500 packets an important HBH is half nodes sizes. at 1000 milliseconds and disappearing and loss values than coming results of available mance HBH of buffering E2E throughput. increased parable, sample. the with E2E still available and E2E worsens with in then than is that the packet nodes nodes the number if 200 buffers the loss rate E2E If the nodes of packet are available at at non-bottleneck to almost have loss because back quickly. at non-bottleneck drops and higher Another using O. Note the same rate throughput worth point that al- of rise the a lower to note setpoint for lower of buffers is that value setpoints accumulated for results both at HBH in fewer losses throughput. Sensitivity Analysis Thus when a max- HBH and amount a small terms HBH possible of design pa- both performing of the controller such buffer affect haa al- as the the control how changes The is affected setpoint, period the performance 3.3.1 lower of buffering to 100 the of behavior the by several gain, and the rise rate. in the values the param- correction In this of these section, parameters of the controller. Buffer Setpoint throughput slightly loss rate. The eters we discuss fewer buffers is The difference the value setpoint. performance steady-state slightly However, and better. portant 119 point may be made are long based performance if connections to have equal 5, illustrates If connections setpoint loss set in Figure As the num- packet in the performance of the as shown of H BH and E2E is com- is decreased HBH, other and setpoint throttle For example HBH a lower factor, control For the same amount maximum having switch. and loss rates for dif- to a connection HBH(5O,1OO) node. 3.3 perfor- requires loss rate to 1000 the performance the hop-by-hop per connection, haa a small of throughput, and ccmtrol cross traffic ber of buffers both control is available When at switches at non-bottleneck it has a setpoint and E2E up as the amount increases loss and the maximum for the particular (in available over the entire that the same level of performance. no packet mainly has a slightly every the same rise rate is 1 indicate hopby-hop of 400 buffers of buffers at 11000 milliseconds. connection end-to-end however, to produce imum per format: setpoint rameters in Table buffering of both improves; most shown the at the bottleneck to notice at the bottleneck both used for both HBH and E2E. The thing nodes with because of the experi- a,re averaged As in previous experiments, interval. buffer Q a mean but The throughput in buffering the bottleneck available non-bottleneck of the total the cross traffic occurs losses decreases. we every with ‘The duration with o% are lost values of maximum is more on a connec- distributed the setpoint of packets 1: The average throughput buffering losses are likely for different Q is exponentially We assume .043% is the lost ferent though The cross connection o% o% number Table the nodes before The that of uncontrolled o% its rate. suggest are limited. setpt of packets LOSS2 is the percentage the bottleneck. I 4400 with experiments column where o% ,031?70 , However In the previous first bmaz) LOSS2 10.91% /- occupancies the first the packets) and brrum is the total to a connection (in packet8). 4’ I 0 in scheme(setpt, +=---- f -.000 am .- 1000) Lossl 2.13% 13.48% o% 4.29% to criterion that so lived scheme and the impact on the are short a set point of the HBH then 200 of the the optimization such lived is high aa power or bursty, enough with packets, buffer choice of of some [13, 14]. it is imso that no throughput enough is lost so that decreases. The the product if excess capacity buffers maximum and the feedback-control throughput any possibility of overflow to a connection ilarly plus occupancy in bottleneck The start-up taken time time occupancy times taken The taken rate from dependson around the gain time; to push 1 packet schemes sliding control have messages either message through is received. to the significantly at the value; The The gain a hop that value. state value determines how the sending sending rate is used. With rate rises and falls the higher faster gain used takes longer to stabilize undershoot and the overshoot pronounced. important In our to make closely when value, around sure that when the value of the setpoint there are large experiments the controller in two round gestion value value attempts trip times. more harmful of gain should high, rate. goal is to consider then decreasing control congestion gestion that control mechanism useful Correction factor and Congestion Control Period Both, the correction period T affect dependence Let the high parameter If the duration 1 /k sees then than one; of the of these parameters frequency have a frequency just a and the duration responsiveness the tual factor transients of k Hz. values in the Let us assume such that otherwise value of the feedback the value interval that should be one. points uses a ON-OFF nodes [16]. oscillations high-speed As to handle controls Because in our which are con- in the of these does not queue factors, appear the to be network. to the network. This and their can be predicted. It would mu- by subjecting and by policing that demands mechanism work occupancy behavior. well sources philosophy The make or without an a-priori traffic to at the is accept- can be accurately does not with require all demands offered proposed modeled hop-by- such an assumption. call admission characterization controls of source to ad- are ignored. Most is less than data to be less Ideally, be long enough The to users, spread used in Autonet controls and does not be chosen traffic. designed type service may be prevented as follows. we want interval should should The buffer these transients of the control/feedback the value of alpha controller. at a switch. We have observed cause such phase. between able if one believes hop control of the control is explained require con- rates. the would a higher admission The and for a wide-area, call admission 3.3.3 nodes and not the participation is not ON-OFF at the bottleneck. differs rates aggregated area network in unfair to resumes one can cent rol flows without scheme that result to other occupancy or at a switch. studies control establishment local this the flow be available mechanism in [16], “source-blind” to the setpoint loss of throughput, a high-speed preliminary espe- of our implies in the connection off in response each connection, we propose correctly out on the is based on the we have proposed of packets, circuit long-term it is otherwise For most the system be used while is more that numbers controls of flow hop-by-hop is based We control about an individual type traffic or when an ON message hop-by-hop proposed pointed two node; that of either Autonet, and the found is fairly If the policy than value used is 0.45 which to drive value is turned information in the scheme of a switch of buffers is not too high, in the sending the gain gain the setpoint the gain at The a higher we have cially rate node. the number at the setpoint experiments, swings the sending at the upstream flow schemes. work on the data [2, 16]. In the cent rol mechanism A window-based accurate past, [1], and the other scheme these of work for one of these mechanism HBH from as the sequence Gain tracks to has been done so far on a congested a timeout The windows. this from body In the been suggested: in [2], the traffic In contrast, 3.3.2 to respond schemes work schemes. use of ON/OFF on the the occupancy control mechanism an OFF is equal enough and growing window phase is the the is a large 11, 13, 14], very little hop-by-hop proposed to the set point and the setpoint there is decided to exceed phase small work of flow/congestion [3, 6,8, if lag. this depends of the second by the controller bottleneck of the first at the source Although the increase the feedback-control but in the cross-traffic. Related design Sim- to zero only is less than rate for the first duration available overshoot. drops Theduration 4 to the value to get toequilibrium by the sending source rise rate. rate phases. bottleneck possible at any point service bytwodlstinct time queue be equal transients changes by thus to eliminate of buffering should the maximum the bottleneck the queue the amount is limited undesirable significant capacity at the bottleneck lag for the system; at any instant out up, and yet low if available size of the overshoot of a connection’s of the setpoint opens do not overflow These the of the flow networks systems of buffer to filter base their occupancy considerations 120 control schemes are feedback-based of the or control traffic predicted proposed flow decisions conditions future for wide control area mechanisms. on the old values and state exclude any of the system. Theoretical results are inevitable controls reported for a fairly due to the delay performance in [3] prove general form strategies, called in the field control model-based In a model-based predictive of process predictive model of the controlled output so as to satisfy serious myths have been about uses a the value function. of the trol accurate, scheme the performance is better than that #1 scheme. This is because decisions feedback are made HBH system. system state and not In a model-based back messages predictive or inaccuracies in the model the model the real state from The controller the new rate within con- As shown adjusts of the system, reported in [9, 10]. The was a split-range time step with time. only ● errors the drift more based fluid Myth #2 waspredicted The taken service occur An end-to-end the predictive effort bottleneck packet-pair rates probe icy is to drive with trip mechanism. value. estimated using The the scheme, modell. rate, this is a Kadman filter point e.g. and a fuzzy ex- ety In this paper we have nism for controlling Simulation namical results behavior based on sliding requires buffer The cost pected congestion of the particularly to provide to support in a packet the scheme than of congestion that windows, such as that switch for switched displays network. better control of TCP. generate schemes rates per traffic stream. information required packet sophisticated scheduling applications. scheme and 121 in [3] show voiced against about We rate-based the traffic inherent have found to support here, a node became congested, and no catastrophic controls fear of instability none unstable localized the a variIn any When past these over conditions. we observed no we have remained with tional strategies. is no cross traffic. described have the contraffic may stem non-discriminatory and it does not on- seem relevant to the we propose. #4 HBH is wstly. cost of the HBH in switches for a variety the trend future cost. already of different towards packet cessors, the processing has form reactive of experiments This provide in number occurred. service and by the actual evidence experience more, by the fears simulator conditions Myth in [3]. We hypothis determined to disprove. experiments, Undamped unstable. These the a large particularly are ex- mechanisms The general. The additional is reason- that switches ● The scheme service is inherently has also been from scheme dy- such No such as the scheme or experimental off type about future real-time mecha- information and current additional Work a new hop-by-hop show that each packet occupancy able, order that proposed Future that damped. even if there of configurations gestion the oscillations versus fear Using of these anid schemes are hard conducted cycle predictive all the in the end-to-end of oscillations HBH in for that quickly proposed #3 beliefs. is with we have examined. strategy theoretical to a occur that control instability pol- occupancy are oscillations controls a predictor. Conclusions and oscillations. we find and reactive Myth wild in the feedback are used filters of the controller reported behavior. 5 scheme are used. in the case of the TCP. A similar by at the bottleneck Two ● switches, scheme at a future occupancy trip scheme the TCP also been of controls, strategy from goal of the control occupancy fluid the service ponential The queue control nature that displays the extent lag time amodel was predicted feedback in learning the responsiveness have esize that Purely estimated the buffer particular predicting are than that is observed congestion was one per round rate model-based of using to changes values small oscillations equation. in [8]. Instead 3.2, the source of end to end round we have found in the buffer continuous investigated the 2 and 3, the HBH cent rol algorithms are damping controller of control of thequeue average in Section in a couple rapidly HBH Oscillations of the experience in the earlier horizon being approximation. the moving from strategy The discussed results by the exact congestion the predictive onedecision Theevolution using control cent rol. (HBH). in Figures In general, the parameter of the system. with popular feed- for prediction so as to minimize obtained received SIOW~y. scheme HBH not following based on a pre- control used here has evolved reacts have to the controls proposed In the HBH of one of the authors currently control on the old state are used to compensate due the results is determined future schemes past As is clear from does. dicted switch If the model of a predictive c)f a purely hop-by-hop in the hop-by-hop times. is reasonably packet by us. that attention * Myth a controller to compute some objective control in a multi-gigabit developed We believe the [15], strategy, system special controls, control being based To improve systems, been implemented oscillations of feedback in the feedback. of time-delayed developed that switches, classes. quality these to easily handle of Further- microprocessors using In our implementation, is marginal, supporting traffic mightier power mechanism will micropro- the addi- we have measured that a revision in a rate instructions. This are changed ted, The scheme after fewer any clocking functions control could rather than in applications suppress bursts controls, better environment large than quires packets traffic fewer limited within of packets waiting service lag time, packet storage environment, not use H BH cent rols. back. [4] D. is likely some sub-networks In such environments is the HBH links, controlled with the end-to-end HBH the typical layer. sub-networks the virtual control oscillatory at the If none bottleneck, 3 will impact then May Journal first networks. internet- and A. Weiss. with 92, Aug. TechnicaJ Science Analy- Delayed Feed- 1992. in packet- Report TR-89- Institute, Berke- 1989. may be viewed on Selected on the the In the future, avoidance IEEE [8] S. Keshav. within A control In Proceedings unaf- [9] Keng-Tai Ko, 2 and pathi. Interaction among dictive Congestion Control. behavior observed within Moreover, not have the the an HBH use adverse controlled [10] strategies, to experiment with both and simpler different more hop- sophisti- P. Mishra, networks. Rate In R-o- CA, Dec. and ISDN Systems. P. Mishra, networks. In ternational Workshop Proc. Palo Alto, CA, control Second Nov. in and Satish IFIP 1990. K. using in high-speed on Protocols 1991. Satish Circuits To appear Ko, Partho congestion to flow control. ’91, Sept and Virtual Networks works, 122 Partho Keng-Tai Predictive we expect S. Keshav. approach SIGCOMM is the link. In 1988. ‘9o, San Diego, theoretic of ACM sub-networks will and GLOBECOM seen in Figures control 9:1052- control. ’88, Aug Kanakia, behavior bottleneck time IEEE 1990. 2 and 3 will remain H. controlled flow and for very high-speed HBH at the Real constraints. in Communications, of A CM SIGCOMM servers ceedings then sub-network; will Areas Congestion Kalmanek, controlled is used and a par- behavior and G. Pacifici. of service 1991. Jacobson. [7] C. R. If, say, is the bottleneck in the quality of the link seen in Figures switch A. A. Lazar, the dynamic end-to-end control Pren- 89, pages communication Computer with 1063, Sept. [6] V. sub-network. by-hop real- Networks. SIGCOMM SIGCOMM wide-area ley, CA, may mechanism. of TCP sub-network behavior of the be observed of TCP’S both In this en- capacity control mechanism the rest of the sub-network fected. service by the HBH controlled be observed ACM Strategy Real-time 022, International an end-to-end to be used at the transport determined ticular Control ACM Ferrari. switching suffers Data M. A. Rodrigues, In Proc. Proceedings being of 1). In a heterogeneous as single in controlling A high-performance In Proc. sis of a Rate-Based scheduling vironment the influence the performance Sep. 1989. [5] J. M. Hyman, control HBH such that sources. Sirpent: [3] K. W. Fendick, cross- capacity under and R. Gallagher. approach. 158-169, lag also re- HBH The 1987. working a cou- when Hall, [2] D. R. Cheriton. at the bottleneck capacity Due to the reduced the total For when new HBH to provide scheme to study scheme traffic derange mechanism, sources. operating control data D. Bertsekas tice follows Due to the reduced 5, 7, 8 and 9). increased losses when (Table 6). [1] data scheme. at the bottleneck, (Figures number decreases. end-to-end rates of a sender rate at the bottleneck (Figure accumulate to use the packet here that as well aa short additional in a narrow the HRR We plan traf- References employ We have shown of data the sending is added all switches integrated traffic with policies. and bursty with was proposed are classes of traffic of this affect- end-to- the corresponding delays a smaller in order without traffic Round-Robh has been integrated of in integrated or rates real-time controls. the performance scheduling to mostly predictive traffic real-time scheme control to experiment to study data of jitter of service separate one We plan In [7], a rate-based Hierarchical there method to use additional layer. amounts the changes in the service cross-traffic where is no reason connection, fewer traffic where quality time at the transport ple of round-trip time, Moreover, significance, to control such as control called of data error-correction acknowledgement there can handle a long lived packet. used is also carried. in Consequently, a block has real-time mands one. of the model-baaed we also propose when fic environments at control. end controls HBH data forward the data In a homogeneous HBH each for HBH do not serve to the role played be sent for where is used or where ing flow in contrast the proposed the framework In the future the HBH are required these packets than within are transmit- since with schemes such as TCP. acknowledgements packets of packets packets because cated 170 machine cost since the rates mitigated acknowledgement layer end-to-end executing few hundreds cost is further the transport could entails is an acceptable Computer K. Tripathi. wide WG6.1/WG6.4 for High TriPre- area In- Speed Net- [11] [12] A. Mankin. Random ceedings of ACM P. Mishra and congestion drop H. hop by hop A scheme. AT&T In Pro- control. ’90, pages 1-7, Sept 1990. Kanakia. control 920225-OITM, congestion SIGCOMJ4 Technical Bell Labs, rate Report Murray based 11273- Hill, NJ, Mar 1992. [13] D. Mitra and J. B. Seery. for high In hceedings Sept [14] speed data K. K. Ramakrishnan with for congestion SIGCOMM D. Seborg, namics ’90, pages avoidance links. et 20-30, S. Shenker, of Pmces9 and Dy- self- point-to-point Areas in Communi- 1991, pp. 1318-1335. D. Clark. of a congestion Communication High-speed, using on Selected L. Zhang, Computer A network 9, No. 8, October on the dynamics networks In Proceedings 1989. Autonet: area feedback in computer layer. and D). Mellichamp. Journal Vol. A binary 1988. Wiley, al. local IEEE cations, tions ’88, Aug T. Edgar, M. Schroeder R. Jain. network and Control. configuring [17] windows and simulations. SIGCOMM and a connectionless ACM [16] of ACM adaptive Theory 1990. scheme [15] Dynamic networks: review, Some control pages observa- algorithm. 30-39, Oct. 1991. [18] L. Zhang. Virtual for packet SIGCOMM [19] L. Zhang, the COMM A new traffic network. ‘9o, pages 8-18, S. Shenker, dynamics effects clock: switching and Sept ‘9I, pages 30-39, algorithm Observations control In Proceedings Sept of ACM 1990. ID. Clark. of a congestion of two way traffic. control In Proceedings algorithm: on The of A CM SIG- 1991. 123