research-article

Sketching the delay: tracking temporally uncorrelated flow-level latencies

Authors:

Josep Sanjuàs-Cuxart,

Pere Barlet-Ros,

Ramana Rao KompellaAuthors Info & Claims

IMC '11: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference

Pages 483 - 498

https://doi.org/10.1145/2068816.2068861

Published: 02 November 2011 Publication History

Abstract

Packet delay is a crucial performance metric for real-time, network-based applications. Obtaining per-flow delay measurements is particularly important to network operators, but is computationally challenging in high-speed links. Recently, passive delay measurement techniques have been proposed that outperform traditional active probing in terms of accuracy and network overhead. However, such techniques rely on the empirical observation that packet delays across different flows are temporally correlated, an assumption that is not met in presence of traffic prioritization, load balancing policies, or due to intricacies of the switch fabric.

We present a novel data structure called Lossy Difference Sketch (LDS) that provides per-flow delay measurements without relying on any specific delay model. LDS obtains a notable accuracy improvement compared to the state of the art with a small memory footprint and network overhead. The data structure can be sized according to target accuracy requirements or to fit a low memory budget.

We deploy an actual implementation of LDS in an operational research and education network and show that it obtains higher accuracy than temporal correlation-based techniques without exploiting any knowledge about the underlying delay model.

References

[1]

Corvil. http://www.corvil.com/.

[2]

Juniper Networks T series Core Routers Architecture Overview. www.juniper.net/us/en/local/pdf/whitepapers/2000302-en.pdf.

[3]

M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. Data center TCP (DCTCP). In Proc. of ACM SIGCOMM, 2010.

Digital Library

[4]

P. Barlet-Ros, G. Iannaccone, J. Sanjuàs-Cuxart, D. Amores-López, and J. Solé-Pareta. Load shedding in network monitoring applications. In Proc. of USENIX Annual Technical Conf., 2007.

Digital Library

[5]

J. Bolot. Characterizing end-to-end packet delay and loss in the internet. Journal of High Speed Networks, 2(3):289--298, 1993.

[6]

B. Choi, S. Moon, R. Cruz, Z. Zhang, and C. Diot. Practical delay monitoring for ISPs. In Proc. of ACM CoNEXT, 2005.

Digital Library

[7]

Cisco. NetFlow. http://www.cisco.com/web/go/netflow.

[8]

G. Cormode and M. Hadjieleftheriou. Methods for finding frequent items in data streams. The VLDB Journal, 19(1):3--20, 2010.

Digital Library

[9]

G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58 -- 75, 2005.

Digital Library

[10]

L. De Vito, S. Rapuano, and L. Tomaciello. One-way delay measurement: State of the art. IEEE Transactions on Instrumentation and Measurement, 57(12):2742--2750, 2008.

[11]

N. Duffield and M. Grossglauser. Trajectory sampling for direct traffic observation. IEEE/ACM Transactions on Networking, 9(3):280--292, 2001.

Digital Library

[12]

Endace. DAG network monitoring cards. http://www.endace.com.

[13]

C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, Ignoring the Mice. ACM Transactions on Computer Systems, 21(3):270--313, 2003.

Digital Library

[14]

H. Finucane and M. Mitzenmacher. An improved analysis of the lossy difference aggregator. ACM SIGCOMM Computer Communication Review, 40(2):4--11, 2010.

Digital Library

[15]

S. Fred, T. Bonald, A. Proutiere, G. Regnie, and J. Roberts. Statistical bandwidth sharing: a study of congestion at flow level. In Proc. of ACM SIGCOMM, 2001.

Digital Library

[16]

IEEE. IEEE/ANSI 1588 standard for a precision clock synchronization protocol for networked measurement and control systems, 2002.

[17]

R. Kompella, K. Levchenko, A. Snoeren, and G. Varghese. Every microsecond counts: tracking fine-grain latencies with a lossy difference aggregator. In Proc. of ACM SIGCOMM, 2009.

Digital Library

[18]

M. Lee, N. Duffield, and R. Kompella. Not all microseconds are equal: fine-grained per-flow measurements with reference latency interpolation. In Proc. of ACM SIGCOMM, 2010.

Digital Library

[19]

M. Lee, N. Duffield, and R. Kompella. Two samples are enough: opportunistic flow-level latency estimation using netflow. In Proc. of IEEE INFOCOM, 2010.

Digital Library

[20]

M. Lee, S. Goldberg, R. Kompella, and G. Varghese. Fine-grained latency and loss measurements in the presence of reordering. In Proc. of ACM SIGMETRICS, 2011.

Digital Library

[21]

R. Martin. Wall street's quest to process data at the speed of light. www.informationweek.com/news/infrastructure/showArticle.jhtml?articleID=199200297.

[22]

S. Moon, P. Skelly, and D. Towsley. Estimation and removal of clock skew from network delay measurements. In Proc. of IEEE INFOCOM, 1999.

[23]

K. Park, G. Kim, and M. Crovella. On the relationship between file sizes, transport protocols, and self-similar network traffic. In Proc. of International Conference on Network Protocols, 2002.

Digital Library

[24]

V. Paxson. Measurements and analysis of end-to-end Internet dynamics. Technical Report CSD-97--945, University of California at Berkeley, 1998.

Digital Library

[25]

V. Paxson. On calibrating measurements of packet transit times. In ACM SIGMETRICS Performance Evaluation Review, volume 26, pages 11--21. ACM, 1998.

Digital Library

[26]

C. Y. Robert and J. Segers. Tails of random sums of a heavy-tailed number of light-tailed terms. Insurance: Mathematics and Economics, 43(1):85 -- 92, 2008.

[27]

J. Sanjuas-Cuxart, P. Barlet-Ros, and J. Solé-Pareta. Validation and Improvement of the Lossy Difference Aggregator to Measure Packet Delays. Traffic Monitoring and Analysis Workshop, 2010.

Digital Library

[28]

J. Sommers, P. Barford, N. Duffield, and A. Ron. Accurate and efficient SLA compliance monitoring. In Proc. of ACM SIGCOMM, 2007.

Digital Library

[29]

L. Zhang, Z. Liu, and C. Honghui Xia. Clock synchronization algorithms for network measurements. In Proc. of IEEE INFOCOM, 2002.

Cited By

Li FLv YYan YGao CWang XCao J(2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3364176
Zhang ZWang HLv HSun JLi GHan X(2023)CHAT: Accurate Network Latency Measurement for 5G E2E NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2023.326400931:6(2854-2869)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3264009
Wang HLin HZhong ZYang TShahzad M(2023)Enhanced Machine Learning Sketches for Network MeasurementsIEEE Transactions on Computers10.1109/TC.2022.318556072:4(957-970)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TC.2022.3185560
Show More Cited By

Index Terms

Sketching the delay: tracking temporally uncorrelated flow-level latencies

Recommendations

Network latency and packet delay variation in cyber-physical systems
NSW '11: Proceedings of the 2011 IEEE Network Science Workshop

The problem addressed in this paper is the limitation imposed by network elements, especially Ethernet elements, on the real-time performance of timecritical systems. Most current network elements are concerned only with data integrity, connection, and ...
A Sketch Algorithm to Monitor High Packet Delay in Network Traffic
APNet '21: Proceedings of the 5th Asia-Pacific Workshop on Networking

Packet delay is a consensual indicator of network conditions. High delay packets of a flow indicate that there may be network congestion or network anomalies. In this paper, we consider Intra-FlowPacketDelay (IFPD), which is defined as the time span ...
An In-Depth Measurement Analysis of 5G mmWave PHY Latency and Its Impact on End-to-End Delay
Passive and Active Measurement
Abstract
5G aims to offer not only significantly higher throughput than previous generations of cellular networks, but also promises millisecond (ms) and sub-millisecond (ultra-)low latency support at the 5G physical (PHY) layer for future applications. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '11: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference

November 2011

612 pages

ISBN:9781450310130

DOI:10.1145/2068816

Program Chairs:
Patrick Thiran
EPFL, Switzerland
,
Walter Willinger
AT&T Labs-Research, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IMC '11

Sponsor:

IMC '11: Internet Measurement Conference

November 2 - 4, 2011

Berlin, Germany

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Upcoming Conference

IMC '24

Sponsor:
sigcomm
sigcomm

ACM Internet Measurement Conference

November 4 - 6, 2024

Madrid , AA , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)2

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li FLv YYan YGao CWang XCao J(2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3364176
Zhang ZWang HLv HSun JLi GHan X(2023)CHAT: Accurate Network Latency Measurement for 5G E2E NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2023.326400931:6(2854-2869)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3264009
Wang HLin HZhong ZYang TShahzad M(2023)Enhanced Machine Learning Sketches for Network MeasurementsIEEE Transactions on Computers10.1109/TC.2022.318556072:4(957-970)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TC.2022.3185560
Yan YLi FWang WWang X(2022)TalentSketch: LSTM-based Sketch for Adaptive and High-Precision Network Measurement2022 IEEE 30th International Conference on Network Protocols (ICNP)10.1109/ICNP55882.2022.9940396(1-12)Online publication date: 30-Oct-2022
https://doi.org/10.1109/ICNP55882.2022.9940396
Zhu JZhang KHuang Q(2021)A Sketch Algorithm to Monitor High Packet Delay in Network TrafficProceedings of the 5th Asia-Pacific Workshop on Networking10.1145/3469393.3469398(43-49)Online publication date: 24-Jun-2021
https://dl.acm.org/doi/10.1145/3469393.3469398
Chefrour D(2021)One-Way Delay Measurement From Traditional Networks to SDNACM Computing Surveys10.1145/346616754:7(1-35)Online publication date: 18-Jul-2021
https://dl.acm.org/doi/10.1145/3466167
Martins Rda Silva Villaça RVerdi F(2020)BitMatrix: A Multipurpose Sketch for Monitoring of Multi-tenant NetworksJournal of Network and Systems Management10.1007/s10922-020-09556-7Online publication date: 30-Jul-2020
https://doi.org/10.1007/s10922-020-09556-7
Bonelli NCallegari CProcissi G(2019)A Probabilistic Counting Framework for Distributed MeasurementsIEEE Access10.1109/ACCESS.2019.28991617(22644-22659)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2899161
Fu YBarlet-Ros PLi D(2018)Every Timestamp CountsIEEE/ACM Transactions on Networking10.1109/TNET.2017.276232826:1(90-103)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1109/TNET.2017.2762328
Salehin KRojas-Cessa RLin CDong ZKijkanjanarat T(2015)Scheme to Measure Packet Processing Time of a Remote Host through Estimation of End-Link CapacityIEEE Transactions on Computers10.1109/TC.2013.20364:1(205-218)Online publication date: Jan-2015
https://doi.org/10.1109/TC.2013.203
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents