article

Free access

A protocol-independent technique for eliminating redundant network traffic

Authors:

Neil T. Spring,

David WetherallAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 30, Issue 4

Pages 87 - 95

https://doi.org/10.1145/347057.347408

Published: 28 August 2000 Publication History

Abstract

We present a technique for identifying repetitive information transfers and use it to analyze the redundancy of network traffic. Our insight is that dynamic content, streaming media and other traffic that is not caught by today's Web caches is nonetheless likely to derive from similar information. We have therefore adapted similarity detection techniques to the problem of designing a system to eliminate redundant transfers. We identify repeated byte ranges between packets to avoid retransmitting the redundant data.

We find a high level of redundancy and are able to detect repetition that Web proxy caches are not. In our traces, after Web proxy caching has been applied, an additional 39% of the original volume of Web traffic is found to be redundant. Moreover, because our technique makes no assumptions about HTTP protocol syntax or caching semantics, it provides immediate benefits for other types of content, such as streaming media, FTP traffic, news and mail.

References

[1]

Squid Web proxy cache. http://www.squid-cache.org/.]]

[2]

National Institute of Standards and Technology, Specifications for secure hash standard, April 1995. Federal Information Processing Standards Publication 180-1.]]

[3]

A. Broder. On the resemblance and containment of documents. In Proceedings of Compression and Complexity of Sequences (SEQUENCES'97), pages 21-29, March 1998.]]

Digital Library

[4]

R. Caceres, F. Douglis, A. Feldmann, G. Glass, and M. Rabinovich. Web proxy caching: The devil is in the details. In Proceedings of the Workshop on Internet Server Performance, June 1998.]]

Digital Library

[5]

CAIDA. Traffic workload overview. http://www.caida.org/Learn/Flow/tcpudp.html, June 1999.]]

[6]

K. Clay, G. Miller, and K. Thompson. The nature of the beast: Recent traffic measurements from an Internet backbone. In Proceedings of INET '98, July 1998.]]

[7]

A. Feldmann, R. Caceres, F. Douglis, G. Glass, and M. Rabinovich. Performance of web proxy caching in heterogeneous bandwidth environments. In Proceedings of IEEE INFOCOM'99, May 1999.]]

[8]

R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. Hypertext transfer protocol - HTTP/1.1, June 1999. Networking Working Group Requests for Comment RFC-2616.]]

Digital Library

[9]

B. C. Housel and D. B. Lindquist. Webexpress: A system for optimizing web browsing in a wireless environment. In Proc. 2nd Annual Intl. Conf. on Mobile Computing and Networking, pages 108-116, Rye, New York, November 1996. ACM. http://www.networking.ibm.com/art/artwewp.htm.]]

Digital Library

[10]

V. Jacobson. Compressing TCP/IP headers for low-speed serial links, February 1990. RFC 1144.]]

Digital Library

[11]

U. Manber. Finding similar les in a large le system. In Proceedings of USENIX Winter 1994 Technical Conference, January 1994.]]

Digital Library

[12]

J. C. Mogul. A trace-based analysis of duplicate suppression in HTTP. Technical Report 99/2, Compaq Computer Corporation Western Research Laboratory, November 1999. available from http://www.research. digital.com/wrl/techreports/abstracts/99.2.html.]]

[13]

J. C. Mogul, F. Douglis, A. Feldmann, and B. Krishnamurthy. Potential benefits of delta encoding and data compression for HTTP. Technical Report 97/4, Compaq Computer Corporation, July 1997. available from http://www.research.digital.com/wrl/techreports/ abstracts/97.4.html.]]

Digital Library

[14]

M. O. Rabin. Fingerprinting by random polynomials. Technical Report TR-15-81, Department of Computer Science, Harvard University, 1981.]]

[15]

R. Rivest. The MD5 message-digest algorithm, 1992. Networking Working Group Requests for Comment, MIT Laboratory for Computer Science and RSA Data Security, Inc., RFC-1321.]]

Digital Library

[16]

J. Santos and D. Wetherall. Increasing effective link bandwidth by suppressing replicated data. In Proceedings of USENIX Annual Technical Conference, 1998.]]

Digital Library

[17]

K. Thompson, G. J. Miller, and R. Wilder. Wide-area Internet traffi patterns and characteristics. IEEE Network, 11(6):10-23, Nov. 1997.]]

Digital Library

[18]

A. van Hoff, J. Giannandrea, M. Hapner, S. Carter, and M. Medin. The HTTP distribution and replication protocol. Technical Report NOTE-DRP, World Wide Web Consortium, August 1997. http://www.w3.org/TR/NOTE-drp-19970825.html.]]

[19]

A. Wolman, G. Voelker, N. Sharma, N. Cardwell, M. Brown, T. Landray, D. Pinnel, A. Karlin, and H. Levy. Organization-based analysis of web-object sharing and caching. In Proceedings of the Second USENIX Symposium on Internet Technologies and Systems (USITS '99), pages 25-36, October 1999.]]

Digital Library

[20]

A. Wolman, G. M. Voelker, N. Sharma, N. Cardwell, A. Karlin, and H. M. Levy. On the scale and performance of cooperative web proxy caching. In Proceedings of the 17th ACM Symposium on Operating Systems Principles (SOSP '99), pages 16-31, December 1999.]]

Digital Library

Cited By

Chen JChen YCai SYin SZhao LZhang Z(2023)An optimized feature extraction algorithm for abnormal network traffic detectionFuture Generation Computer Systems10.1016/j.future.2023.07.039149(330-342)Online publication date: Dec-2023
https://doi.org/10.1016/j.future.2023.07.039
YOSHIDA MMORI KINOUE TTANAKA H(2022)Edge Computing-Enhanced Network Redundancy Elimination for Connected CarsIEICE Transactions on Communications10.1587/transcom.2021TMP0003E105.B:11(1372-1379)Online publication date: 1-Nov-2022
https://doi.org/10.1587/transcom.2021TMP0003
Nam WLee JShroff NLee K(2020)An Inter-Data Encoding Technique that Exploits Synchronized Data for Network ApplicationsIEEE Transactions on Mobile Computing10.1109/TMC.2019.294057820:1(76-92)Online publication date: 3-Dec-2020
https://dl.acm.org/doi/10.1109/TMC.2019.2940578
Show More Cited By

Index Terms

A protocol-independent technique for eliminating redundant network traffic

Recommendations

Redundancy in network traffic: findings and implications
SIGMETRICS '09: Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems

A large amount of popular content is transferred repeatedly across network links in the Internet. In recent years, protocol-independent redundancy elimination, which can remove duplicate strings from within arbitrary network flows, has emerged as a ...
A protocol-independent technique for eliminating redundant network traffic
SIGCOMM '00: Proceedings of the conference on Applications, Technologies, Architectures, and Protocols for Computer Communication

We present a technique for identifying repetitive information transfers and use it to analyze the redundancy of network traffic. Our insight is that dynamic content, streaming media and other traffic that is not caught by today's Web caches is ...
Enhancing redundant network traffic elimination

Protocol-independent redundant traffic elimination (RTE) is a method to detect and remove redundant chunks of data from network-layer packets by using caching at both ends of a network link or path. In this paper, we propose a set of techniques to ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 30, Issue 4

October 2000

319 pages

ISSN:0146-4833

DOI:10.1145/347057

Editor:
Roch Guerin
Univ. of Pennsylvania, Philadelphia

Issue’s Table of Contents

SIGCOMM '00: Proceedings of the conference on Applications, Technologies, Architectures, and Protocols for Computer Communication
August 2000
348 pages
ISBN:1581132239
DOI:10.1145/347059
Chairman:
Craig Partridge
BBN Technologies, Cambridge, MA

Copyright © 2000 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 August 2000

Published in SIGCOMM-CCR Volume 30, Issue 4

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

241
Total Citations
View Citations
1,872
Total Downloads

Downloads (Last 12 months)131
Downloads (Last 6 weeks)26

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen JChen YCai SYin SZhao LZhang Z(2023)An optimized feature extraction algorithm for abnormal network traffic detectionFuture Generation Computer Systems10.1016/j.future.2023.07.039149(330-342)Online publication date: Dec-2023
https://doi.org/10.1016/j.future.2023.07.039
YOSHIDA MMORI KINOUE TTANAKA H(2022)Edge Computing-Enhanced Network Redundancy Elimination for Connected CarsIEICE Transactions on Communications10.1587/transcom.2021TMP0003E105.B:11(1372-1379)Online publication date: 1-Nov-2022
https://doi.org/10.1587/transcom.2021TMP0003
Nam WLee JShroff NLee K(2020)An Inter-Data Encoding Technique that Exploits Synchronized Data for Network ApplicationsIEEE Transactions on Mobile Computing10.1109/TMC.2019.294057820:1(76-92)Online publication date: 3-Dec-2020
https://dl.acm.org/doi/10.1109/TMC.2019.2940578
Tan HZhang ZZou XLiao QXia W(2020)Exploring the Potential of Fast Delta Encoding: Marching to a Higher Compression Ratio2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00030(198-208)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00030
Zhang ZWu JChen LJiang GLam S(2019)Collaborative Task Offloading with Computation Result Reusing for Mobile Edge ComputingThe Computer Journal10.1093/comjnl/bxz027Online publication date: 15-Apr-2019
https://doi.org/10.1093/comjnl/bxz027
Baydeti NVeilumuthu RVaithilingam M(2019)Scalable Models for Redundant Data Flow Analysis in Online Social NetworksWireless Personal Communications10.1007/s11277-019-06375-1Online publication date: 23-Apr-2019
https://doi.org/10.1007/s11277-019-06375-1
Tootoonchian APanda ALan CWalls MArgyraki KRatnasamy SShenker SSeshan SBanerjee S(2018)ResQProceedings of the 15th USENIX Conference on Networked Systems Design and Implementation10.5555/3307441.3307466(283-297)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.5555/3307441.3307466
SHIOMOTO K(2018)Research Challenges for Network Function Virtualization - Re-Architecting Middlebox for High Performance and Efficient, Elastic and Resilient Platform to Create New Services -IEICE Transactions on Communications10.1587/transcom.2017EBI0001E101.B:1(96-122)Online publication date: 2018
https://doi.org/10.1587/transcom.2017EBI0001
Fan JGuan CRen KQiao C(2018)Middlebox-Based Packet-Level Redundancy Elimination Over Encrypted Network TrafficIEEE/ACM Transactions on Networking10.1109/TNET.2018.284679126:4(1742-1753)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.1109/TNET.2018.2846791
He SShen HSoundararaj VYu L(2018)Cloud Assisted Traffic Redundancy Elimination for Power Efficiency in Smartphones2018 IEEE 15th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)10.1109/MASS.2018.00060(371-379)Online publication date: Oct-2018
https://doi.org/10.1109/MASS.2018.00060
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents