article

Combining active and passive network measurements to build scalable monitoring systems on the grid

Author:

Bruce B. LowekampAuthors Info & Claims

ACM SIGMETRICS Performance Evaluation Review, Volume 30, Issue 4

Pages 19 - 26

https://doi.org/10.1145/773056.773061

Published: 01 March 2003 Publication History

Abstract

Because the network provides the wires that connect a grid, understanding the performance provided by a network is crucial to achieving satisfactory performance from many grid applications. Monitoring the network to predict its performance for applications is an effective solution, but the costs and scalability challenges of actively injecting measurement traffic, as well as the information access and accuracy challenges of using passively collected measurements, complicate the problem of developing a monitoring solution for a global grid. This paper is a preliminary report on the Wren project, which is focused on developing scalable solutions for network performance monitoring. By combining active and passive monitoring techniques, Wren is able to reduce the need for invasive measurements of the network without sacrificing measurement accuracy on either the WAN or LAN levels. Specifically, we present topology-based steering, which dramatically reduces the number of measurements taken for a system by using passively acquired topology and utilization to select the bottleneck links that require active bandwidth probing. Furthermore, by using passive measurements while an application is running and active measurements when none is running, we preserve our ability to offer accurate, timely predictions of network performance, while eliminating additional invasive measurements.

References

[1]

A. Adams, J. Mahdavi, M. Mathis, and V. Paxson. Creating a scalable architecture for internet monitoring. In Proceedings of the 8th Annual Internet Society Conference (INET'98), Geneva, Switzerland, July 1998.

[2]

J. Bolliger, T. Gross, and U. Hengartner. Bandwidth modelling for network-aware applications. In Proceedings of lnfocomm '99, 1999.

[3]

Y. Breitbart, M. Garofalakis, C. Martin, R. Rastogi, S. Seshadri, and A. Silberschatz. Topology discovery in heterogeneous IP networks. In Proceedings of INFOCOM 2000, March 2000.

[4]

M. Coates, A. O. H. III, R. Nowak, and B. Yu. Internet tomography. IEEE Signal Processing Magazine, 19(3):47--65, May 2002.

[5]

L. Cottrell. Network monitoring tools. http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html.

[6]

M. den Burger, T. Kielmann, and H. E. Bal. TOPOMON: A monitoring tool for grid network topology. In International Conference on Computational Science (2), pages 558--567, 2002.

Digital Library

[7]

P. Dinda, T. Gross, R. Karrer, B. B. Lowekamp, N. Miller, P. Steenkiste, and D. Sutherland. The architecture of the remos system. In Proceedings of the Tenth IEEE International Symposium on High Performance Distributed Computing (HPDC 10), August 2001.

Digital Library

[8]

C. Dovrolis and M. Jain. Pathload: A measurement tool for end-to-end available bandwidth. In Passive and Active Measurements Workshop, 2002.

[9]

A. B. Downey. Using pathchar to estimate internet link characteristics. In Proceedings of ACM SIGCOMM 1999, 1999.

Digital Library

[10]

B. Gaidioz, R. Wolski, and B. Tourancheau. Synchronizing network probes to avoid measurement intrusiveness with the network weather service. In Proceedings of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC 9), pages 147--154, Pittsburgh, PA, August 2000.

Digital Library

[11]

R. Govindan and H. Tangmunarunkit. Heuristics for internet map discovery. In IEEE 1NFOCOM 2000, Tel Aviv, Israel, March 2000.

[12]

S. Jamin, C. Jin, Y. Jin, D. Raz, Y. Shavitt, and L. Zhang. On the placement of internet instrumentation. In IEEE INFOCOM 2000, Tel Aviv, Israel, March 2000.

[13]

G. Kin, G. Yang, B. R. Crowley, and D. A. Agarwal. Network characterization server (NCS). In HPDC11. IEEE, August 2001.

Digital Library

[14]

K. Lai and M. Baker. Nettimer: A tool for measuring bottleneck link bandwidth. In In Proceedings of USENIX Symposium on lnternet Technologies and Systems, 2001.

Digital Library

[15]

C. Lee, N. Chrisochoides, and B. B. Lowekamp. Mesh generation and optimistic computation on the grid. In Proceedings of the Workshop on Performance Analysis and Distributed Computing, Performance Analysis and Grid Computing, Schloss Dagstuhl, August 2002. to appear.

Digital Library

[16]

B. Lowekamp, B. Tiemey, L. Cottrell, R. Hughes-Jones, T. Kielmann, and M. Swany. A hierarchy of network measurements for grid applications and services, draft at http://www-didc.lbl.gov/NMWG/.

[17]

B. B. Lowekamp, D. O'Hallaron, and T. Gross. Direct queries for discovering network resource properties in a distributed environment. In Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing (HPDC), pages 38--46. IEEE Computer Society, August 1999.

Digital Library

[18]

B. B. Lowekamp, D. R. O'Hallaron, and T. Gross. Topology discovery for large ethernet networks. In Proceedings of SIGCOMM 2001. ACM, August 2001.

Digital Library

[19]

M. Mathis, J. Semke, and J. Mahdavi. The macroscopic behavior of the TCP congestion avoidance algorithm. Computer Communications Review, 27(3), 1997.

Digital Library

[20]

V. N. Padmanabhan and L. Subramanian. An investigation of geographic mapping techniques for internet hosts. In Proceedings of ACM SIGCOMM 2001, pages 173--185, 2001.

Digital Library

[21]

S. Seshan, M. Stemm, and R. H. Katz. SPAND: Shared passing network performance discovery. In Proceedings of the USENIX Symposium on Internet Technologies and Systems, pages 135--46, December 1997.

Digital Library

[22]

G. Shao, F. Berman, and R. Wolski. Using effective network views to promote distributed application performance. In Proceedings of the 1999 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'99), 1999.

[23]

S. Small and B. B. Lowekamp. Topology-based clique organization for distributed resource monitoring. In Proceedings of the IEEE/ACM SC2002 Conference, Baltimore, MD, November 2002. Poster.

[24]

M. Swany and R. Wolski. Multivariate resource performance forecasting in the network weather service. In Proceedings of the IEEE/ACM SC2002 Conference, Baltimore, MD, November 2002.

Digital Library

[25]

W. Theilmann and K. Rothermel. Dynamic distance maps of the internet. In IEEE INFOCOM 2000, Tel Aviv, Israel, March 2000.

[26]

Web100. http://www.web100.org/.

[27]

R. Wolski. Forecasting network performance to support dynamic scheduling using the network weather service. In Proceedings of the 6th High Performance Distributed Computing Conference (HPDC), pages 316--25, August 1997.

Digital Library

[28]

R. Wolski. Dynamically forecasting network performance using the network weather service. Cluster Computing, 1(1):119--132, 1998.

Digital Library

Cited By

Janak JRetty HChee DBaloian ASchulzrinne H(2021)Talking After Lights Out: An Ad Hoc Network for Electric Grid Recovery2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)10.1109/SmartGridComm51999.2021.9632338(181-187)Online publication date: 25-Oct-2021
https://doi.org/10.1109/SmartGridComm51999.2021.9632338
Liu HChen RLee CCao WChen L(2020)Using Grid computing architecture in computing resource allocating of IC designJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-020-02246-xOnline publication date: 8-Jul-2020
https://doi.org/10.1007/s12652-020-02246-x
Košťál KHelebrandt PBelluš MRies MKotuliak I(2019)Management and Monitoring of IoT Devices Using BlockchainSensors10.3390/s1904085619:4(856)Online publication date: 19-Feb-2019
https://doi.org/10.3390/s19040856
Show More Cited By

Index Terms

Combining active and passive network measurements to build scalable monitoring systems on the grid

Index terms have been assigned to the content through auto-classification.

Recommendations

Optimal positioning of active and passive monitoring devices
CoNEXT '05: Proceedings of the 2005 ACM conference on Emerging network experiment and technology

Network measurement is essential for assessing performance issues, identifying and locating problems. Two common strategies are the passive approach that attaches specific devices to links in order to monitor the traffic that passes through the network ...
Comparing Passive Network Monitoring of Grid Application Traffic with Active Probes
GRID '03: Proceedings of the 4th International Workshop on Grid Computing

Distributed applications require timely network measurementsso that they can adapt to changing network conditionsand make efficient use of grid resources. One of thekey issues in obtaining network measurements is the intrusivenessof the measurements ...
Using Passive Traces of Application Traffic in a Network Monitoring System
HPDC '04: Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing

Adaptive grid applications require up-to-date network resource measurements and predictions to help steer their adaptation to meet performance goals. To this end, we are interested in monitoring the available bandwidth of the underlying networks in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGMETRICS Performance Evaluation Review

ACM SIGMETRICS Performance Evaluation Review Volume 30, Issue 4

March 2003

48 pages

ISSN:0163-5999

DOI:10.1145/773056

Issue’s Table of Contents

Copyright © 2003 Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 2003

Published in SIGMETRICS Volume 30, Issue 4

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
1,073
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Janak JRetty HChee DBaloian ASchulzrinne H(2021)Talking After Lights Out: An Ad Hoc Network for Electric Grid Recovery2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)10.1109/SmartGridComm51999.2021.9632338(181-187)Online publication date: 25-Oct-2021
https://doi.org/10.1109/SmartGridComm51999.2021.9632338
Liu HChen RLee CCao WChen L(2020)Using Grid computing architecture in computing resource allocating of IC designJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-020-02246-xOnline publication date: 8-Jul-2020
https://doi.org/10.1007/s12652-020-02246-x
Košťál KHelebrandt PBelluš MRies MKotuliak I(2019)Management and Monitoring of IoT Devices Using BlockchainSensors10.3390/s1904085619:4(856)Online publication date: 19-Feb-2019
https://doi.org/10.3390/s19040856
Helebrandt PBellus MRies MKotuliak IKhilenko V(2018)Blockchain Adoption for Monitoring and Management of Enterprise Networks2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON)10.1109/IEMCON.2018.8614960(1221-1225)Online publication date: Nov-2018
https://doi.org/10.1109/IEMCON.2018.8614960
Yang ZYeung K(2016)An efficient flow monitoring algorithm using a flexible match structure2016 IEEE 17th International Conference on High Performance Switching and Routing (HPSR)10.1109/HPSR.2016.7525663(176-181)Online publication date: Jun-2016
https://doi.org/10.1109/HPSR.2016.7525663
Zhang MSwany MYavanamanda AKissel E(2015)HELM: Conflict-free active measurement scheduling for shared network resource management2015 IFIP/IEEE International Symposium on Integrated Network Management (IM)10.1109/INM.2015.7140283(113-121)Online publication date: May-2015
https://doi.org/10.1109/INM.2015.7140283
Garg RSingh A(2015)Adaptive workflow scheduling in grid computing based on dynamic resource availabilityEngineering Science and Technology, an International Journal10.1016/j.jestch.2015.01.00118:2(256-269)Online publication date: Jun-2015
https://doi.org/10.1016/j.jestch.2015.01.001
Elkhatib YEdwards C(2015)Passive Network Awareness as a Means for Improved Grid SchedulingJournal of Grid Computing10.1007/s10723-015-9332-013:2(275-291)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1007/s10723-015-9332-0
Hussain HMalik SHameed AKhan SBickler GMin-Allah NQureshi MZhang LYongji WGhani NKolodziej JZomaya AXu CBalaji PVishnu APinel FPecero JKliazovich DBouvry PLi HWang LChen DRayes A(2013)A survey on resource allocation in high performance distributed computing systemsParallel Computing10.1016/j.parco.2013.09.00939:11(709-736)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1016/j.parco.2013.09.009
Park JRoh HHwang CYoo J(2011)User-friendly bottleneck detection algorithm2011 13th Asia-Pacific Network Operations and Management Symposium10.1109/APNOMS.2011.6077036(1-5)Online publication date: Sep-2011
https://doi.org/10.1109/APNOMS.2011.6077036
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents