research-article

CloudScout: A Non-Intrusive Approach to Service Dependency Discovery

Authors:

Zhaohui WuAuthors Info & Claims

IEEE Transactions on Parallel and Distributed Systems, Volume 28, Issue 5

Pages 1271 - 1284

https://doi.org/10.1109/TPDS.2016.2619715

Published: 01 May 2017 Publication History

Abstract

Nowadays, numerous enterprises are migrating their applications into cloud computing environments. Typically, the applications are composed of several dependent service components that span many hosts and network devices. In light of this, exploring the dependency between service components can be beneficial for achieving fast network application response time. Moreover, it is significant to consolidate service components according to resource constraints, service dependency, and network structure. However, it is a tedious task to discover the dependency among service components without expert knowledge of the running application. In this paper, we propose CloudScout, a non-intrusive approach that is capable of automatically discovering dependent service components. CloudScout analyzes the correlation among service components based on the time-series information from system monitoring logs. We address two key challenges in CloudScout: service distance calculation and dependent service clustering. We conduct experiments on five applications with 290 service components that span 20 physical hosts across two data centers. The experimental results demonstrate that CloudScout can successfully discover the dependency among service components and facilitate reducing the network latency of network applications and distributed applications.

References

[1]

M. P. Papazoglou and W.-J. Van Den Heuvel, “Service oriented architectures: Approaches, technologies and research issues,” VLDB J., vol. Volume 16, no. Issue 3, pp. 389–415, 2007.

Digital Library

[2]

X. Chen, M. Zhang, Z. M. Mao, and P. Bahl, “Automating network application dependency discovery: Experiences, limitations, and new solutions,” in Proc. Operating Syst. Des. Implementation, 2008, pp. 117–130.

Digital Library

[3]

L. Hu, K. Schwan, A. Gulati, J. Zhang, and C. Wang, “Net-cohort: Detecting and managing VM ensembles in virtualized data centers,” in Proc. 9th Int. Conf. Autonomic Comput., 2012, pp. 3–12.

Digital Library

[4]

X. Pu, et al., “Who is your neighbor: Net i/o performance interference in virtualized clouds,” IEEE Trans. Services Comput., vol. Volume 6, no. Issue 3, pp. 314–329, 2013.

Digital Library

[5]

J. T. Piao and J. Yan, “A network-aware virtual machine placement and migration approach in cloud computing,” in Proc. 9th Int. Conf. Grid Cooperative Comput., 2010, pp. 87–92.

Digital Library

[6]

J. Lee, et al., “Application-driven bandwidth guarantees in datacenters,” in Proc. ACM Conf. SIGCOMM, 2014, pp. 467–478.

Digital Library

[7]

O. Biran, et al., “A stable network-aware VM placement for cloud systems,” in Proc. 12th IEEE/ACM Int. Symp. Cluster Cloud Grid Comput., 2012, pp. 498–506.

Digital Library

[8]

X. Zhao, J. Yin, Z. Chen, and X. Lu, “Distance-aware virtual cluster performance optimization: A hadoop case study,” in Proc. IEEE Int. Conf. Cluster Comput., 2013, pp. 1–8.

[9]

N. Tziritas, C.-Z. Xu, T. Loukopoulos, S. U. Khan, and Z. Yu, “Application-aware workload consolidation to minimize both energy consumption and network load in cloud environments,” in Proc. 42nd Int. Conf. Parallel Process., 2013, pp. 449–457.

Digital Library

[10]

Ú. Erlingsson, M. Peinado, S. Peter, M. Budiu, and G. Mainar-Ruiz, “Fay: Extensible distributed tracing from kernels to clusters,” ACM Trans. Comput. Syst., vol. Volume 30, no. Issue 4, 2012, Art. no. .

Digital Library

[11]

B. C. Tak, C. Tang, C. Zhang, S. Govindan, B. Urgaonkar, and R. N. Chang, “vPath: Precise discovery of request processing paths from black-box observations of thread and network activities,” in Proc. Conf. USENIX Annu. Tech. Conf., 2009, pp. 19–19.

Digital Library

[12]

M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. A. Brewer, “Pinpoint: Problem determination in large, dynamic internet services,” in Proc. Int. Conf. Dependable Syst. Netw., 2002, pp. 595–604.

Digital Library

[13]

P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, “Using Magpie for request extraction and workload modelling,” in Proc. 6th Symp. Operating Syst. Des. Implementation, 2004, pp. 259–272.

Digital Library

[14]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen, “Performance debugging for distributed systems of black boxes,” in Proc. ACM Symp. Operating Syst. Principles, 2003, pp. 74–89.

Digital Library

[15]

P. Reynolds, J. L. Wiener, J. C. Mogul, M. K. Aguilera, and A. Vahdat, “Wap5: Black-box performance debugging for wide-area systems,” in Proc. 15th Int. Conf. World Wide Web, 2006, pp. 347–356.

Digital Library

[16]

P. Bahl, R. Chandra, A. Greenberg, S. Kandula, D. A. Maltz, and M. Zhang, “Towards highly reliable enterprise network services via inference of multi-level dependencies,” SIGCOMM Comput. Commun. Rev., vol. Volume 37, no. Issue 4, pp. 13–24, 2007.

Digital Library

[17]

J.-G. Lou, Q. Fu, Y. Wang, and J. Li, “Mining dependency in distributed systems through unstructured logs analysis,” SIGOPS Oper. Syst. Rev., vol. Volume 44, no. Issue 1, pp. 91–96, 2010.

Digital Library

[18]

R. Apte, L. Hu, K. Schwan, and A. Ghosh, “Look who's talking: Discovering dependencies between virtual machines using cpu utilization,” in Proc. 2nd USENIX Conf. Hot Topics Cloud Comput., 2010, pp. 17–17.

Digital Library

[19]

B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R. Das, “Modeling and synthesizing task placement constraints in Google compute clusters,” in Proc. 2nd ACM Symp. Cloud Comput., 2011, pp. 3:1–3:14.

Digital Library

[20]

F. P. Tso, K. Oikonomou, E. Kavvadia, and D. P. Pezaros, “Scalable traffic-aware virtual machine management for cloud data centers,” in Proc. IEEE 34th Distrib. Comput. Syst., 2014, pp. 238–247.

Digital Library

[21]

X. Zhao, J. Yin, Z. Chen, and S. He, “Workload classification model for specializing virtual machine operating system,” in Proc. IEEE 6th Int. Conf. Cloud Comput., Jun. 2013, pp. 343–350.

Digital Library

[22]

A. Natarajan, P. Ning, Y. Liu, S. Jajodia, and S. E. Hutchinson, “NSDminer: Automated discovery of network service dependencies,” Proc. IEEE INFOCOM, 2012, pp. 2507–2515.

[23]

I. Shafer, K. Ren, V. N. Boddeti, Y. Abe, G. R. Ganger, and C. Faloutsos, “RainMon: An integrated approach to mining bursty timeseries monitoring data,” in Proc. 18th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2012, pp. 1158–1166.

Digital Library

[24]

R. Xu and D. Wunsch, “Survey of clustering algorithms,” IEEE Trans. Neural Netw., vol. Volume 16, no. Issue 3, pp. 645–678, 2005.

Digital Library

[25]

B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi, “An analytical model for multi-tier internet services and its applications,” ACM SIGMETRICS Performance Eval. Rev., vol. Volume 33, no. Issue 1, pp. 291–302, 2005.

Digital Library

[26]

S. Chen, K. R. Joshi, M. A. Hiltunen, R. D. Schlichting, and W. H. Sanders, “Blackbox prediction of the impact of DVFs on end-to-end performance of multitier systems,” ACM SIGMETRICS Performance Eval. Rev., vol. Volume 37, no. Issue 4, pp. 59–63, 2010.

Digital Library

[27]

R. Singh, P. Shenoy, M. Natu, V. Sadaphal, and H. Vin, “Analytical modeling for what-if analysis in complex cloud computing applications,” ACM SIGMETRICS Performance Eval. Rev., vol. Volume 40, no. Issue 4, pp. 53–62, 2013.

Digital Library

Cited By

Chen HChen PYu GLi XHe Z(2024)MicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microservice ApplicationsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.336390221:5(4921-4938)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TDSC.2024.3363902
Yang TShen JSu YLing XYang YLyu MGrundy J(2021)AIDProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678534(653-665)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678534
Yu LLi Y(2020)Adaptive Method for Discovering Service Provider in Cloud Composite ServicesDatabase Systems for Advanced Applications10.1007/978-3-030-59410-7_15(246-260)Online publication date: 24-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-59410-7_15

Recommendations

Nosv

nOSV can provide a bare-metal like performance for HPC applications on Cloud.The CPU cores and main memory are not shared among guest VMs of nOSV.Dedicated I/O resources are allocated to I/O sensitive HPC guests.Other virtualization environments can run ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Parallel and Distributed Systems

IEEE Transactions on Parallel and Distributed Systems Volume 28, Issue 5

May 2017

306 pages

ISSN:1045-9219

Issue’s Table of Contents

Copyright © 2017.

Publisher

IEEE Press

Publication History

Published: 01 May 2017

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen HChen PYu GLi XHe Z(2024)MicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microservice ApplicationsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.336390221:5(4921-4938)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TDSC.2024.3363902
Yang TShen JSu YLing XYang YLyu MGrundy J(2021)AIDProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678534(653-665)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678534
Yu LLi Y(2020)Adaptive Method for Discovering Service Provider in Cloud Composite ServicesDatabase Systems for Advanced Applications10.1007/978-3-030-59410-7_15(246-260)Online publication date: 24-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-59410-7_15
Arthur-Durett KCarroll TChikkagoudar S(2018)Discovering Significant Co-Occurrences to Characterize Network BehaviorsHuman Interface and the Management of Information. Interaction, Visualization, and Analytics10.1007/978-3-319-92043-6_49(609-623)Online publication date: 15-Jul-2018
https://dl.acm.org/doi/10.1007/978-3-319-92043-6_49

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents