research-article

Estimating traffic and anomaly maps via network tomography

Authors:

Morteza Mardani,

Georgios B. GiannakisAuthors Info & Claims

IEEE/ACM Transactions on Networking (TON), Volume 24, Issue 3

Pages 1533 - 1547

https://doi.org/10.1109/TNET.2015.2417809

Published: 01 June 2016 Publication History

Abstract

Mapping origin-destination (OD) network traffic is pivotal for network management and proactive security tasks. However, lack of sufficient flow-level measurements as well as potential anomalies pose major challenges towards this goal. Leveraging the spatiotemporal correlation of nominal traffic, and the sparse nature of anomalies, this paper brings forth a novel framework to map out nominal and anomalous traffic, which treats jointly important network monitoring tasks including traffic estimation, anomaly detection, and traffic interpolation. To this end, a convex program is first formulated with nuclear and l<sb>1</sb>-norm regularization to effect sparsity and low rank for the nominal and anomalous traffic with only the link counts and a small subset of OD-flow counts. Analysis and simulations confirm that the proposed estimator can exactly recover sufficiently low-dimensional nominal traffic and sporadic anomalies so long as the routing paths are sufficiently "spread-out" across the network, and an adequate amount of flow counts are randomly sampled. The results offer valuable insights about data acquisition strategies and network scenaria giving rise to accurate traffic estimation. For practical networks where the aforementioned conditions are possibly violated, the inherent spatiotemporal traffic patterns are taken into account by adopting a Bayesian approach along with a bilinear characterization of the nuclear and l₁ norms. The resultant nonconvex program involves quadratic regularizers with correlation matrices, learned systematically from (cyclo)stationary historical data. Alternating-minimization based algorithms with provable convergence are also developed to procure the estimates. Insightful tests with synthetic and real Internet data corroborate the effectiveness of the novel schemes.

References

[1]

Internet-2 Observatory. {Online}. Available: http://internet2.edu/observatory/archive/data-collections.html

[2]

P. Barford and D. Plonka, "Characteristics of network traffic flow anomalies," in Proc. 1st ACM SIGCOMM Workshop on Internet Measurements, San Francisco, CA, USA, Nov. 2001.

Digital Library

[3]

J. A. Bazerque, G. Mateos, and G. B. Giannakis, "Rank regularization and Bayesian inference for tensor completion and extrapolation," IEEE Trans. Signal Process., vol. 61, no. 22, pp. 5689--5703, Nov. 2013.

Digital Library

[4]

D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, 2nd ed. Belmont, MA, USA: Athena Scientific, 1999.

[5]

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Found. Trends Mach. Learning, vol. 3, pp. 1--122, 2010.

Digital Library

[6]

S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, MA, USA: Cambridge Univ. Press, 2004.

Digital Library

[7]

E. J. Candès, X. Li, Y. Ma, and J. Wright, "Robust principal component analysis?," J. ACM, vol. 58, no. 1, pp. 1--37, 2011.

Digital Library

[8]

E. J. Candès and Y. Plan, "Matrix completion with noise," Proc. IEEE, vol. 98, pp. 925--936, 2009.

[9]

E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Found. Comput. Math., vol. 9, no. 6, pp. 717--722, 2009.

[10]

E. J. Candès and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203--4215, 2005.

Digital Library

[11]

E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Found. Computat. Math., vol. 9, no. 6, pp. 717--772, 2009.

[12]

E. Cascetta, "Estimation of trip matrices from traffic counts and survey data: A generalized least-squares estimator," Transport. Res. B: Methodological, vol. 18, pp. 289--299, 1984.

[13]

V. Chandrasekaran, S. Sanghavi, P. R. Parrilo, and A. S. Willsky, "Rank-sparsity incoherence for matrix decomposition," SIAM J. Optim., vol. 21, no. 2, pp. 572--596, 2011.

[14]

F. Deutsch, Best Approximation in Inner Product Spaces, 2nd ed. New York, NY, USA: Springer-Verlag, 2001.

[15]

T. Eltoft, T. Kim, and T.-W. Lee, "On the multivariate Laplace distribution," IEEE Signal Process. Lett., vol. 13, pp. 300--303, May 2006.

[16]

A. Ganesh, K. Min, J. Wright, and Y. Ma, "Principal component pursuit with reduced linear measurements," 2012 {Online}. Available: arXiv:1202.6445v1 {cs.IT}

[17]

G. B. Giannakis, "Digital signal processing handbook," in Cyclostationary Signal Analysis, V. K. Madisetti and D. Williams, Eds. Boca Raton, FL, USA: CRC Press, 1998.

[18]

R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge, MA, USA: Cambridge Univ. Press, 1985.

Digital Library

[19]

H. Kim, S. Lee, X. Ma, and C. Wang, "Higher-order PCA for anomaly detection in large-scale networks," in Proc. 3rd Workshop on Comp. Advances in Multi-Sensor Adaptive Process., Aruba, Dutch Antilles, Dec. 2009.

[20]

E. D. Kolaczyk, Analysis of Network Data: Methods and Models. New York, NY, USA: Springer, 2009.

Digital Library

[21]

A. Lakhina, M. Crovella, and C. Diot, "Diagnosing network-wide traffic anomalies," in Proc. ACM SIGCOMM, Portland, OR, USA, Aug. 2004.

Digital Library

[22]

A. Lakhina, M. Crovella, and C. Diot, "Mining anomalies using traffic feature distributions," Proc. ACM SIGCOMM '05, vol. 35, pp. 217--228, Aug. 2005.

Digital Library

[23]

A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, E. D. Kolaczyk, and N. Taft, "Structural analysis of network traffic flows," in Proc. ACM SIGMETRICS, New York, NY, USA, Jul. 2004.

Digital Library

[24]

M. Mardani, G. Mateos, and G. B. Giannakis, "Dynamic anomalography: Tracking network anomalies via sparsity and low rank," IEEE J. Sel. Topics in Signal Process., vol. 7, no. 11, pp. 50--66, Feb. 2013.

[25]

M. Mardani, G. Mateos, and G. B. Giannakis, "In-network sparsity regularized rank minimization: Applications and algorithms," IEEE Trans. Signal Process., vol. 59, pp. 5374--5388, Nov. 2013.

Digital Library

[26]

M. Mardani, G. Mateos, and G. B. Giannakis, "Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies," IEEE Trans. Inf. Theory., vol. 59, pp. 5186--5205, Aug. 2013.

Digital Library

[27]

M. Mardani, G. Mateos, and G. B. Giannakis, "Subspace learning and imputation for streaming big data matrices and tensors," 2014 {Online}. Available: arXiv:1404.4667v1 {stat.ML}

[28]

B. K. Natarajan, "Sparse approximate solutions to linear systems," SIAM J. Comput., vol. 24, pp. 227--234, 1995.

Digital Library

[29]

Y. Nesterov, "A method of solving a convex programming problem with convergence rate o(1/k<sp>2</sp>)," Soviet Mathemat. Doklady, vol. 27, pp. 372--376, 1983.

[30]

K. Papagiannaki, R. Cruz, and C. Diot, "Network performance monitoring at small time scales," in Proc. 1st ACM SIGCOMM Workshop on Internet Measurements, Miami Beach, FL, USA, Oct. 2003.

Digital Library

[31]

M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization," SIAM J. Optimization, vol. 23, pp. 1126--1153, 2013.

[32]

B. Recht, M. Fazel, and P. A. Parrilo, "Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization," SIAM Rev., vol. 52, no. 3, pp. 471--501, 2010.

Digital Library

[33]

M. Roughan, Y. Zhang, W. Willinger, and L. Qiu, "Spatio-temporal compressive sensing and Internet traffic matrices," IEEE/ACM Trans. Netw., vol. 20, pp. 662--676, 2012.

Digital Library

[34]

S. G. Samko, A. A. Kilbas, and O. I. Marichev, Fractional Integrals and Derivatives. Yverdon, Switzerland: Gordon and Breach, Springer, 1993.

[35]

Y. Shavitt, X. Sun, A. Wool, and B. Yener, "Computing the unmeasured: An algebraic approach to Internet mapping," in Proc. IEEE Int. Conf. Computer Commun. (INFOCOM), Anchorage, AK, USA, Apr. 2001.

[36]

V. Solo and X. Kong, Adaptive Signal Processing Algorithms: Stability and Performance. Englewood Cliffs, NJ, USA: Prentice-Hall, 1995.

Digital Library

[37]

N. Srebro and A. Shraibman, "Rank, trace-norm and max-norm," in Proc. Learning Theory, 2005, pp. 545--560.

Digital Library

[38]

M. Thottan and C. Ji, "Anomaly detection in IP networks," IEEE Trans. Signal Process., vol. 51, pp. 2191--2204, Aug. 2003.

Digital Library

[39]

Y. Vardi, "Network tomography: Estimating source-destination traffic intensities from link data," J. Amer. Statistical Assoc., vol. 91, pp. 365--377, 1996.

[40]

X. Wu, K. Yu, and X. Wang, "On the growth of Internet application flows: A complex network perspective," in Proc. IEEE Int. Conf. Computer Commun. (INFOCOM), Shangai, China, 2011, pp. 2096--2104.

[41]

Y. Zhang, Z. Ge, A. Greenberg, and M. Roughan, "Network anomography," in Proc. ACM SIGCOM Conf. Interent Measurements, Berkeley, CA, USA, Oct. 2005.

Digital Library

[42]

Y. Zhang, M. Roughan, N. Duffield, and A. Greenberg, "Fast accurate computation of large-scale IP traffic matrices from link loads," in ACM SIGMETRICS Perform. Eval. Rev., 2003, vol. 31, pp. 206--217.

Digital Library

[43]

Q. Zhao, Z. Ge, J. Wang, and J. Xu, "Robust traffic matrix estimation with imperfect information: Making use of multiple data sources," ACM SIGMETRICS '06, vol. 34, pp. 133--144, 2006.

Digital Library

[44]

H. V. Zuylen and L. Willumsen, "The most likely trip matrix estimated from traffic counts," Transport. Res. B: Methodological, vol. 14, pp. 281--293, 1980.

Cited By

Sacco AEsposito FMarchetto G(2023)Completing and Predicting Internet Traffic Matrices Using Adversarial Autoencoders and Hidden Markov ModelsIEEE Transactions on Network and Service Management10.1109/TNSM.2023.327016620:3(2244-2258)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TNSM.2023.3270166
Polverini MCianfrani AListanti MSiano GLavacca FCampanile C(2023)Investigating on Black Holes in Segment Routing Networks: Identification and DetectionIEEE Transactions on Network and Service Management10.1109/TNSM.2022.319745320:1(14-29)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TNSM.2022.3197453
Memon RQazi SKhan B(2021)Design and Implementation of a Robust Convolutional Neural Network-Based Traffic Matrix Estimator for Cloud NetworksWireless Communications & Mobile Computing10.1155/2021/10396132021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/1039613
Show More Cited By

Estimating traffic and anomaly maps via network tomography
1. Networks
  1. Network services

Recommendations

End-to-End Network Traffic Reconstruction Via Network Tomography Based on Compressive Sensing

Traffic matrices (TM) represent the volumes of end-to-end network traffic between each of the origin---destination pairs. Accurate estimates of TM are used by network operators to perform network management functions and traffic engineering tasks. ...
Anomaly-Aware Network Traffic Estimation via Outlier-Robust Tensor Completion
Accurately estimating network traffic from the partial measurements plays a crucial role in network management. However, the potential anomaly existing in real networks usually makes this goal difficult to achieve. Existing network traffic estimation ...
Image compressive sensing recovery using adaptively learned sparsifying basis via L0 minimization

From many fewer acquired measurements than suggested by the Nyquist sampling theory, compressive sensing (CS) theory demonstrates that, a signal can be reconstructed with high probability when it exhibits sparsity in some domain. Most of the ...

Comments

Information & Contributors

Information

Published In

cover image IEEE/ACM Transactions on Networking

IEEE/ACM Transactions on Networking Volume 24, Issue 3

June 2016

638 pages

ISSN:1063-6692

Editor:
R. Srikant
Univ. Illinois at Urbana-Champaign, Urbana, IL

Issue’s Table of Contents

Publisher

IEEE Press

Publication History

Published: 01 June 2016

Published in TON Volume 24, Issue 3

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
134
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sacco AEsposito FMarchetto G(2023)Completing and Predicting Internet Traffic Matrices Using Adversarial Autoencoders and Hidden Markov ModelsIEEE Transactions on Network and Service Management10.1109/TNSM.2023.327016620:3(2244-2258)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TNSM.2023.3270166
Polverini MCianfrani AListanti MSiano GLavacca FCampanile C(2023)Investigating on Black Holes in Segment Routing Networks: Identification and DetectionIEEE Transactions on Network and Service Management10.1109/TNSM.2022.319745320:1(14-29)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TNSM.2022.3197453
Memon RQazi SKhan B(2021)Design and Implementation of a Robust Convolutional Neural Network-Based Traffic Matrix Estimator for Cloud NetworksWireless Communications & Mobile Computing10.1155/2021/10396132021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/1039613
Wang QChen LWang QZhu HWang X(2020)Anomaly-Aware Network Traffic Estimation via Outlier-Robust Tensor CompletionIEEE Transactions on Network and Service Management10.1109/TNSM.2020.302493217:4(2677-2689)Online publication date: 1-Dec-2020
https://dl.acm.org/doi/10.1109/TNSM.2020.3024932
Qin TLiu ZWang PLi SGuan XGao L(2020)Symmetry Degree Measurement and its Applications to Anomaly DetectionIEEE Transactions on Information Forensics and Security10.1109/TIFS.2019.293373115(1040-1055)Online publication date: 1-Jan-2020
https://dl.acm.org/doi/10.1109/TIFS.2019.2933731
Tayal ASharma NHubballi NNatu M(2020)Traffic Dynamics-Aware Probe Selection for Fault Detection in NetworksJournal of Network and Systems Management10.1007/s10922-020-09514-328:4(1055-1084)Online publication date: 1-Oct-2020
https://dl.acm.org/doi/10.1007/s10922-020-09514-3
Montasari RHosseinian-Far AHill RMontaseri FSharma MShabbir S(2018)Are Timing-Based Side-Channel Attacks Feasible in Shared, Modern Computing Hardware?International Journal of Organizational and Collective Intelligence10.4018/IJOCI.20180401038:2(32-59)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.4018/IJOCI.2018040103
Polverini MCianfrani AListanti MBaiocchi A(2018)Routing Perturbation for Traffic Matrix Evaluation in a Segment Routing NetworkIEEE Transactions on Network and Service Management10.1109/TNSM.2018.286242315:4(1645-1660)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1109/TNSM.2018.2862423
Malboubi MPeng SSharma PChuah C(2018)A learning-based measurement framework for traffic matrix inference in software defined networksComputers and Electrical Engineering10.1016/j.compeleceng.2017.11.02066:C(369-387)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.compeleceng.2017.11.020
Kaloorazi Mde Lamare R(2017)Anomaly detection in IP networks based on randomized subspace methods2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2017.7952952(4222-4226)Online publication date: 5-Mar-2017
https://dl.acm.org/doi/10.1109/ICASSP.2017.7952952
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents