-
What is LTE actually used for? An answer through multi-operator, crowd-sourced measurement
Authors:
Francesco Malandrino,
Scott Kirkpatrick,
Danny Bickson
Abstract:
LTE networks are commonplace nowadays; however, comparatively little is known about where (and why) they are deployed, and the demand they serve. We shed some light on these issues through large-scale, crowd-sourced measurement. Our data, collected by users of the WeFi app, spans multiple operators and multiple cities, allowing us to observe a wide variety of deployment patterns. Surprisingly, we…
▽ More
LTE networks are commonplace nowadays; however, comparatively little is known about where (and why) they are deployed, and the demand they serve. We shed some light on these issues through large-scale, crowd-sourced measurement. Our data, collected by users of the WeFi app, spans multiple operators and multiple cities, allowing us to observe a wide variety of deployment patterns. Surprisingly, we find that LTE is frequently used to improve the {\em coverage} of network rather than the capacity thereof, and that no evidence shows that video traffic be a primary driver for its deployment. Our insights suggest that such factors as pre-existing networks and commercial policies have a deeper impact on deployment decisions than purely technical considerations.
△ Less
Submitted 23 November, 2016;
originally announced November 2016.
-
GraphLab: A New Framework For Parallel Machine Learning
Authors:
Yucheng Low,
Joseph E. Gonzalez,
Aapo Kyrola,
Danny Bickson,
Carlos E. Guestrin,
Joseph Hellerstein
Abstract:
Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions…
▽ More
Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions like MapReduce by compactly expressing asynchronous iterative algorithms with sparse computational dependencies while ensuring data consistency and achieving a high degree of parallel performance. We demonstrate the expressiveness of the GraphLab framework by designing and implementing parallel versions of belief propagation, Gibbs sampling, Co-EM, Lasso and Compressed Sensing. We show that using GraphLab we can achieve excellent parallel performance on large scale real-world problems.
△ Less
Submitted 9 August, 2014;
originally announced August 2014.
-
Distributed GraphLab: A Framework for Machine Learning in the Cloud
Authors:
Yucheng Low,
Joseph Gonzalez,
Aapo Kyrola,
Danny Bickson,
Carlos Guestrin,
Joseph M. Hellerstein
Abstract:
While high-level data parallel frameworks, like MapReduce, simplify the design and implementation of large-scale data processing systems, they do not naturally or efficiently support many important data mining and machine learning algorithms and can lead to inefficient learning systems. To help fill this critical void, we introduced the GraphLab abstraction which naturally expresses asynchronous,…
▽ More
While high-level data parallel frameworks, like MapReduce, simplify the design and implementation of large-scale data processing systems, they do not naturally or efficiently support many important data mining and machine learning algorithms and can lead to inefficient learning systems. To help fill this critical void, we introduced the GraphLab abstraction which naturally expresses asynchronous, dynamic, graph-parallel computation while ensuring data consistency and achieving a high degree of parallel performance in the shared-memory setting. In this paper, we extend the GraphLab framework to the substantially more challenging distributed setting while preserving strong data consistency guarantees. We develop graph based extensions to pipelined locking and data versioning to reduce network congestion and mitigate the effect of network latency. We also introduce fault tolerance to the GraphLab abstraction using the classic Chandy-Lamport snapshot algorithm and demonstrate how it can be easily implemented by exploiting the GraphLab abstraction itself. Finally, we evaluate our distributed implementation of the GraphLab abstraction on a large Amazon EC2 deployment and show 1-2 orders of magnitude performance gains over Hadoop-based implementations.
△ Less
Submitted 26 April, 2012;
originally announced April 2012.
-
Efficient Multicore Collaborative Filtering
Authors:
Yao Wu,
Qiang Yan,
Danny Bickson,
Yucheng Low,
Qing Yang
Abstract:
This paper describes the solution method taken by LeBuSiShu team for track1 in ACM KDD CUP 2011 contest (resulting in the 5th place). We identified two main challenges: the unique item taxonomy characteristics as well as the large data set size.To handle the item taxonomy, we present a novel method called Matrix Factorization Item Taxonomy Regularization (MFITR). MFITR obtained the 2nd best predic…
▽ More
This paper describes the solution method taken by LeBuSiShu team for track1 in ACM KDD CUP 2011 contest (resulting in the 5th place). We identified two main challenges: the unique item taxonomy characteristics as well as the large data set size.To handle the item taxonomy, we present a novel method called Matrix Factorization Item Taxonomy Regularization (MFITR). MFITR obtained the 2nd best prediction result out of more then ten implemented algorithms. For rapidly computing multiple solutions of various algorithms, we have implemented an open source parallel collaborative filtering library on top of the GraphLab machine learning framework. We report some preliminary performance results obtained using the BlackLight supercomputer.
△ Less
Submitted 17 August, 2011; v1 submitted 12 August, 2011;
originally announced August 2011.
-
GraphLab: A Distributed Framework for Machine Learning in the Cloud
Authors:
Yucheng Low,
Joseph Gonzalez,
Aapo Kyrola,
Danny Bickson,
Carlos Guestrin
Abstract:
Machine Learning (ML) techniques are indispensable in a wide range of fields. Unfortunately, the exponential increase of dataset sizes are rapidly extending the runtime of sequential algorithms and threatening to slow future progress in ML. With the promise of affordable large-scale parallel computing, Cloud systems offer a viable platform to resolve the computational challenges in ML. However, de…
▽ More
Machine Learning (ML) techniques are indispensable in a wide range of fields. Unfortunately, the exponential increase of dataset sizes are rapidly extending the runtime of sequential algorithms and threatening to slow future progress in ML. With the promise of affordable large-scale parallel computing, Cloud systems offer a viable platform to resolve the computational challenges in ML. However, designing and implementing efficient, provably correct distributed ML algorithms is often prohibitively challenging. To enable ML researchers to easily and efficiently use parallel systems, we introduced the GraphLab abstraction which is designed to represent the computational patterns in ML algorithms while permitting efficient parallel and distributed implementations. In this paper we provide a formal description of the GraphLab parallel abstraction and present an efficient distributed implementation. We conduct a comprehensive evaluation of GraphLab on three state-of-the-art ML algorithms using real large-scale data and a 64 node EC2 cluster of 512 processors. We find that GraphLab achieves orders of magnitude performance gains over Hadoop while performing comparably or superior to hand-tuned MPI implementations.
△ Less
Submitted 5 July, 2011;
originally announced July 2011.
-
Kernel Belief Propagation
Authors:
Le Song,
Arthur Gretton,
Danny Bickson,
Yucheng Low,
Carlos Guestrin
Abstract:
We propose a nonparametric generalization of belief propagation, Kernel Belief Propagation (KBP), for pairwise Markov random fields. Messages are represented as functions in a reproducing kernel Hilbert space (RKHS), and message updates are simple linear operations in the RKHS. KBP makes none of the assumptions commonly required in classical BP algorithms: the variables need not arise from a finit…
▽ More
We propose a nonparametric generalization of belief propagation, Kernel Belief Propagation (KBP), for pairwise Markov random fields. Messages are represented as functions in a reproducing kernel Hilbert space (RKHS), and message updates are simple linear operations in the RKHS. KBP makes none of the assumptions commonly required in classical BP algorithms: the variables need not arise from a finite domain or a Gaussian distribution, nor must their relations take any particular parametric form. Rather, the relations between variables are represented implicitly, and are learned nonparametrically from training data. KBP has the advantage that it may be used on any domain where kernels are defined (Rd, strings, groups), even where explicit parametric models are not known, or closed form expressions for the BP updates do not exist. The computational cost of message updates in KBP is polynomial in the training data size. We also propose a constant time approximate message update procedure by representing messages using a small number of basis functions. In experiments, we apply KBP to image denoising, depth prediction from still images, and protein configuration prediction: KBP is faster than competing classical and nonparametric approaches (by orders of magnitude, in some cases), while providing significantly more accurate results.
△ Less
Submitted 27 May, 2011;
originally announced May 2011.
-
Parallel Coordinate Descent for L1-Regularized Loss Minimization
Authors:
Joseph K. Bradley,
Aapo Kyrola,
Danny Bickson,
Carlos Guestrin
Abstract:
We propose Shotgun, a parallel coordinate descent algorithm for minimizing L1-regularized losses. Though coordinate descent seems inherently sequential, we prove convergence bounds for Shotgun which predict linear speedups, up to a problem-dependent limit. We present a comprehensive empirical study of Shotgun for Lasso and sparse logistic regression. Our theoretical predictions on the potential fo…
▽ More
We propose Shotgun, a parallel coordinate descent algorithm for minimizing L1-regularized losses. Though coordinate descent seems inherently sequential, we prove convergence bounds for Shotgun which predict linear speedups, up to a problem-dependent limit. We present a comprehensive empirical study of Shotgun for Lasso and sparse logistic regression. Our theoretical predictions on the potential for parallelism closely match behavior on real data. Shotgun outperforms other published solvers on a range of large problems, proving to be one of the most scalable algorithms for L1.
△ Less
Submitted 26 May, 2011;
originally announced May 2011.
-
Inference with Multivariate Heavy-Tails in Linear Models
Authors:
Danny Bickson,
Carlos Guestrin
Abstract:
Heavy-tailed distributions naturally occur in many real life problems. Unfortunately, it is typically not possible to compute inference in closed-form in graphical models which involve such heavy-tailed distributions.
In this work, we propose a novel simple linear graphical model for independent latent random variables, called linear characteristic model (LCM), defined in the characteristic func…
▽ More
Heavy-tailed distributions naturally occur in many real life problems. Unfortunately, it is typically not possible to compute inference in closed-form in graphical models which involve such heavy-tailed distributions.
In this work, we propose a novel simple linear graphical model for independent latent random variables, called linear characteristic model (LCM), defined in the characteristic function domain. Using stable distributions, a heavy-tailed family of distributions which is a generalization of Cauchy, Lévy and Gaussian distributions, we show for the first time, how to compute both exact and approximate inference in such a linear multivariate graphical model. LCMs are not limited to stable distributions, in fact LCMs are always defined for any random variables (discrete, continuous or a mixture of both).
We provide a realistic problem from the field of computer networks to demonstrate the applicability of our construction. Other potential application is iterative decoding of linear channels with non-Gaussian noise.
△ Less
Submitted 21 March, 2011; v1 submitted 31 August, 2010;
originally announced August 2010.
-
GraphLab: A New Framework for Parallel Machine Learning
Authors:
Yucheng Low,
Joseph Gonzalez,
Aapo Kyrola,
Danny Bickson,
Carlos Guestrin,
Joseph M. Hellerstein
Abstract:
Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions…
▽ More
Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions like MapReduce by compactly expressing asynchronous iterative algorithms with sparse computational dependencies while ensuring data consistency and achieving a high degree of parallel performance. We demonstrate the expressiveness of the GraphLab framework by designing and implementing parallel versions of belief propagation, Gibbs sampling, Co-EM, Lasso and Compressed Sensing. We show that using GraphLab we can achieve excellent parallel performance on large scale real-world problems.
△ Less
Submitted 25 June, 2010;
originally announced June 2010.
-
Fault Identification via Non-parametric Belief Propagation
Authors:
Danny Bickson,
Dror Baron,
Alex T. Ihler,
Harel Avissar,
Danny Dolev
Abstract:
We consider the problem of identifying a pattern of faults from a set of noisy linear measurements. Unfortunately, maximum a posteriori probability estimation of the fault pattern is computationally intractable. To solve the fault identification problem, we propose a non-parametric belief propagation approach. We show empirically that our belief propagation solver is more accurate than recent stat…
▽ More
We consider the problem of identifying a pattern of faults from a set of noisy linear measurements. Unfortunately, maximum a posteriori probability estimation of the fault pattern is computationally intractable. To solve the fault identification problem, we propose a non-parametric belief propagation approach. We show empirically that our belief propagation solver is more accurate than recent state-of-the-art algorithms including interior point methods and semidefinite programming. Our superior performance is explained by the fact that we take into account both the binary nature of the individual faults and the sparsity of the fault pattern arising from their rarity.
△ Less
Submitted 1 February, 2011; v1 submitted 13 August, 2009;
originally announced August 2009.
-
Distributed Sensor Selection using a Truncated Newton Method
Authors:
Danny Bickson,
Danny Dolev
Abstract:
We propose a new distributed algorithm for computing a truncated Newton method, where the main diagonal of the Hessian is computed using belief propagation. As a case study for this approach, we examine the sensor selection problem, a Boolean convex optimization problem. We form two distributed algorithms. The first algorithm is a distributed version of the interior point method by Joshi and Boy…
▽ More
We propose a new distributed algorithm for computing a truncated Newton method, where the main diagonal of the Hessian is computed using belief propagation. As a case study for this approach, we examine the sensor selection problem, a Boolean convex optimization problem. We form two distributed algorithms. The first algorithm is a distributed version of the interior point method by Joshi and Boyd, and the second algorithm is an order of magnitude faster approximation. As an example application we discuss distributed anomaly detection in networks. We demonstrate the applicability of our solution using both synthetic data and real traffic logs collected from the Abilene Internet backbone.
△ Less
Submitted 14 January, 2010; v1 submitted 6 July, 2009;
originally announced July 2009.
-
Fixing Convergence of Gaussian Belief Propagation
Authors:
Jason K. Johnson,
Danny Bickson,
Danny Dolev
Abstract:
Gaussian belief propagation (GaBP) is an iterative message-passing algorithm for inference in Gaussian graphical models. It is known that when GaBP converges it converges to the correct MAP estimate of the Gaussian random vector and simple sufficient conditions for its convergence have been established. In this paper we develop a double-loop algorithm for forcing convergence of GaBP. Our method…
▽ More
Gaussian belief propagation (GaBP) is an iterative message-passing algorithm for inference in Gaussian graphical models. It is known that when GaBP converges it converges to the correct MAP estimate of the Gaussian random vector and simple sufficient conditions for its convergence have been established. In this paper we develop a double-loop algorithm for forcing convergence of GaBP. Our method computes the correct MAP estimate even in cases where standard GaBP would not have converged. We further extend this construction to compute least-squares solutions of over-constrained linear systems. We believe that our construction has numerous applications, since the GaBP algorithm is linked to solution of linear systems of equations, which is a fundamental problem in computer science and engineering. As a case study, we discuss the linear detection problem. We show that using our new construction, we are able to force convergence of Montanari's linear detection algorithm, in cases where it would originally fail. As a consequence, we are able to increase significantly the number of users that can transmit concurrently.
△ Less
Submitted 3 July, 2009; v1 submitted 27 January, 2009;
originally announced January 2009.
-
A Low Density Lattice Decoder via Non-Parametric Belief Propagation
Authors:
Danny Bickson,
Alexander T. Ihler,
Danny Dolev
Abstract:
The recent work of Sommer, Feder and Shalvi presented a new family of codes called low density lattice codes (LDLC) that can be decoded efficiently and approach the capacity of the AWGN channel. A linear time iterative decoding scheme which is based on a message-passing formulation on a factor graph is given.
In the current work we report our theoretical findings regarding the relation between…
▽ More
The recent work of Sommer, Feder and Shalvi presented a new family of codes called low density lattice codes (LDLC) that can be decoded efficiently and approach the capacity of the AWGN channel. A linear time iterative decoding scheme which is based on a message-passing formulation on a factor graph is given.
In the current work we report our theoretical findings regarding the relation between the LDLC decoder and belief propagation. We show that the LDLC decoder is an instance of non-parametric belief propagation and further connect it to the Gaussian belief propagation algorithm. Our new results enable borrowing knowledge from the non-parametric and Gaussian belief propagation domains into the LDLC domain. Specifically, we give more general convergence conditions for convergence of the LDLC decoder (under the same assumptions of the original LDLC convergence analysis). We discuss how to extend the LDLC decoder from Latin square to full rank, non-square matrices. We propose an efficient construction of sparse generator matrix and its matching decoder. We report preliminary experimental results which show our decoder has comparable symbol to error rate compared to the original LDLC decoder.%
△ Less
Submitted 7 October, 2009; v1 submitted 21 January, 2009;
originally announced January 2009.
-
Peer-to-Peer Secure Multi-Party Numerical Computation Facing Malicious Adversaries
Authors:
Danny Bickson,
Tzachy Reinman,
Danny Dolev,
Benny Pinkas
Abstract:
We propose an efficient framework for enabling secure multi-party numerical computations in a Peer-to-Peer network. This problem arises in a range of applications such as collaborative filtering, distributed computation of trust and reputation, monitoring and other tasks, where the computing nodes is expected to preserve the privacy of their inputs while performing a joint computation of a certa…
▽ More
We propose an efficient framework for enabling secure multi-party numerical computations in a Peer-to-Peer network. This problem arises in a range of applications such as collaborative filtering, distributed computation of trust and reputation, monitoring and other tasks, where the computing nodes is expected to preserve the privacy of their inputs while performing a joint computation of a certain function. Although there is a rich literature in the field of distributed systems security concerning secure multi-party computation, in practice it is hard to deploy those methods in very large scale Peer-to-Peer networks. In this work, we try to bridge the gap between theoretical algorithms in the security domain, and a practical Peer-to-Peer deployment.
We consider two security models. The first is the semi-honest model where peers correctly follow the protocol, but try to reveal private information. We provide three possible schemes for secure multi-party numerical computation for this model and identify a single light-weight scheme which outperforms the others. Using extensive simulation results over real Internet topologies, we demonstrate that our scheme is scalable to very large networks, with up to millions of nodes. The second model we consider is the malicious peers model, where peers can behave arbitrarily, deliberately trying to affect the results of the computation as well as compromising the privacy of other peers. For this model we provide a fourth scheme to defend the execution of the computation against the malicious peers. The proposed scheme has a higher complexity relative to the semi-honest model. Overall, we provide the Peer-to-Peer network designer a set of tools to choose from, based on the desired level of security.
△ Less
Submitted 18 January, 2009;
originally announced January 2009.
-
A Hybrid Multicast-Unicast Infrastructure for Efficient Publish-Subscribe in Enterprise Networks
Authors:
Danny Bickson,
Ezra N. Hoch,
Nir Naaman,
Yoav Tock
Abstract:
One of the main challenges in building a large scale publish-subscribe infrastructure in an enterprise network, is to provide the subscribers with the required information, while minimizing the consumed host and network resources. Typically, previous approaches utilize either IP multicast or point-to-point unicast for efficient dissemination of the information.
In this work, we propose a novel…
▽ More
One of the main challenges in building a large scale publish-subscribe infrastructure in an enterprise network, is to provide the subscribers with the required information, while minimizing the consumed host and network resources. Typically, previous approaches utilize either IP multicast or point-to-point unicast for efficient dissemination of the information.
In this work, we propose a novel hybrid framework, which is a combination of both multicast and unicast data dissemination. Our hybrid framework allows us to take the advantages of both multicast and unicast, while avoiding their drawbacks. We investigate several algorithms for computing the best mapping of publishers' transmissions into multicast and unicast transport.
Using extensive simulations, we show that our hybrid framework reduces consumed host and network resources, outperforming traditional solutions. To insure the subscribers interests closely resemble those of real-world settings, our simulations are based on stock market data and on recorded IBM WebShpere subscriptions.
△ Less
Submitted 12 April, 2010; v1 submitted 18 January, 2009;
originally announced January 2009.
-
A Statistical Approach to Performance Monitoring in Soft Real-Time Distributed Systems
Authors:
Danny Bickson,
Gidon Gershinsky,
Ezra N. Hoch,
Konstantin Shagin
Abstract:
Soft real-time applications require timely delivery of messages conforming to the soft real-time constraints. Satisfying such requirements is a complex task both due to the volatile nature of distributed environments, as well as due to numerous domain-specific factors that affect message latency. Prompt detection of the root-cause of excessive message delay allows a distributed system to react a…
▽ More
Soft real-time applications require timely delivery of messages conforming to the soft real-time constraints. Satisfying such requirements is a complex task both due to the volatile nature of distributed environments, as well as due to numerous domain-specific factors that affect message latency. Prompt detection of the root-cause of excessive message delay allows a distributed system to react accordingly. This may significantly improve compliance with the required timeliness constraints.
In this work, we present a novel approach for distributed performance monitoring of soft-real time distributed systems. We propose to employ recent distributed algorithms from the statistical signal processing and learning domains, and to utilize them in a different context of online performance monitoring and root-cause analysis, for pinpointing the reasons for violation of performance requirements. Our approach is general and can be used for monitoring of any distributed system, and is not limited to the soft real-time domain.
We have implemented the proposed framework in TransFab, an IBM prototype of soft real-time messaging fabric. In addition to root-cause analysis, the framework includes facilities to resolve resource allocation problems, such as memory and bandwidth deficiency. The experiments demonstrate that the system can identify and resolve latency problems in a timely fashion.
△ Less
Submitted 18 January, 2009;
originally announced January 2009.
-
Distributed Large Scale Network Utility Maximization
Authors:
Danny Bickson,
Yoav Tock,
Argyris Zymnis,
Stephen Boyd,
Danny Dolev
Abstract:
Recent work by Zymnis et al. proposes an efficient primal-dual interior-point method, using a truncated Newton method, for solving the network utility maximization (NUM) problem. This method has shown superior performance relative to the traditional dual-decomposition approach. Other recent work by Bickson et al. shows how to compute efficiently and distributively the Newton step, which is the m…
▽ More
Recent work by Zymnis et al. proposes an efficient primal-dual interior-point method, using a truncated Newton method, for solving the network utility maximization (NUM) problem. This method has shown superior performance relative to the traditional dual-decomposition approach. Other recent work by Bickson et al. shows how to compute efficiently and distributively the Newton step, which is the main computational bottleneck of the Newton method, utilizing the Gaussian belief propagation algorithm.
In the current work, we combine both approaches to create an efficient distributed algorithm for solving the NUM problem. Unlike the work of Zymnis, which uses a centralized approach, our new algorithm is easily distributed. Using an empirical evaluation we show that our new method outperforms previous approaches, including the truncated Newton method and dual-decomposition methods. As an additional contribution, this is the first work that evaluates the performance of the Gaussian belief propagation algorithm vs. the preconditioned conjugate gradient method, for a large scale problem.
△ Less
Submitted 9 May, 2009; v1 submitted 18 January, 2009;
originally announced January 2009.
-
Self-stabilizing Numerical Iterative Computation
Authors:
Danny Bickson,
Ezra N. Hoch,
Harel Avissar,
Danny Dolev
Abstract:
Many challenging tasks in sensor networks, including sensor calibration, ranking of nodes, monitoring, event region detection, collaborative filtering, collaborative signal processing, {\em etc.}, can be formulated as a problem of solving a linear system of equations. Several recent works propose different distributed algorithms for solving these problems, usually by using linear iterative numer…
▽ More
Many challenging tasks in sensor networks, including sensor calibration, ranking of nodes, monitoring, event region detection, collaborative filtering, collaborative signal processing, {\em etc.}, can be formulated as a problem of solving a linear system of equations. Several recent works propose different distributed algorithms for solving these problems, usually by using linear iterative numerical methods.
The main problem with previous approaches is that once the problem inputs change during the process of computation, the computation may output unexpected results. In real life settings, sensor measurements are subject to varying environmental conditions and to measurement noise.
We present a simple iterative scheme called SS-Iterative for solving systems of linear equations, and examine its properties in the self-stabilizing perspective. We analyze the behavior of the proposed scheme under changing input sequences using two different assumptions on the input: a box bound, and a probabilistic distribution.
As a case study, we discuss the sensor calibration problem and provide simulation results to support the applicability of our approach.
△ Less
Submitted 18 January, 2009;
originally announced January 2009.
-
Self-stabilizing Numerical Iterative Computation
Authors:
Ezra N. Hoch,
Danny Bickson,
Danny Dolev
Abstract:
Many challenging tasks in sensor networks, including sensor calibration, ranking of nodes, monitoring, event region detection, collaborative filtering, collaborative signal processing, {\em etc.}, can be formulated as a problem of solving a linear system of equations. Several recent works propose different distributed algorithms for solving these problems, usually by using linear iterative numer…
▽ More
Many challenging tasks in sensor networks, including sensor calibration, ranking of nodes, monitoring, event region detection, collaborative filtering, collaborative signal processing, {\em etc.}, can be formulated as a problem of solving a linear system of equations. Several recent works propose different distributed algorithms for solving these problems, usually by using linear iterative numerical methods.
In this work, we extend the settings of the above approaches, by adding another dimension to the problem. Specifically, we are interested in {\em self-stabilizing} algorithms, that continuously run and converge to a solution from any initial state. This aspect of the problem is highly important due to the dynamic nature of the network and the frequent changes in the measured environment.
In this paper, we link together algorithms from two different domains. On the one hand, we use the rich linear algebra literature of linear iterative methods for solving systems of linear equations, which are naturally distributed with rapid convergence properties. On the other hand, we are interested in self-stabilizing algorithms, where the input to the computation is constantly changing, and we would like the algorithms to converge from any initial state. We propose a simple novel method called \syncAlg as a self-stabilizing variant of the linear iterative methods. We prove that under mild conditions the self-stabilizing algorithm converges to a desired result. We further extend these results to handle the asynchronous case.
As a case study, we discuss the sensor calibration problem and provide simulation results to support the applicability of our approach.
△ Less
Submitted 19 November, 2008;
originally announced November 2008.
-
Gaussian Belief Propagation: Theory and Aplication
Authors:
Danny Bickson
Abstract:
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the soluti…
▽ More
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the solution algorithm. In the first part of this thesis, we address the properties of the GaBP solver. We characterize the rate of convergence, enhance its message-passing efficiency by introducing a broadcast version, discuss its relation to classical solution methods including numerical examples. We present a new method for forcing the GaBP algorithm to converge to the correct solution for arbitrary column dependent matrices.
In the second part we give five applications to illustrate the applicability of the GaBP algorithm to very large computer networks: Peer-to-Peer rating, linear detection, distributed computation of support vector regression, efficient computation of Kalman filter and distributed linear programming. Using extensive simulations on up to 1,024 CPUs in parallel using IBM Bluegene supercomputer we demonstrate the attractiveness and applicability of the GaBP algorithm, using real network topologies with up to millions of nodes and hundreds of millions of communication links. We further relate to several other algorithms and explore their connection to the GaBP algorithm.
△ Less
Submitted 12 July, 2009; v1 submitted 15 November, 2008;
originally announced November 2008.
-
Gaussian Belief Propagation Solver for Systems of Linear Equations
Authors:
Ori Shental,
Paul H. Siegel,
Jack K. Wolf,
Danny Bickson,
Danny Dolev
Abstract:
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the soluti…
▽ More
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the solution algorithm. We also address some properties of the GaBP solver, including convergence, exactness, its max-product version and relation to classical solution methods. The application example of decorrelation in CDMA is used to demonstrate the faster convergence rate of the proposed solver in comparison to conventional linear-algebraic iterative solution methods.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
Gaussian Belief Propagation Based Multiuser Detection
Authors:
Danny Bickson,
Danny Dolev,
Ori Shental,
Paul H. Siegel,
Jack K. Wolf
Abstract:
In this work, we present a novel construction for solving the linear multiuser detection problem using the Gaussian Belief Propagation algorithm. Our algorithm yields an efficient, iterative and distributed implementation of the MMSE detector. We compare our algorithm's performance to a recent result and show an improved memory consumption, reduced computation steps and a reduction in the number…
▽ More
In this work, we present a novel construction for solving the linear multiuser detection problem using the Gaussian Belief Propagation algorithm. Our algorithm yields an efficient, iterative and distributed implementation of the MMSE detector. We compare our algorithm's performance to a recent result and show an improved memory consumption, reduced computation steps and a reduction in the number of sent messages. We prove that recent work by Montanari et al. is an instance of our general algorithm, providing new convergence results for both algorithms.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
A Gaussian Belief Propagation Solver for Large Scale Support Vector Machines
Authors:
Danny Bickson,
Elad Yom-Tov,
Danny Dolev
Abstract:
Support vector machines (SVMs) are an extremely successful type of classification and regression algorithms. Building an SVM entails solving a constrained convex quadratic programming problem, which is quadratic in the number of training samples. We introduce an efficient parallel implementation of an support vector regression solver, based on the Gaussian Belief Propagation algorithm (GaBP).…
▽ More
Support vector machines (SVMs) are an extremely successful type of classification and regression algorithms. Building an SVM entails solving a constrained convex quadratic programming problem, which is quadratic in the number of training samples. We introduce an efficient parallel implementation of an support vector regression solver, based on the Gaussian Belief Propagation algorithm (GaBP).
In this paper, we demonstrate that methods from the complex system domain could be utilized for performing efficient distributed computation. We compare the proposed algorithm to previously proposed distributed and single-node SVM solvers. Our comparison shows that the proposed algorithm is just as accurate as these solvers, while being significantly faster, especially for large datasets. We demonstrate scalability of the proposed algorithm to up to 1,024 computing nodes and hundreds of thousands of data points using an IBM Blue Gene supercomputer. As far as we know, our work is the largest parallel implementation of belief propagation ever done, demonstrating the applicability of this algorithm for large scale distributed computing systems.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
Polynomial Linear Programming with Gaussian Belief Propagation
Authors:
Danny Bickson,
Yoav Tock,
Ori Shental,
Danny Dolev
Abstract:
Interior-point methods are state-of-the-art algorithms for solving linear programming (LP) problems with polynomial complexity. Specifically, the Karmarkar algorithm typically solves LP problems in time O(n^{3.5}), where $n$ is the number of unknown variables. Karmarkar's celebrated algorithm is known to be an instance of the log-barrier method using the Newton iteration. The main computational…
▽ More
Interior-point methods are state-of-the-art algorithms for solving linear programming (LP) problems with polynomial complexity. Specifically, the Karmarkar algorithm typically solves LP problems in time O(n^{3.5}), where $n$ is the number of unknown variables. Karmarkar's celebrated algorithm is known to be an instance of the log-barrier method using the Newton iteration. The main computational overhead of this method is in inverting the Hessian matrix of the Newton iteration. In this contribution, we propose the application of the Gaussian belief propagation (GaBP) algorithm as part of an efficient and distributed LP solver that exploits the sparse and symmetric structure of the Hessian matrix and avoids the need for direct matrix inversion. This approach shifts the computation from realm of linear algebra to that of probabilistic inference on graphical models, thus applying GaBP as an efficient inference engine. Our construction is general and can be used for any interior-point algorithm which uses the Newton method, including non-linear program solvers.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
Distributed Kalman Filter via Gaussian Belief Propagation
Authors:
Danny Bickson,
Ori Shental,
Danny Dolev
Abstract:
Recent result shows how to compute distributively and efficiently the linear MMSE for the multiuser detection problem, using the Gaussian BP algorithm. In the current work, we extend this construction, and show that operating this algorithm twice on the matching inputs, has several interesting interpretations. First, we show equivalence to computing one iteration of the Kalman filter. Second, we…
▽ More
Recent result shows how to compute distributively and efficiently the linear MMSE for the multiuser detection problem, using the Gaussian BP algorithm. In the current work, we extend this construction, and show that operating this algorithm twice on the matching inputs, has several interesting interpretations. First, we show equivalence to computing one iteration of the Kalman filter. Second, we show that the Kalman filter is a special case of the Gaussian information bottleneck algorithm, when the weight parameter $β= 1$. Third, we discuss the relation to the Affine-scaling interior-point method and show it is a special case of Kalman filter.
Besides of the theoretical interest of this linking estimation, compression/clustering and optimization, we allow a single distributed implementation of those algorithms, which is a highly practical and important task in sensor and mobile ad-hoc networks. Application to numerous problem domains includes collaborative signal processing and distributed allocation of resources in a communication network.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
Peer-to-Peer Secure Multi-Party Numerical Computation
Authors:
Danny Bickson,
Genia Bezman,
Danny Dolev,
Benny Pinkas
Abstract:
We propose an efficient framework for enabling secure multi-party numerical computations in a Peer-to-Peer network. This problem arises in a range of applications such as collaborative filtering, distributed computation of trust and reputation, monitoring and numerous other tasks, where the computing nodes would like to preserve the privacy of their inputs while performing a joint computation of…
▽ More
We propose an efficient framework for enabling secure multi-party numerical computations in a Peer-to-Peer network. This problem arises in a range of applications such as collaborative filtering, distributed computation of trust and reputation, monitoring and numerous other tasks, where the computing nodes would like to preserve the privacy of their inputs while performing a joint computation of a certain function.
Although there is a rich literature in the field of distributed systems security concerning secure multi-party computation, in practice it is hard to deploy those methods in very large scale Peer-to-Peer networks. In this work, we examine several possible approaches and discuss their feasibility. Among the possible approaches, we identify a single approach which is both scalable and theoretically secure.
An additional novel contribution is that we show how to compute the neighborhood based collaborative filtering, a state-of-the-art collaborative filtering algorithm, winner of the Netflix progress prize of the year 2007. Our solution computes this algorithm in a Peer-to-Peer network, using a privacy preserving computation, without loss of accuracy.
Using extensive large scale simulations on top of real Internet topologies, we demonstrate the applicability of our approach. As far as we know, we are the first to implement such a large scale secure multi-party simulation of networks of millions of nodes and hundreds of millions of edges.
△ Less
Submitted 9 October, 2008;
originally announced October 2008.
-
Gaussian Belief Propagation for Solving Systems of Linear Equations: Theory and Application
Authors:
Ori Shental,
Danny Bickson,
Paul H. Siegel,
Jack K. Wolf,
Danny Dolev
Abstract:
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the soluti…
▽ More
The canonical problem of solving a system of linear equations arises in numerous contexts in information theory, communication theory, and related fields. In this contribution, we develop a solution based upon Gaussian belief propagation (GaBP) that does not involve direct matrix inversion. The iterative nature of our approach allows for a distributed message-passing implementation of the solution algorithm. We address the properties of the GaBP solver, including convergence, exactness, computational complexity, message-passing efficiency and its relation to classical solution methods. We use numerical examples and applications, like linear detection, to illustrate these properties through the use of computer simulations. This empirical study demonstrates the attractiveness (e.g., faster convergence rate) of the proposed GaBP solver in comparison to conventional linear-algebraic iterative solution methods.
△ Less
Submitted 7 October, 2008;
originally announced October 2008.
