Blind Separation of Instantaneous Mixtures of Dependent Sources Marc Castella1 and Pierre Comon2 1 GET/INT, UMR-CNRS 5157, 9 rue Charles Fourier, 91011 Évry Cedex, France marc.castella@int-evry.fr 2 CNRS, I3S, UMR 6070, BP.121, Sophia-Antipolis cedex, France pcomon@i3s.unice.fr Abstract. This paper deals with the problem of Blind Source Separation. Contrary to the vast majority of works, we do not assume the statistical independence between the sources and explicitly consider that they are dependent. We introduce three particular models of dependent sources and show that their cumulants have interesting properties. Based on these properties, we investigate the behaviour of classical Blind Source Separation algorithms when applied to these sources: depending on the source vector, the separation may be sucessful or some additionnal indeterminacies can be identified. 1 Introduction Independent Component Analysis (ICA) is now a well recognized concept, which has fruitfully spread out to a wide panel of scientific areas and applications. Contrary to other frameworks where techniques take advantage of a strong information on the diversity, for instance through the knowledge of the array manifold in antenna array processing, the core assumption in ICA is much milder and reduces to the statistical mutual independence between the inputs. However, this assumption is not mandatory in Blind Source Separation (BSS). For instance, in the case of static mixtures, sources can be separated if they are only decorrelated when their nonstationarity or their color can be exploited. Other properties such as the fact that sources belong to a finite alphabet can alternatively be utilized [1,2] and do not require statistical independence. Inspired from [3,4], we investigate the case of dependent sources, without assuming nonstationarity nor color. To our knowledge, only few references have tackled this issue [5,6]. 2 Mixture Model and Notations We consider a set of N source signals (si (n))n∈Z , i = 1, . . . , N . The dependence on time of the signals will not be made explicit in the paper. The sources are mixed, yielding a P -dimensional observation vector x = (x(n))n∈Z according to the model: x = As (1) M.E. Davies et al. (Eds.): ICA 2007, LNCS 4666, pp. 9–16, 2007. c Springer-Verlag Berlin Heidelberg 2007  10 M. Castella and P. Comon where s = (s1 , . . . , sN )T , x = (x1 , . . . , xP )T and A is a P × N matrix called the mixing matrix. We assume that A is left-invertible. Source separation consists in finding a N ×P separating matrix B such that its output y = Bx corresponds to the original sources. When only the observations are used for this, the problem is referred to as the BSS problem. Introducing the N × N global matrix G  BA, the BSS is problem is solved if G is a so-called trivial matrix, i.e. the product of a diagonal matrix with a permutation: these are well known ambiguities of BSS. In this paper, we will study separation criteria as functions of G. Source separation sometimes proceeds iteratively, extracting one source at a time (e.g. deflation approach). In this case, we will write y = bx = gs where b and g = bA respectively correspond to a row of B and G and y denotes the only output of the separating algorithm. In this case, the separation criteria are considered as functions of g. Finally, we denote by E {.} the expectation operator and by Cum {.} the cumulant of a set of random variables. Cum4 {y} is equivalent to Cum {y, y, y, y} and, for complex variables, Cum2,2 {y} stands for Cum {y, y, y ∗ , y ∗ }. 3 Examples and Properties of Dependent Sources We introduce in this section different cases of vector sources that are dependent and that will be considered in this paper. 3.1 Three Dependent Sources Binary phase shift keying (BPSK) signals have specificities that will allow us to obtain source vectors with desired properties. In this paper, we will consider BPSK sources that take values s = +1 or s = −1 with equal probability1/2. We define the following source vector: is real-valued non Gaussian, indepenA1. s  (s1 , s2 , s3 )T where s1 is BPSK;s2  dent of s1 and satisfies E {s2 } = E s32 = 0; and s3 = s1 s2 . Interestingly, the following lemma holds true: Lemma 1. The sources s1 , s2 , s3 defined by A1 are obviously mutually dependent. Nevertheless they are decorrelated and their fourth-order cross-cumulants vanish, that is: Cum {si , sj } = 0 except if i = j, (2) Cum {si , sj , sk , sl } = 0 except if i = j = k = l. (3) Proof. Using the definition of s1 , s2 and their independence, one can easily check that E {s1 } = E {s2 } = E {s3 } = 0. For these centered random variables, it is known that cumulants can be expressed in terms of moments: Cum {si , sj } = E {si sj } Cum {si , sj , sk , sl } = E {si sj sk sl } − E {si sj } E {sk sl } − E {si sk } E {sj sl } − E {si sl } E {sj sk } (4) (5) Blind Separation of Instantaneous Mixtures of Dependent Sources 11 Using again the definition of s1 , s2 and their independence, it is then easy to check all cases of equations (4) and (5) and to verify that these fourth order crosscumulants are indeed null. On the other hand, the third order cross-cumulant reads:       (6) Cum {s1 , s2 , s3 } = E {s1 s2 s3 } = E s21 s22 = E s21 E s22 > 0 and this proves that s1 , s2 , s3 are mutually dependent. ⊓ ⊔ Depending on s2 , more can be proved about the source vector defined by A1. For example, if the probability density function of s2 is symmetric, then s1 and s3 are independent. On the contrary s2 and s3 are generally not independent. An even more specific case is obtained when s2 is itself BPSK. In this case, one can check that the sources (s1 , s2 , s3 ) are pairwise independent, although not mutually independent. 3.2 Pairwise Independent Sources We now investigate further the case of pairwise independent sources and introduce the following source vector: A2. s = (s1 , s2 , s3 , s4 )T where s1 , s2 and s3 are independent BPSK and s4 = s1 s2 s3 . This case has been considered in [3], where it has been shown that ∀i ∈ {1, . . . , 4}, Cum {si , si , si , si } = −2 , Cum {s1 , s2 , s3 , s4 } = 1 (7) and all other cross-cumulants vanish. The latter cumulant value shows that the sources are mutually dependent; although it can be shown that they are pairwise independent. It should be clear that pairwise independence is not equivalent to mutual independence but in an ICA context, it is relevant to recall the following proposition, which is a direct consequence of Darmois’ theorem [7, p.294]: Proposition 1. Let s be a random vector with mutually independent components, and x = Gs. Then the mutual independence of the entries of x is equivalent to their pairwise independence. Based on this proposition, the ICA algorithm in [7] searches for an output vector with pairwise independent component. Let us stress that this holds only if the source vector has mutually independent components: pairwise independence is indeed not sufficient to ensure identifiability as we will see in Section 4.2. 3.3 Complex Valued Sources We consider quaternary phase shift keying (QPSK) sources which take their π 5π 5π π values in {eı 4 , e−ı 4 , eı 4 , e−ı 4 } with equal probability 1/4. We then define the following source vector: A3. s = (s1 , s2 , s3 , s4 )T where s1 , s2 and s3 are mutually independent QPSK and s4 = s1 s2 s3 . 12 M. Castella and P. Comon Based on the Equations (4) and (5) which hold for the above centered sources, one can check the following proposition: Lemma 2. The sources in A3 are dependent and Cum {s1 , s2 , s3 , s∗4 } = 1. However, they are second-order decorrelated and all their fourth order circular crosscumulants (i.e. with as many conjugates as non-conjugates) vanish, that is:   (8) Cum si , s∗j = 0 and Cum {si , sj } = 0 except if i = j, Cum {si , sj , s∗k , s∗l } = 0 except if i = j = k = l. (9) Actually, we can prove that the above Lemma, as well as Lemma 4 and Proposition 6 still hold in the case when s1 , s2 and s3 are second order circular and have unit modulus: this is not detailed for reasons of space and clarity. 4 4.1 ICA Algorithms and Dependent Sources Independence Is Not Necessarily Required The sources given by A1 provide us with a specific example of dependent sources that are sucessfully separated by several ICA methods: Proposition 2. Let y = gs where the vector of sources is defined by A1. Then, the function (10) g → |Cum4 {y}|α , α ≥ 1 defines a MISO contrast function, that is, its maximization over the set of unit norm vectors ( g 2 = 1) leads to a vector g with only one non-zero component. Proof. The above proposition follows straightforwardly from Lemma 1 since the proof of the validity of the above contrast functions only relies on the property in Equation (3). ⊓ ⊔ Considering again the argument in the proof, one should easily notice that the above proposition can be generalized to the case of sources which satisfy: A4. s = (s1 , . . . , s3K ) where: ∀i ∈ {0, . . . ,K − 1},  s3i+1 is BPSK; s3i+2 is non Gaussian and satisfies E {s3i+2 } = E s33i+2 = 0; s3i+3 = s3i+1 s3i+2 ; and the random variables {s3i+1 , s3i+2 ; i = 0, . . . , K − 1} are mutually independent. In addition, the above result can be generalized convolutive systems and to MIMO (multiple input/multiple output) contrast functions as defined in [7,2]: Proposition 3. Let y = Gs where the vector of sources is defined by A1. Then the function: N  |Cum4 {yi }|α , α ≥ 1 (11) G → i=1 is a MIMO contrast, that is, its maximization over the group of orthogonal matrices leads to a solution G which is a trivial matrix (permutation, scaling). Blind Separation of Instantaneous Mixtures of Dependent Sources 13 Many classical algorithms for BSS or ICA first whiten the data: it is known that in so doing, they constrain matrix G to be orthogonal. In particular so does the algorithm proposed in [7], which relies on the contrast function in (11). It justifies that this algorithm successfully separates the sources A1. Actually, any algorithm relying on a prewhitening and associated with a contrast function based on the vanishing of the fourth-order cross cumulants (e.g. JADE) is able to separate sources such as A1. 4.2 Pairwise Independence Is Not Sufficient We now consider the pairwise independent sources given by A2 and show that pairwise independence is not sufficient to ensure identifiability of the ICA model. We first have the following preliminary result: Lemma 3. Let y = gs where the vector of sources is defined by A2. Assume that the vector (s1 , s2 , s3 ) takes all 23 possible values. If the signal y has values in {−1, +1}, then g = (g1 , g2 , g3 , g4 ) is either one of the solutions below:  ∃i ∈ {1, . . . , 4} gi = ±1, and: ∀j = i, gj = 0 (12) ∃i ∈ {1, . . . , 4} gi = ±1/2, and: ∀j = i, gj = −gi Proof. If y = gs, using the fact that s2i = 1 for i = 1, . . . , 4, we have with the particular sources given by A2:       y 2 = g12 +g22 +g32 +g42 +2 g1 g2 +g3 g4 s1 s2 + g1 g3 +g2 g4 s1 s3 + g2 g3 +g1 g4 s2 s3 Since (s1 , s2 , s3 ) take all possible values in {−1, 1}3, we deduce from y 2 = 1 that the following equations necessarily hold:  g12 + g22 + g32 + g42 = 1 (13) g1 g2 + g3 g4 = g1 g3 + g2 g4 = g2 g3 + g1 g4 = 0 First observe that values given in (12) indeed satisfy (13). Yet, if a polynomial system of N equations of degree d in N variables admits a finite number of solutions1 , then there can be at most dN distinct solutions. Hence we have found ⊔ them all in (12), since (12) provides us with 16 solutions for (g1 , g2 , g3 , g4 ). ⊓ Using the above result, we are now able to specify the output of classical ICA algorithms when applied to a mixture of sources which satisfy A2. Constant modulus and contrasts based on fourth order cumulants. The constant modulus (CM) criterion is one of the most known criteria for BSS. In the real valued case, it simplifies to: JCM (g)  E 1  2 2 y −1 with: y = gs One can show that the number of solutions of (13) is indeed finite. (14) 14 M. Castella and P. Comon Proposition 4. For the sources given by A2, the minimization of the constant modulus criterion with respect to g leads to either one of the solutions given by Equation (12). Proof. We know that the minimum value of the constant modulus criterion is zero and that this value can be reached (for g having one entry being ±1 and other entries zero). Moreover, the vanishing of the constant modulus criterion implies that y 2 − 1 = 0 almost surely and one can then apply Lemma 3. ⊓ ⊔ A connection can now be established with the fourth-order autocumulant if we impose the following constraint:   (15) E y 2 = 1 (or equivalently g = 1 since y = gs) Because of the scaling ambiguity of BSS, the above normalization can be freely  2 imposed. Under (15), we have Cum4 {y} = E y 2 − 1 − 2 and minimizing JCM (g) thus amounts to maximizing −Cum4 {y}. Unfortunately, since Cum4 {y} may be positive or negative, no simple relation between |Cum4 {y}| and JCM (g) can be deduced from the above equation. However, we can state: Proposition 5. Let y = gs where the vector of sources is defined by A2. Then, under the constraint (15) ( g = 1), we have: (i) The maximization of g → −Cum4 {y} leads to either one of the solutions given by Equation (12). (ii) |Cum4 {y}| ≤ 2 and the equality |Cum4 {y}| = 2 holds true if and only if g is one of the solutions given in Equation (12). Proof. Part (i) follows from the arguments given above. In addition, using multilinearity of the cumulants and (7), we have for y = gs:   Cum4 {y} = −2 g14 + g24 + g34 + g44 + 24 (g1 g2 g3 g4 ) (16) The result then follows straightfowardly from the study of the polynomial function in Equation (16). Indeed, optimizing (16) leads to the following Lagrangian:  4 4 4   2 4 gi − 1 (17) gi − λ gi + 24 L = −2 i=1 i=1 i=1 After solving the polynomial system which cancels the Jacobian of the above expresssion, one can check that all solutions are such that |Cum4 {y}| ≤ 2. Details are omitted for reasons of space. Part (ii) of the proposition easily follows. ⊓ ⊔ Similarly to the previous section, the above proposition can be generalized to MIMO contrast functions. In particular, this explains why, for a particular set of mixing matrices such as that studied in [3], the pairwise maximization algorithm of [7] still succeeded: a separation has luckily been obtained for the considered mixing matrices and initialization point of the algorithm, but it actually would not succeed in separating BPSK dependent sources for general mixing matrices. Blind Separation of Instantaneous Mixtures of Dependent Sources 15 Let us stress also that the results in this section are specific to the contrast functions given by (10) or (11). In particular, these results do no apply to algorithms based on other contrast functions such as JADE, contrary to the results in Sections 4.1 and 4.3. 4.3 Complex Case The output given by separation algorithms in case of complex valued signals may differ from the previous results which have been proved for real valued signals only. Indeed, complex valued BSS does not always sum up to an obvious generalization of the real valued case [8]. We illustrate it in our context and show that, quite surprisingly, blind separation of the sources given by A3 can be achieved up to classical inderterminations of ICA. This is in contrast with the result in Equation (12) where additionnal indeterminacies appeared. First, we have: Lemma 4. Let y = gs where the vector of sources is defined by A3. Assume that the vector (s1 , s2 , s3 ) takes all 43 possible values. If the signal y is such that its values satisfy |y|2 = 1, then g = (g1 , g2 , g3 , g4 ) satisfies: ∃i ∈ {1, . . . , 4} |gi | = 1, and: ∀j = i, gj = 0 (18) Proof. If y = gs, using the fact that |si |2 = 1 for i = 1, . . . , 4, we have with the particular sources given by A3: |y|2 = 4  i=1 |gi |2 +  gi gj∗ si s∗j (19) i=j Since (s1 , s2 , s3 ) take all possible values in {1, ı, −1, −ı}3, we deduce from |y|2 = 1 that the following equations necessarily hold:  |g1 |2 + |g2 |2 + |g3 |2 + |g4 |2 = 1 (20) g1 g2∗ = g1 g3∗ = g1 g4∗ = g2 g3∗ = g2 g4 ∗ = g3 g4∗ = 0 Solving for the polynomial system in the variables |g1 |, |g2 |, |g3 | and |g4 |, we obtain that the solutions are the ones given in Equation (18). ⊓ ⊔ Constant modulus and fourth-order cumulant based contrasts. In contrast with Propositions 4 and 5 we have the following result: Proposition 6. Let y = gs where the sources satisfy A3. Then, the functions: 2  and: (21) g → −E |y|2 − 1  2 (22) g → |Cum2,2 {y}| under constraint E |y| = 1 are contrast functions, that is, their maximization leads to g satisfying (18). Proof. The validity of the first contrast function is obtained with the same argum.s. ments as in the proof of Proposition 4: we have |y|2 = 0, which yields (20) via 16 M. Castella and P. Comon (19). In the case of independent sources, the proof of the validity of the second contrast involves only cumulants with equal number of conjugate and non conjugate variables: invoking Lemma 2, one can see that the same proof still holds here. ⊓ ⊔ Note that the same arguments can be applied to ICA methods such as the pairwise algorithm in [2] or JADE [9]. Figure 1 illustrates our result. 5 0 −5 −5 0 5 5 0 −5 −10 0 10 10 1 1 5 0.5 0.5 0 0 0 −5 −0.5 −0.5 −10 −10 0 10 −1 −1 0 1 −1 −1 4 1 1 2 0.5 0.5 0 0 0 −2 −0.5 −0.5 −4 −5 0 5 −1 −1 0 1 −1 −1 0 1 0 1 Fig. 1. Typical observed separation result of the sources A3 with the algorithm JADE (left: sensors, right: separation result) References 1. Li, T.H.: Finite-alphabet information and multivariate blind deconvolution and identification of linear systems. IEEE Trans. on Information Theory 49(1), 330–337 (2003) 2. Comon, P.: Contrasts, independent component analysis, and blind deconvolution. Int. Journal Adapt. Control Sig. Proc. 18(3) 225–243 (April 2004) special issue on Signal Separation: Preprint: I3S Research Report RR-2003-06 (2004), http://www3.interscience.wiley.com/cgi-bin/jhome/4508 3. Comon, P.: Blind identification and source separation in 2 × 3 under-determined mixtures. IEEE Trans. Signal Processing 52(1), 11–22 (2004) 4. Comon, P., Grellier, O.: Non-linear inversion of underdetermined mixtures. In: Proc. of ICA’99, Aussois, France, pp. 461–465 (January 1999) 5. Hyvärinen, A., Shimizu, S.: A quasi-stochastic gradient algorithm for variancedependent component analysis. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4131, pp. 211–220. Springer, Heidelberg (2006) 6. Cardoso, J.F.: Multidimensional independent component analysis. In: Proc. ICASSP ’98. Seattle (1998) 7. Comon, P.: Independent component analysis, a new concept. Signal Processing 36(3), 287–314 (1994) 8. Eriksson, J., Koivunen, V.: Complex random vectors and ICA models: Identifiability, uniqueness and separability. IEEE Trans. on Information Theory 52(3), 1017–1029 (2006) 9. Cardoso, J.F., Souloumiac, A.: Blind beamforming for non gaussian signals. In: IEEE- Proceedings-F. vol. 140, pp. 362–370 (1993)