ISSN 1995-0802, Lobachevskii Journal of Mathematics, 2022, Vol. 43, No. 4, pp. 882–894.


c Pleiades Publishing, Ltd., 2022.

Fast Nonnegative Tensor Factorizations with Tensor Train Model

E. M. Shcherbakova1* and E. E. Tyrtyshnikov2**

(Submitted by V. V. Voevodin)
1
Lomonosov Moscow State University, Moscow, 119991 Russia
2
Marchuk Institute of Numerical Mathematics of Russian Academy of Sciences, Moscow, 119333 Russia
Received February 24, 2022; revised March 13, 2022; accepted March 16, 2022

Abstract—Tensor train model is a low-rank approximation for multidimensional data. In this article
we demonstrate how it can be succesfully used for fast computation of nonnegative tensor train,
nonnegative canonical and nonnegative Tucker factorizations. The proposed approaches can be
incorporated in wide range of methods to solve big data problems.

DOI: 10.1134/S1995080222070228
Keywords and phrases: nonnegative tensor factorization, tensor train, Tucker decomposition,
canonical decomposition.

1. INTRODUCTION
In time of exponential data growth higher-order generalisations of matrices, referred as tensors,
are successfully applied to many different fields. However as the number of dimensions increases and
reaches several tens, it becomes close to impossible to handle the computations due to the “curse of
dimensionality”. To solve this problem tensor decompositions are used.
Canonical Polyadic Decomposition (CPD/CANDECOMP) is one of the most famous methods for
tensor factorization. This decomposition was first studied by Hitchcock in 1927 [1], and was later known
as parallel factor analysis (PARAFAC), a tool for chemometric analysis popularized by Harshman [2],
Carroll and Chang [3], and Kruskal [4]. The CP representation isn’t only interpretable, dimensionality
reduction is better than that of other factorizations. However despite these attractive properties it is still
susceptible to the “curse of dimensionality”.
Not long ago, it was considered that the cheapest in computations algorithm for CPD using

alternating least squares updates for a tensor of size n1 × . . . × nd has O(dR di=1 ni ) complexity, where
R is a rank of the decomposition. But recently method based on prior tensor train decomposition was
introduced which allows to significantly reduce time for calculations [6].
However in practice we often have to deal with nonnegative multidimensional data. Then it is natural
to wish for tensor decomposition with the same property. And thus nonnegative CPD (NCPD) model
was introduced [5]. Just like the model without constraints standard algorithms suffer from the “curse of
dimensionality”. The logical idea is to try using tensor train decomposition, as it was done in [6]. To the
best of our knowledge, there was only one attempt, and it was unsuccesful because of the high relative
error [9].
In this article we suggest extremely useful approach for speeding up existing NCPD algorithms,
reducing initial complexity of O(dRN d ), N = maxi (ni ) to just O(dN R3 ).
Moreover the same ideas can be applied for constructing Nonnegative Tucker Decomposition (NTD).
The Tucker decomposition model assumes that an input multiway array is decomposed into a core
tensor and a set of low-rank matrices, which allows to capture the multilinear features associated
*
E-mail: lena19592@mail.ru
**
E-mail: eugene.tyrtyshnikov@gmail.com

882
 FAST NONNEGATIVE TENSOR FACTORIZATIONS 883

with all the modes of the input tensor. The baseline model was proposed by L. R. Tucker in 1966
[22] as a multilinear extension to principal component analysis (PCA). For dealing with nonnegative
data NTD was proposed, where nonnegativity constraints are imposed on the core tensor and all the
factor matrices. This model has been used in multiple applications, including image classification [23],
audio pattern extraction [24], clustering [25], etc. In Section 4 we discuss how to construct NTD
approximation faster using tensor train.
As most proposed methods are based on the nonnegative tensor train (TT) factorization, it is
important to develop effective methods for it’s construction. By definition nonnegative TT (NTT)
requires elements of each core to be nonnegative. It is done for interpretability of the results and so that
all elements of the approximation are guaranteed to be nonnegative. But often in practice this constraint
may be excessive. In Section 2 we suggest novel method for building NTT.
In Section 5 some numerical experiments are discussed.

2. NONNEGATIVE TENSOR TRAIN

Tensor train (TT) decomposition is used for compact representation and approximation of high-
dimensional tensors. The nonnegative TT decomposition is written as a tensor train of the form

y(i1 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 ) . . . Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id ),
α1 ,...,αd−1

with tensor carriages G1 , ..., Gd , where any two neighbors have a common summation index. The
summation indices αk run from 1 to rk and are called auxiliary indices. The quantities rk are referred to
r ×n ×r
as tensor train (TT) ranks. The tensor carriages G2 , . . . , Gd−1 ∈ R+k−1 k k . It is sometimes easier
to assume that G1 , Gd are not two-dimensional but three-dimensional with additional auxiliary indices
α0 = αd = 1 and tensor train ranks r0 = rd = 1. This assumption helps to simplify some algorithms.
For nonnegative tensor decompositions it is standard to require core matrices or tensors to have only
nonnegative elements. There are two main reasons. Based on intuition, the parts of a dataset represented
by the factors are thought to in general case combine additively. And in many applications the quantities
involved should be nonnegative otherwise it will be difficult to interpret the results of a decomposition.
Another reason is much more practical: after imposing such constraint we are guaranteed to get
nonnegative approximation.
However there are cases when nonnegativity of factors isn’t necessary as we are only interested in
the elements of the resulting approximation. Tensor decomposition is an instrument allowing to operate
with large amounts of data efficiently. Many algorithms can use such representations in order to speed
up calculations. And there are examples when nonnegativity of the tensor decomposition is crucial for
the method convergence. So when the main agenda is to get all the pros of tensor factorization such as
lower memory cost and faster computations and simultaneously keep the property of the original tensor
we can consider the following problem.
Given nonnegative tensor build the nonnegative TT decomposition

y(i1 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 ) . . . Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id ) ≥ 0,
α1 ,...,αd−1

where tensor carriages G1 , . . . , Gd ∈ Rrk−1 ×nk ×rk . Because we discard constraints for the factor
nonnegativity we can expect there to be an algorithm with lesser complexity or higher accuracy.
In this article we propose new approach to nonnegative tensor train factorization. The idea follows
from the simple rationale: when we construct tensor approximation with prescribed accuracy, we expect
it’s elements to be close to the original, and if initial data is nonnegative it is natural to assume that the
majority of the tensor train model elements would also keep this property. Then when the number of
elements turned negative is small, we can find and correct them.
So first of all we need a method to find negative elements in tensor train approximation. We will
start with looking for the element with maximum absolute value. Of course item-by-item examination
is unacceptable as it has O(N d ) complexity, where N = maxi (ni ). In case of a diagonal matrix A it’s
maximum by module value is simultaneously it’s largest (in absolute value) eigenvalue. One of the most
well known and simple methods to solve this task is power method. In the algorithm we start with

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

884 SHCHERBAKOVA, TYRTYSHNIKOV
M rk
random vector r0 . And then we use iterative formula rk+1 = ||M rk || which under some assumptions
should converge to an eigenvector associated with the dominant eigenvalue. But our aim is to find
minimum value. And we can do it by using power method on matrix A − λmax I, where λmax is largest
in magnitude eigenvalue and I is an identity matrix.
Now back to tensors. There are several approaches to solving eigenvalue problem in TT. They
include iterative methods with additional rank rounding [7], Riemann optimization and alternating linear
scheme. Let us explain how it is possible to find negative elements in TT approximation with classic
eigenvalue method. We consider our tensor train approximation Y as diagonal of the diagonal matrix.
M rk
With this rk+1 = ||M rk || computation speeds up greatly. We will discuss each step in detail and estimate
the resulting complexity.
First of all we will use the following fact:
Ba = b  a, B ∈ Rm×m , a ∈ Rm , b ∈ Rm ,
where  denotes Hadamard product and B is a diagonal matrix with diagonal b. And if we have vectors
b and a in TT format then we can execute elementwise multiplication quite fast.
Hadamard product C of two tensor trains b and a can be presented as
C (i1 , . . . , id ) = A1 (i1 ) . . . Ad (id ) B1 (i1 ) . . . Bd (id )
= (A1 (i1 ) . . . Ad (id )) ⊗ (B1 (i1 ) . . . Bd (id ))
= (A1 (i1 ) ⊗ B1 (i1 )) (A2 (i2 ) ⊗ B2 (i2 )) . . . (Ad (id ) ⊗ Bd (id )) .
Symbol ⊗ denotes Kronecker product. This means that the cores of C are
Ck (ik ) = Ak (ik ) ⊗ Bk (ik ), k = 1, d,

where Bk (ik ) and Ak (ik ) are matrices of size rk−1 × rk and rk−1  × rk respectively. Thus overall the
complexity is equal to O(dN r 4 ).
But ranks of the C cores are rk−1 
rk−1 and rk rk , k = 0, . . . , d. It means that with each iteration
computation cost will be increasing. To avoid rank growth one has to reduce ranks while maintaining
accuracy, and it is possible with TT-rounding algorithm [8]. Because ranks are r 2 it has O(dN r 6 )
complexity. Next we have to compute tensor train norm, the cost for this operation is O(dN r 3 ) [8]. The
remaining step—division by a number—is trivial; we just scale one of cores by it. Below is presented
pseudocode for power method.

Algorithm 1: Pseudocode for power method in TT format.

Data: Tensor Y , number of iterations niters
Result: Vector xniters in TT format
1 Set x1 as TT of all ones with the same size as Y .
2 for i = 1 : niters do
3 xi+1 = Y  xi
4 TT-round(xi+1 )
5 Normalize xi+1
6 end

If tensor has only one dominant eigenvalue then in exact arithmetics method should converge to the
tensor with ranks 1. One can use the Rayleigh quotient in order to get the associated eigenvalue.
It has to be noted that besides eigenvalue algorithms there are other approaches to search for
minimum element in tensor train. For example, in [12] there is a routine which computes statistics such

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

 FAST NONNEGATIVE TENSOR FACTORIZATIONS 885

as largest/smallest in magnitude entries or entries having largest/smallest real parts of a TT tensor via
maxvol cross. It has proved to be very competing with power method during numerical experiments.
In the future we plan to test other eigenvalue algorithms (for example inverse power method, Rayleigh
quotient iteration, etc.) and compare their performance. Of course, each method has it’s own pros ad
cons. For instance, inverse iteration demonstrates fast convergence, but only if the optimal shifts are
chosen. And for each iteration we have to solve linear system.
But finding the minimum entry y(k1 , · · · , kd ) of TT is only part of the solution. Our main goal
is to get tensor train approximation with only nonnegative values. Of course, if minimum is bigger
than zero then we don’t have to act any further as we already have our answer. But usually it isn’t
the case. When we know which element is negative we can forcefully amount it to the desired value
by adding rank one tensor. This tensor C will have all zero elements except one: c(k1 , · · · , kd ) =
y(k1 , · · · , kd ) − y(k1 , · · · , kd ). After element correction we can repeat the process: find minimum value
of the fixed TT and correct it if it is negative. And in the end we will construct TT with only nonnegative
values. However this approach has very serious flaw: for sum of two TT their ranks are also summed.
So for each iteration the ranks will grow by one. And unlike for power method, in this case we can’t
employ TT rounding, because it doesn’t keep nonnegativity of the tensor. To avoid the uncontrolled
growth of ranks we suggest first to add some small constant to all elements of the TT approximation.
It’s value should be chosen depending on the minimum value. It is logical to expect negative elements of
the approximation for nonnegative tensor to be close to zero. Then adding constant tensor will not only
allow us to turn most elements positive but also won’t strongly affect approximation accuracy. After this
step we can resume individual element correction.
Let us summarise proposed steps for building tensor train approximation with nonnegative entries.

• Build tensor train approximation Y for the original tensor Y . It can be found using, for example,
TT-SVD or TT-CROSS [13]. Fortunately, the approximation should be computed only once.

• Find minimum entry of Y . There are different approaches for this step. For example, we can find
maximum in magnitude value with power method, subtract it from TT and repeat the search for
this new tensor train with only negative elements.

• To the Y we add small constant, which value depends on the found minimum. With this we
correct most of the negative values in approximation. Ranks are increased by one.

• We continue to search for remaining negative elements, determine their indices and then build TT
with ranks equal 1 to add to the approximation.

Of course there remains much scope for future research. We are working on analizing negative
elements distribution in order to define more strict rule for determining the constant from third step.
Also it is interesting to compare different methods for minimum value search in TT. But the results are
looking very promising for experiments as we demonstrate in Section 5.

3. FAST NONNEGATIVE CPD USING TENSOR TRAIN

A Canonical Polyadic Decomposition for tensor Y ∈ Rn1 ×n2 ×···×nd is composed of factor matrices
U (i) = [u1 . . . uR ] ∈ Rni ×R and is defined as
(i) (i)


R
y(i1 , . . . , id ) = u1r (i1 )u2r (i2 ) . . . udr (id ),
r=1

where R is a rank of the decomposition.

As we have already said in introduction Canonical Polyadic Decomposition (CPD) is very popular
and has been used for many applications, for example signal processing and machine learning tasks such
as image completion, feature extraction, etc. This model and it’s properties, including uniqueness, are
well studied. However despite all these pros there is a serious con: computational costs of most existing

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

886 SHCHERBAKOVA, TYRTYSHNIKOV

algorithms for CPD increase exponentially with the tensor order. But there is a way to significantly
decrease complexity from O(dRN d ) to just O(dN R3 ) in case n1 = . . . = nd = N .
In [6] authors suggested to compress initial tensor to TT prior to CPD. For noiseless tensors, an
exact mapping from the core tensors of a TT-representation of a given data tensor to the factor matrices
of its CPD was proposed and for the noisy case, iterative algorithm was developed for estimation of factor
matrices, with a cost of O(dN R3 ).
Now let us consider CPD with nonnegative constraints. A Nonnegative Canonical Polyadic De-
composition (NCPD) for tensor Y ∈ Rn+1 ×n2 ×···×nd is composed of factor matrices U (i) = [u1 . . . uR ]
(i) (i)

ni ×R
∈ R+ . And by analogy with CPD well known algorithms have the same flaw—O(dRN d ) complexity.
In this case it seems appropriate to use the same technique to speed up methods.
And indeed in 2020 the group of researchers tried to implement this idea and represented results
as report for the conference [9]. Their algorithm included three steps: dimensionality reduction by
nonnegative tensor train, factorization of the NTT core tensors by low rank nonnegative CPD and
reconstruction of the final NCPD of the initial tensor. But this method was unsuccessful because of
the high approximation error even for noiseless artificial data.
We suggest different from [9] approach for NCPD representation construction with a cost of only
O(dN R3 ) and desired accuracy. Most algorithms for CPD and NCPD use initial tensor unfoldings, and
operations with them (usually matrix multiplication) give us final complexity of O(dRN d ).
For example we will consider FAST-HALS NTF method from [10]. In the pseudocode for
Algorithm 2 the elementwise division, Kronecker, Khatri-Rao (columnwise Kronecker), Hadamard and
outer products are denoted, respectively, by , ⊗, , , ◦. Operation [· ]+ changes the negative elements
of its argument to some very small nonnegative constant ρ.
For the faster version of the algorithm we will need the tensor-by-matrix multiplication referred to
as the mode-k multiplication. Given a tensor A = [A (i1 , i2 , . . . , id )] and a matrix U = [U (α, ik )], we
define the mode-k multiplication result as a tensor B = [B (i1 , . . . , α, . . . , id )] (α is on the kth place)
obtained by the contraction over the kth axis:

nk
B (i1 , . . . , α, . . . , id ) = A (i1 , i2 , . . . , id ) U (α, ik ) .
ik =1

We denote this operation as follows:

B = A ×k U.
Other basic tensor operations and notations are presented in Table 1.
Although the Algorithm 2 is called FAST HALS, there is a major bottleneck for swift computations
in Y(n) {U −n } operation, because size of Y(n) is N × N d−1 .
To tackle this problem we will use the property of the Khatri–Rao and Kronecker products of two
vectors
   
(1) (2) (1) (2) (1) (2)
U (1)  U (2) = u1 ⊗ u1 . . . uR ⊗ uR = uj  uj .
j j

Suppose we have initial tensor Y in tensor train form


y(i1 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 ) . . . Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id ).
α1 ,...,αd−1

Then elements of matrix T2 = Y(k) {U −k } ∈ Rnk ×R can be written as

 (1) (k−1) (k+1) (d)
T2 (s, j) = y(i1 , . . . , s, . . . , id )uj (i1 ) . . . uj (ik−1 )uj (ik+1 ) . . . uj (id )
i1 ,...,ik−1 ,ik+1 ,...,id
 (1) (k−1) (k+1) (d)
= G1 (i1 ) . . . Gd (id )uj (i1 ) . . . uj (ik−1 )uj (ik+1 ) . . . uj (id ),
i1 ,...,ik−1 ,ik+1 ,...,id

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

 FAST NONNEGATIVE TENSOR FACTORIZATIONS 887

where Gk (ik ) is an rk−1 × rk matrix.

 (1)
 (d)
T2 (s, j) = G1 (i1 )uj (i1 ) . . . Gk (s) · · · Gd (id )uj (id ),
i1 id

with s = 1, nk , j = 1, R.
(m)  (m)
Introduce matrices Γj = im Gm (im )uj (im ). So for each j we need to compute d − 1 tensor by
(1) (d)
vector multiplication over a given mode. Then calculation of Γj . . . Gk (s) . . . Γj for all s and j requires
O(dN r 2 R) operations. So final complexity in case when tensor ranks are less or equal NCPD ranks is
O(dN R3 ).
Then the Algorithm 2 can be rewritten. Implementations of the necessary operations with TT are
available, for example, in [12].
Naive approach of calculating T2 from scratch on each iteration will result in O(d2 N R3 ) complexity.
The smarter way is to evaluate common parts of computations beforehand and then only update them.
Then we can keep overall complexity of iteration at O(dN R3 ) with additional memory of only O(dR3 ).

Algorithm 2: Pseudocode for FAST-HALS NTF from [10].

Data: Tensor Y (i1 , . . . , id ), ρ ≥ 0, nonnegative CPD rank R.
ni ×R
Result: factor matrices U (i) = [ui1 . . . uiR ] ∈ R+ .
1 Nonnegative random or nonnegative ALS initialization U (n) .
(i)
2 Normalize all uj for i = 1, . . . , d − 1 to unit length.
 
3 T1 = U (1)T U (1)  . . .  U (d)T U (d)
4 while TRUE do

5 γ = diag U (d)T U (d)
6 for i = 1:d do
7 if i = d then
8 γ=1
9 end
10 T2 = Y(i) {U −i }

11 T3 = T1  U (i)T U (i)
12 for j = 1:R do
 
(i) (i)
13 uj = γj uj + [T2 ]j − U (i) [T3 ]j
+
14 if i = d then
(i) (i) (i)
15 uj = uj / uj
2
16 end
17 end
18 T1 = T3  U (i)T U (i)
19 end
20 if stopping criterion is met then
21 break;
22 end
23 end

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

888 SHCHERBAKOVA, TYRTYSHNIKOV

Table 1. Basic tensor operations and notations

[U ]j jth column vector of [U ]

(n)
uj jth column vector of U n
U −n U (d)  · · ·  U (n+1)  U (n−1)  · · ·  U (1)
U ⊗−n U (d) ⊗ · · · ⊗ U (n+1) ⊗ U (n−1) ⊗ · · · ⊗ U (1)
Y(n) n-Mode matricized version of Y

So if we have tensor train representation of the initial tensor we are able to dramatically speed up
NCPD algorithm. Let us notice additional useful property of the Algorithm 3: even if TT approximation
has negative elements, the method will construct nonnegative CPD. And in this case we don’t need
nonnegative tensor train factorization. TT approximation Y for the original tensor can be found using,
for example, TT-SVD or TT-CROSS [13].
Matlab package for Tensor decompositions “TensorBox” [14] includes different methods for NCPD:
ANLS with Active Set Method and Column Grouping, Multiplicative Updating Method, ANLS with
Block Principal Pivoting Method. All of them have the same time consuming operation Y(n) {U −n }.
Using approach described above it is possible to significantly accelerate all these NCPD algorithms.
But there are methods for which we need to build nonnegative tensor train prior. As a way to do so we
can use technique described in Section 2, because in this case we only need nonnnegative entries for
the approximation, not for all the factors. Methods for constructing NTT with nonnegative cores can be
found in [11, 16, 18].

4. FAST NONNEGATIVE TUCKER DECOMPOSITION USING TENSOR TRAIN

The higher-order tensor Tucker decomposition is described as a decomposition of a given tensor Y ∈
Rn1 ×n2 ×···×nd
into an unknown core tensor G ∈ RR1 ×R2 ···×Rd multiplied by a set of d unknown com-
ponent matrices, A(i) = a1 , a2 , . . . , aRi ∈ Rni ×Ri (i = 1, 2, . . . , d), representing common factors or
(i) (i) (i)

loadings:

y(i1 , . . . , id ) = g(j1 , j2 , . . . , jd )a1j1 (i1 )a2j2 (i2 ) . . . adjd (id ).
j1 ,j2 ,...,jd

1 ×R2 ···×Rd
Classical nonnegative Tucker factorization (NTD) is supposed to have G ∈ RR
+ and A(i) =
a1 , a2 , . . . , aRi ∈ Rn+i ×Ri (i = 1, 2, . . . , d).
(i) (i) (i)

To construct nonnegative Tucker approximation Y we need to minimize the following cost function:
1
DNTD = ||Y − Y ||2F .
2
To optimize A(n) we consider the mode-n matricization of Y and Y and rewrite functional:
1 2
DNTD = Y(n) − A(n) G(n) (A⊗−n ) , A(n) ≥ 0.
2 F

To optimize G, considering the vectorization of Y and Y , function DNTD is reformulated:

1
DNTD = vec(Y ) − (A1 ⊗ · · · ⊗ Ad ) vec(G) 2F , G ≥ 0.
2
To extend methods with alternating least squares scheme to NTD, we need to compute the following
respective gradients of DNTD with respect to A(n) and G:
∂DNTD
= A(n) B (n) B (n) − Y(n) B (n) ,
∂A(n)

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

 FAST NONNEGATIVE TENSOR FACTORIZATIONS 889

Algorithm 3. Pseudocode for FAST-HALS NTF based on tensor train.

Data: Tensor train Y (i1 , . . . , id ), ρ ≥ 0, nonnegative CPD rank R.
ni ×R
Result: factor matrices U (i) = [ui1 . . . uiR ] ∈ R+ .
1 Nonnegative random or nonnegative ALS initialization U (n) .
(i)
2 Normalize all uj for i = 1, . . . , d − 1 to unit length.
 
3 T1 = U (1)T U (1)  . . .  U (d)T U (d)
4 while TRUE do

5 γ = diag U (d)T U (d)
6 for j = 1 : R do
7 Wj = Y
8 for m = 1 : d do
(m) 
9 Wj = Wj ×m (uj )
10 end
11 end
12 for i = 1 : d do
13 if i = d then
14 γ=1
15 end

16 T3 = T1  U (i)T U (i)
17 for j = 1 : R do
18 W = Wj
19 Change ith core of W to Gi .
20 [T2 ]j is a full tensor from TT-tensor W .
 
(i) (i)
21 uj = γj uj + [T2 ]j − U (i) [T3 ]j
+
22 if i = d then
(i) (i) (i)
23 uj = uj / uj
2
24 end
25 Change ith core of Wj to Gi .
Wj = Wj ×i (uj )
(i)
26
27 end
28 T1 = T3  U (i)T U (i)
29 end
30 if stopping criterion is met then
31 break;
32 end
33 end

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

890 SHCHERBAKOVA, TYRTYSHNIKOV

where B (n) = (A⊗−n )G

(n) , and
∂DNTD
= Y ×1 (A(1) ) . . . ×d (A(d) ) − Y ×1 (A(1) ) . . . ×d (A(d) ) .
∂G
These partial gradients and the existing NMF algorithms are the basis for a set of first-order NTD
algorithms [19]. But all operations involving original tensor Y are extremely time and space demanding,
especially for large-scale problems. Similar to NCPD we can incorporate tensor train approximation of
Y in the algorithm and thus effectively construct NTD. And in this case it is very easy to do: all we need is
tensor in TT format by matrix multiplication over a given mode. For ∂D∂G NTD
it follows from the formula.
And for Y(n) B (n) ∂D
from ∂A(n) we take into account that Y(n) A
NTD ⊗ −n is exactly the mode-n matricized
(1) 
version of the tensor Y ×1 (A ) . . . ×n−1 (A (n−1) 
) ×n+1 (A (n+1) ) . . . ×d (A(d) ) . From [8] we can
reduce operation complexity to O(dN R ). 3

In [15] an alternating proximal gradient method (APG) for solving NTD is proposed. Unlike
alternating nonnegative least squares method (ANLS) that exactly solves each subproblem, APG
updates every factor matrix by solving a relaxed subproblem with a separable quadratic objective.
Each relaxed subproblem has a closed form solution, which makes low per-iteration cost. Using an
extrapolation technique, APG also converges quite fast. But as the authors commented, computing
partial gradients is very time consuming process for large tensors. Adding tensor train decomposition
as it was discussed earlier helps to considerably accelerate this algorithm. Alternating proximal gradient
for NTD corrects values for A(n) and G turning negative in each iteration, this means that we don’t need
nonnegative tensor train approximation and can use any TT factorization method as it doesn’t affect the
result.

5. NUMERICAL EXPERIMENTS
Relative accuracy is computed as A−B
AF × 100%, where B is a constructed nonnegative approxi-
F

mation of the original nonnegative tensor A. Experiments were performed on a desktop computer with
an Intel CPU at 3.10 GHz using 8.0 GB of RAM running macOS Catalina 10.15.7 and Matlab R2018a.

5.1. NTT for Coagulation Kernel for Smoluchowski Equation

Smoluchowski equation is a mathematical model of coagulation processes involving huge numbers
of inelastically colliding particles. When the particles are uniformly distributed in space and can have
only pairwise collisions we get the Smoluchowski coagulation equation
v ∞
∂n(v, t) 1
= K(v − u, u)n(v − u, t)n(u, t)du − n(v, t) K(v, u)n(u, t)du.
∂t 2
0 0

This equation describes evolution in time of the concentration function n(v, t) of particles of size v per
unit volume of the system at the moment t. When particles are composed of l different components, their
sizes form a vector v = (v1 , . . . , vl ). Then coagulation process is described by multicomponent version
of Smoluchowski equation.
We have considered coagulation kernel for Smoluchowski equation K(u, v) = (u1 + u2 )μ (v1 +
v2 )ν + (u1 + u2 )ν (v1 + v2 )μ , ui ≥ 0, i = 1, 2, vi ≥ 0, i = 1, 2, where different parameters μ, ν were
chosen so that μ + ν ≤ 1, |μ − ν| ≤ 2. Variables ui , vi , i = 1, 2 range from 0 to 10 with step 0.1. So
we have fourth order tensor with 108 elements. For each pair μ, ν the experiments were repeated 10
times. We considered μ = 0.2 and ν = 0.1. Power method for this tensor converges in 300 iterations,
which takes 0.2 s, and gives the right answer. The problem here is that two maximum values are very
close to each other. Despite this it works quite well because approximation TT ranks turn out to be
relatively small. But TT-crossbased minimization procedure from [12] takes fewer time and in our
experiments we decided to use this method for faster performance, although in general case it may need
some tuning to find absolute extrema. For experiment different TT decompositions from [12] were tested.
For comparison NTTF method for nonnegative TT factorization from [18] is considered.

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

 FAST NONNEGATIVE TENSOR FACTORIZATIONS 891

This example has good TT approximations, because appearing negative values are very small and by
adding a constant all elements become positive. At the same time relative error changes insignificantly
and ranks increase only by one.
The results from Table 2 prove proposed approach very promising for nonnegative tensor train
factorization in considered application.

5.2. NTT for ERP Data

In article [11] the authors considered power spectrum obtained from the event-related potential
(ERP) tensor as an example for numerical experiments. Data is presented in form of fourth order tensor
of size 71 × 60 × 42 × 9 [20, 21]. These dimensions correspond to the spectral, temporal, subject and
spatial modes.
From [11] we have estimations for ranks of NTT. They are 4, 10, 6.
Tensor ERP approximation with TT-SVD and correction through search of minimum elements gives
the ranks 5, 11, 10.

Table 2. Method to K(u, v), size: 100 × 100 × 100 × 100

Method Relative error (%) Time (s, ± std) Average ranks Min ranks Max ranks
NTTF 0.12 63.2 ± 2.8 3, 25, 41 3, 1, 3 3, 85, 100
AMEN-CROSS + min correction 0.0028 1.3 ± 0.4 7, 3, 7 7, 3, 7 7, 3, 7
DMRG-CROSS + min correction 0.0446 9.2 ± 5.2 7, 5, 7 7, 5, 7 7, 5, 7
TT-SVD + min correction 0.0338 4.3 ± 0.1 5, 3, 5 5, 3, 5 5, 3, 5

Table 3. Methods performance for ERP data

Method Relative error (%) Time (s)

NTTF 13.6 2.5
NTT-MU 13.6 4.1
TT-SVD + minimum correction 11.5 0.3

Table 4. One iteration of algorithms to positive tensors with nonnegative NTD ranks equal 5

Method Size Time (s)

NTD with APG 50 × 50 × 50 × 50 0.14
NTD with APG based on TT 50 × 50 × 50 × 50 0.02
NTD with APG 100 × 100 × 100 × 100 2.96
NTD with APG based on TT 100 × 100 × 100 × 100 0.03

Table 5. NTD algorithms to Smoluchowski coagulation kernel

Method Size Relative error (%) Time (s)

NTD with APG 50 × 50 × 50 × 50 0.8 80.2
NTD with APG based on TT 50 × 50 × 50 × 50 0.8 6.5

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

892 SHCHERBAKOVA, TYRTYSHNIKOV

Table 6. Comparison of algorithms applied to positive tensors with nonnegative NCP ranks equal 5

Method Size Time (s) Relative error (%)

FAST-HALS NTF 50 × 50 × 50 × 50 6.8 6.6
FAST-HALS NTF based on TT 50 × 50 × 50 × 50 2.1 6.8
FAST-HALS NTF 100 × 100 × 100 × 100 135.1 6.5
FAST-HALS NTF based on TT 100 × 100 × 100 × 100 2.6 6.5

Table 7. NCP algorithms to Smoluchowski coagulation kernel

Method Size Relative error (%) Time (s)

FAST-HALS NTF 50 × 50 × 50 × 50 0.5 8
FAST-HALS NTF based on TT 50 × 50 × 50 × 50 0.5 2.4
FAST-HALS NTF 100 × 100 × 100 × 100 0.6 146.1
FAST-HALS NTF based on TT 100 × 100 × 100 × 100 0.6 8

We can check the quality of nonnegative TT approximation with ranks 4, 10, 6, which is built by
NTT-MU (300 sweeps) from [16] for this data. Values in Table 3 are the averaged results for 100
repetitions of the experiment.
For this example in terms of time and relative error we also achieve better results for building
nonnegative TT with method proposed in Section 2.

5.3. Performance of NTD Based on TT

First we checked what acceleration using TT for the algorithm from [15] brings. For the purity of
the experiment one iteration of each algorithm was considered for tensors of size 50 × 50 × 50 × 50 and
100 × 100 × 100 × 100. There were 10 repetitions of each experiment, all results were averaged. At
the beginning positive tensor train was generated with all ranks equal 5. So it was used as input for
algorithm version based on TT, whereas for initial method from [15] full tensor form was used. The
results are presented in Table 4.
As expected we observe that algorithm based on TT works much faster.
Now let us consider coagulation kernel for Smoluchowski equation from subsection 5.5.1. For this
example variables ui , vi , i = 1, 2, range from 0 to 5 with step 0.1. Ranks for NTD were chosen equal
5. As prior TT decomposition method we used TT-SVD. The experiment was repeated 10 times, the
averaged results are presented in Table 5.
Despite the need for additional time to compute TT approximation with high accuracy so that relative
error doesn’t worsen, computations become several times faster.

5.4. Performance of NCP Based on TT

We chose for experiments method NCP-HALS for two reasons. It is well known for fast performance
and also we can use any tensor train approximation for it’s changed version because it already includes
possible negative value correction after. First, as it was done for NTD, we checked what acceleration
using TT for the algorithm brings. Again for the objectivity of the experiment positive tensor train was
generated with all ranks equal 5. So it was used as input for algorithm version based on TT, whereas
for initial method from [10] full tensor form was used. The averaged results for 10 repetitions of each
experiment results are presented in Table 6.
As we used the same random initialization for both algorithms we can confirm that method 3 gives
the same results much faster.

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

 FAST NONNEGATIVE TENSOR FACTORIZATIONS 893

Next we use algorithms to construct NCP for coagulation kernel for Smoluchowski equation from
subsection 5.5.1. For the first example variables ui , vi , i = 1, 2 range from 0 to 5 with step 0.1, whereas
for the second they range from 0 to 10 with step 0.1. Ranks for NCP were chosen equal 5. As prior
TT decomposition method we used TT-SVD with accuracy 1e − 6. Values in Table 7 are the averaged
results for 10 repetitions of the experiment.
From Table 7 it follows that high accuracy of TT approximation allows to keep the same relative error
as for the initial tensor. And we still have great acceleration for the computations, and with TT-CROSS
methods it is possible to decrease work time even more.

6. CONCLUSION
In this article we propose new approaches to nonnegative tensor factorization based on tensor train
model. They allow to construct decompositions much faster.
First we consider nonnegative tensor train factorization problem. For nonnegative tensor decompo-
sitions by definition all factors are required to be nonnegative. Partly it is done for the interpretability
of the results. The other reason is more practical: with this condition we don’t have to worry about
nonnegativity of the resulting approximation. But there are many cases when TT factorization is
used to simply save memory and speed up computations. Of course, if the data is nonnegative, we
want decomposition to have the same property. Moreover, there exist algorithms for which tensor
nonnegativity is crucial for convergence. In these cases it makes sense to consider the problem where
we have to build TT approximation to nonnegative data so that all resulting entries are nonnegative. We
suggest the method for solving the aforementioned task based on correction of negative elements in TT
decomposition. We study examples where this approach gives the best results in time and accuracy.
Secondly we suggest a technique to use tensor train decompositions for constructing nonnegative
canonical and Tucker models. It allows to reduce the complexity of some operations from O(N d ) to
linear in N , thus bringing impressive acceleration to algorithms. What is more, some methods for
NCP and NTD such as NCP-HALS, NTD with APG, etc. don’t require tensor train approximation
to be nonnegative, and it is well known that additional constraints usually slow down computations and
worsen the accuracy of decomposition. And for TT factorization there are effective algorithms based on
cross methods.
Proposed techniques for constructing NTT, NCP and NTD decompositions based on TT are very
useful for big data problems as they allow to greatly reduce complexity of the methods. In the future we
plan to further develop the proposed approaches and test them for more applications.

FUNDING
The work was supported by the Moscow Center of Fundamental and Applied Mathematics (agree-
ment 075-15-2019-1624 with Ministery of education and science of Russian Federation).

REFERENCES
1. F. L. Hitchcock, “Multiple invariants and generalized rank of a p-way matrix or tensor,” J. Math. Phys. 7,
39–79 (1927).
2. R. A. Harshman, “Determination and proof of minimum uniqueness conditions for PARAFAC1,” in UCLA
Working Papers in Phonetics (1972), Vol. 22.
3. J. D. Carroll and J. J. Chang, “Analysis of individual differences in multidimensional scaling via an n-way
generalization of Eckart–Young decomposition,” Psychometrika 35, 283–319 (1970).
4. J. B. Kruskal, “Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to
arithmetic complexity and statistics,” Linear Algebra Appl. 18, 95–138 (1977).
5. J. D. Carroll, G. de Soete, and S. Pruzansky, “Fitting of the latent class model via iteratively reweighted
least squares CANDECOMP with nonnegativity constraints,” in Multiway Data Analysis (Elsevier,
Amsterdam, The Netherlands, 1989), pp. 463–472.
6. A.-H. Phan, A. Cichocki, I. Oseledets, G. G. Calvi, S. Ahmadi-Asl, and D. P. Mandic, “Tensor networks
for latent variable analysis: Higher order canonical polyadic decomposition,” IEEE Trans. Neural Networks
Learn. Syst. 31, 2174–2188 (2020). https://doi.org/10.1109/TNNLS.2019.2929063
7. O. Lebedeva, “Tensor conjugate-gradient-type method for Rayleigh quotient minimization in block qtt-
format,” Russ. J. Numer. Anal. Math. Model. 26, 465–489 (2011).

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022

894 SHCHERBAKOVA, TYRTYSHNIKOV

8. I. Oseledets, “Tensor-train decomposition,” SIAM J. Sci. Comput. 33, 2295–2317 (2011).

9. S. Kuksova, E. W. Skau, and B. S. Alexandrov, “Error analysis of nonnegative tensor train utilized for
nonnegative canonical polyadic decomposition,” in Proceedings of the 37th International Conference on
Machine Learning ICML, 2020.
10. A. Cichocki and Phan Anh-Huy, “Fast local algorithms for large scale nonnegative matrix and tensor factor-
izations,” in IEICE Transactions on the Fundamentals of Electronics, Communication and Computer
Science, 2009.
11. N. Lee, A. Phan, F. Cong, and A. Cichocki, “Nonnegative tensor train decompositions for multi-domain
feature extraction and clustering,” Lect. Notes Comput. Sci. 9949, 87–95 (2016).
12. I. Oseledets et al., TT-Toolbox (TT=Tensor Train), Version 2.2.2 (Inst. Numer. Math., Moscow, Russia,
2009–2013). https://github.com/oseledets/TT-Toolbox.
13. I. Oseledets and E. Tyrtyshnikov, “TT-cross approximation for multidimensional arrays,” Linear Algebra
Appl. 432, 70–88 (2010).
14. Anh-Huy Phan, P. Tichavsky, and A. Cichocki, TENSORBOX: A MATLAB Package for Tensor Decompo-
sition. https://github.com/phananhhuy/TensorBox. Accessed 2019.
15. Y. Xu, “Alternating proximal gradient method for sparse nonnegative Tucker decomposition,” Math. Program.
Comput. 7, 39–70 (2015).
16. E. Shcherbakova, “Nonnegative tensor train factorization with DMRG technique,” Lobachevskii J. Math.
40, 1863–1872 (2019).
17. E. Tyrtyshnikov and E. Shcherbakova, “Nonnegative matrix factorization methods based on low-rank cross
approximations,” Comput. Math. Math. Phys. 59 (8) (2019).
18. E. Shcherbakova and E. Tyrtyshnikov, “Nonnegative tensor train factorizations and some applications,” in
Large-Scale Scientific Computing (Springer Int., Cham, 2020), pp. 156–164.
19. G. Zhou, A. Cichocki, Q. Zhao, and S. Xie, “Efficient nonnegative Tucker decompositions: Algorithms and
uniqueness,” IEEE Trans. Image Process. 24, 4990–5003 (2015).
20. F. Cong, Q.-H. Lin, L.-D. Kuang, X.-F. Gong, P. Astikainen, and T. Ristaniemi, “Tensor decomposition of
EEG signals: A brief review,” J. Neurosci. Methods 248, 59–69 (2015).
21. F. Cong, A. Phan, Q. Zhao, T. Huttunen-Scott, J. Kaartinen, T. Ristaniemi, H. Lyytinen, and A. Cichocki,
“Benefits of multi-domain feature of mismatch negativity extracted by nonnegative tensor factorization from
low-density array EEG,” Int. J. Neural Syst. 22 (6) (2012).
22. L. R. Tucker, “The extension of factor analysis to three-dimensional matrices,” in Contributions to
Mathematical Psychology, Ed. by H. Gulliksen and N. Frederiksen (Holt, Rinehart and Winston, New
York, 1964), pp. 110–127.
23. Y. D. Kim and S. Choi, “Nonnegative Tucker Decomposition,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR07) (Minneapolis, 2007), pp. 1–8.
24. A. Marmoret, J. E. Cohen, N. Bertin, and F. Bimbot, “Uncovering audio patterns in music with Nonnegative
Tucker Decomposition for structural segmentation,” arXiv (2021), arXiv:2104.08580.
25. Y. Qiu, G. Zhou, Y. Wang, Y. Zhang, and S. Xie, “A Generalized Graph Regularized Non-Negative Tucker
Decomposition Framework for Tensor Data Representation,” IEEE Transactions on Cybernetics 52 (1),
594–607 (2022).
26. Brett W. Bader, Tamara G. Kolda, et al., Tensor Toolbox for MATLAB, Version 3.2.1 (2021).
https://tensortoolbox.org.

LOBACHEVSKII JOURNAL OF MATHEMATICS Vol. 43 No. 4 2022