Two-Dimensional Systems & Mathematical Preliminaries
Two-Dimensional Systems & Mathematical Preliminaries
dx x x x
) ( ' ) ' ( ) ' ( x f dx x x x f =
a
x
ax
) (
) (
=
=
0 1
0 0
) (
n
n
x
) ( ) ( ) ( n f m n m f =
>
=
2 / 1 0
2 / 1 1
) (
x
x
x rect
<
=
>
=
0 1
0 0
0 1
) (
x
x
x
x sign
= ) ( ) ( n x x comb
x
x
x c
sin
) ( sin =
>
=
1 0
1 1
) (
x
x x
x tri
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 4/27
2. Linear systems and shift invariance
2D linear systems
[ ]
[ ] ) , ( ) , ( n m x n m y =
) , ( n m x
Linear superposition property:
Impulse response:
Impulse response is called PSF: Input and output are positive quantities
In general: Impulse response can take negative or complex values
Region of support (RoS) of impulse response
Finite impulse response (FIR) and infinite impulse response (IIR): When
RoS is finite or infinite
[ ] ) ' , ' ( ) ' , ' ; , ( n n m m n m n m h =
[ ] [ ] [ ] ) , ( ) , ( ) , ( ) , ( ) , ( ) , (
2 2 1 1 2 2 1 1 2 2 1 1
n m y a n m y a n m x a n m x a n m x a n m x a + = + = +
3
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 5/27
Output of a linear system:
[ ]
(
= =
) ' , ' ( ) ' , ' ( ) , ( ) , (
' '
n n m m n m x n m x n m y
m n
[ ]
= =
' ' ' '
) ' , ' ; , ( ) ' , ' ( ) ' , ' ( ) ' , ' ( ) , (
m n m n
n m n m h n m x n n m m n m x n m y
Spatially invariant or shift-invariant system: [ ] ) 0 , 0 ; , ( ) , ( n m h n m =
[ ] ) 0 , 0 ; ' , ' ( ) ' , ' ( ) ' , ' ; , ( n n m m h n n m m n m n m h = =
) ' , ' ( ) ' , ' ; , ( n n m m h n m n m h =
The shape of impulse response does not change as the impulse
response moves about the (m,n) plan
Discrete convolution: ) , ( * ) , ( ) ' , ' ( ) ' , ' ( ) , (
' '
n m x n m h n m x n n m m h n m y
m n
= =
Continuous convolution:
' ' ) ' , ' ( ) ' , ' ( ) , ( * ) , ( ) , ( dy dx x x f y y x x h y x f y x h y x g
= =
Exp. 2.1
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 6/27
3. Fourier transform
FT of a 1D function: f(x)
[ ]
[ ]
+
+
= =
= =
d e F F x f
dx e x f x f F
x j
x j
2 1 1
2
) ( ) ( ) ( :
) ( ) ( ) ( :
FT of a 2D function: f(x,y)
[ ]
[ ]
+
+
+
+
+
+
= =
= =
2 1
) ( 2
2 1 2 1
1 1
) ( 2
2 1
2 1
2 1
) , ( ) , ( ) , ( :
) , ( ) , ( ) , ( :
d d e F F y x f
dxdy e y x f y x f F
y x j
y x j
4
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 7/27
Performing a change of variables:
Properties of FT
Spatial frequencies
Uniqueness
Separability
+
+
+
+
(
= = dy e dx e y x f dxdy e y x f F
y j x j y x j
2 1 2 1
2 2 ) ( 2
2 1
) , ( ) , ( ) , (
Frequency response and eigenfunctions of shift invariant systems
) , ( y x h H
+
+
+
= ' ' ) ' , ' ( ) , (
) ' ' ( 2
2 1
y d dx e y y x x h y x g
y x j
' , ' y y y x x x = =
) ( 2
2 1
: where
y x j
e
+
=
) ( 2
2 1
2 1
) , ( ) , (
y x j
e y x g
+
=
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 8/27
Convolution theorem
Inner product preservation
) , ( F ) , ( ) , G( ) , ( * ) , ( ) , (
2 1 2 1 2 1
= = y x f y x h y x g
Correlation between 2 real functions
+
+
+ + = = ' ' ) ' , ' ( ) ' , ' ( ) , ( ) , ( ) , ( dy dx y y x x f y x h y x f y x h y x c
Performing a change of variables:
) , F( ) , ( ) , ( C ) , ( ) , ( ) , (
2 1 2 1 2 1
= = y x f y x h y x c
+
+
+
+
= =
2 1 2 1
*
2 1
*
) , ( H ) , ( F ) , ( ) , ( d d dxdy y x h y x f I
Setting h=f Parseval energy conservation formula
+
+
+
+
=
2 1
2
2 1
2
) , ( F ) , ( d d dxdy y x f
5
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 9/27
Fourier transform pairs
comb(x,y)
tri(x,y)
rect(x,y)
1
) , ( y x f ) , ( F
2 1
) , ( y x
yl j xk j
e e
2 2
2 0 1 0
2 2 y j x j
e e
) , (
2 1
l k m m
) , (
0 0
y y x x
) (
2 2
y x
e
+ ) (
2
2
2
1
y
e
+
) , ( s
2 1
inc
) , ( s
2 1
2
inc
) , (
2 1
comb
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 10/27
Inner product
Spatial correlation
Multiplication
Convolution
Modulation
Shifting
Scaling
Separability
Conjungation
Linearity
Rotation
Property of 2D FT Fourier transform Function
) , F(
2 1
) ( 2
2 0 1 0
y x j
e
+
) ( ) (
2 1
y f x f
) , ( y x f
) , ( F ) , ( H ) , ( G
2 1 2 1 2 1
=
) , ( ) , (
2 2 1 1
y x f a y x f a + ) , ( F ) , ( F
2 1 2 2 2 1 1 1
a a +
) , (
*
y x f ) , ( F
2 1
*
) ( F ) ( F
2 2 1 1
) , ( y x f
) , ( by ax f [ ] ab b a / ) / , / ( F
2 1
) , (
0 0
y y x x f
) , (
) ( 2
2 1
y x f e
y x j +
) , ( F
2 2 1 1
m m
) , ( ) , ( ) , ( y x f y x h y x g =
) , ( ) , ( ) , ( y x f y x h y x g = ) , ( F ) , ( H ) , ( G
2 1 2 1 2 1
=
) , ( ) , ( ) , ( y x f y x h y x c = ) , ( F ) , ( H ) , ( C
2 1 2 1 2 1
=
+
+
= dxdy y x h y x f I ) , ( ) , (
*
+
+
2 1 2 1
*
2 1
) , ( H ) , ( F d d
) , ( F
2 2 1 1
m m
) , ( F
2 1
6
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 11/27
The evaluation of at and yields FT of
4. Z-Transform or Laurent series
Fourier transform of sequences (Fourier series): Self-reading
Generalization of FT series: Z-transform
For 2D sequence x(m,n):
where z
1
, z
2
are complex variables
+
=
+
=
=
m n
n m
z z n m x z z X
2 1 2 1
) , ( ) , (
Region of converge (RoC): this series converges uniformly in this region
Z-transform of a LSI system is called transfer function
) , (
) , (
) , (
) , ( ) , ( ) , (
2 1
2 1
2 1
2 1 2 1 2 1
z z X
z z Y
z z H
z z X z z H z z Y
=
=
Inverse Z-transform:
1 ; 1 where , ) , (
) 2 (
1
) , (
2 1
2
1
1
2
1
1 2 1 2
= = =
z z dz dz z z z z X
j
n m x
n m
) , (
2 1
z z X
1
1
j
e z =
2
2
j
e z = ) , ( n m x
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 12/27
Properties of 2D Z-transform
Multiplication
Convolution
Modulation
Shifting
Separability
Conjungation
Linearity
Rotation
Property Fourier transform Function
) , (
2 1 2 1
0 0
z z X z z
n m
) ( ) (
2 1
n x m x
) , ( n m x
) , ( ) , (
2 1 2 1
z z F z z H
) , ( ) , (
2 2 1 1
n m x a n m x a + ) , ( ) , (
2 1 2 2 2 1 1 1
z z X a z z X a +
) , (
*
n m X ) , (
*
2
*
1
*
z z X
) ( F ) ( F
2 2 1 1
) , ( y x f
) , (
0 0
n n m m x
) , ( n m x b a
n m
) , ( ) , ( n m x n m h
) , ( ) , ( n m y n m x
) , (
1
2
1
1
z z X
) , ( F
2 1
|
\
|
b
z
a
z
X
2 1
,
|
|
\
|
|
|
\
|
1 2
'
2
2
'
1
1
'
2
'
1 '
2
2
'
1
1
2
) , ( ,
2
1
C C
z
dz
z
dz
z z Y
z
z
z
z
X
j
7
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 13/27
Causality
Causal: Impulse response for and its transfer function
must have a one-sided Laurent series
0 ) ( = n h 0 < n
=
0
) ( ) (
n
n
z n h z H
Anticausal: Impulse response for and its transfer function
must have a one-sided Laurent series
0 ) ( = n h 0 n
Noncausal: Neither causal or anticausal
Stability: Output remains uniformly bounded for any bounded input
<
=0
) (
n
n h
Causal and stable system: poles of H(z) must lie inside the unit circle
2D case: RoC of must include the unit circles
<
m n
n m h ) , ( ) , (
2 1
z z H
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 14/27
5. Matrix theory and results
Vectors and matrices
Column vector of size N: N n n u U = = 1 ), (
Row vector of size M: M m m u U = = 1 ), (
Matrix A of size MxN containing M rows, N columns
(
(
(
(
=
) , ( ) 2 , ( ), 1 , (
) , 2 ( ) 2 , 2 ( ), 1 , 2 (
) , 1 ( ) 2 , 1 ( ), 1 , 1 (
N M a M a M a
N a a a
N a a a
A
L
L
L
L
Index notation:
{ } 1 , 0 ), , ( =
N n m n m a A
N N
An image is usually visualized as a matrix Exp. 2.2
8
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 15/27
Row and column ordering
Row-ordered vector (row stacking)
[ ]
T T
N M x M x N x x N x x x x ) , ( ..., ), 1 , ( ) , 2 ( ..., ), 1 , 2 ( ), , 1 ( ..., ), 2 , 1 ( ), 1 , 1 ( L =
[ ]
T T
N M x M x M x x M x x x x ) , ( ..., ), , 1 ( ) 2 , ( ..., ), 2 , 1 ( ), 1 , ( ..., ), 1 , 2 ( ), 1 , 1 ( L =
Column-ordered vector (column stacking)
Matrix theory definitions
{ } ) , ( n m a A = Matrix:
Transpose: { } ) , ( m n a A
T
= { } ) , (
* *
n m a A = Complex conjungate:
{ } ) ( n m I =
Conjungate transpose:
Identity matrix:
Null matrix: { } 0 = O
{ } ) , (
* *
m n a A
T
=
Matrix addition: { } ) , ( ) , ( n m b n m a B A + = + : A, B: Same dimension
Scalar multiplication: { } ) , ( n m a A =
Matrix multiplication:
=
=
K
k
n k b k m a n m c
1
) , ( ) , ( ) , ( A: MxK, B: KxN, C: MxN
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 16/27
Vector inner product:
= = ) ( ) ( ,
* *
n y n x Y X Y X
T
: Scalar quantity, if equal 0
X and Y are orthogonal
Vector outer product: { } ) ( ) ( n y m x XY
T
= : X: Mx1, Y: Nx1, XY
T
: MxN
Symmetric:
T
A A =
Hermitian:
T
A A
*
= : Real symmetric matrix is Hermitan. Eigenvalues are real
Determinant: A
Rank of A: Number of independent rows or columns
Inverse matrix: I AA A A = =
1 1
: Square matrix only
Singular: A
-1
does not exist and
0 = A
Eigen values : all roots of
k
0 = I A
k
+
+
0 1 2 1
1 2
2 1 0 1
1 1 0
, , ,
, ,
,
t t t t
t t
t t t t
t t t
A
N
N
N
L
L
L
L
Circulant matrix C
(
(
(
(
(
(
0 1 2 1
2
2 1 0 1
1 2 1 0
, , ,
, ,
, ,
c c c c
c
c c c c
c c c c
C
N
N N
N
L
L
L
L
C is also Toeplitz and c(m,n)=c((m-n) modulo N)
Exp. 2.3
Exp. 2.4
t(i,j) = t
i-j
: Constant elements along the
main diagonal and sub-diagonal
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 18/27
where and are eigenvalues and eigenvectors of R
Other form ,which is the set of eigenvalue equations
Orthogonal and unitary matrices
Orthogonal matrix:
Unitary matrix: I A A AA A A
T T T
= = =
* * * 1
or
Exp. 2.5a
Diagonal forms
If R is Hermitian matrix, there exists a unitary matrix such that
where is a diagonal matrix containing eigenvalues of R
= R
*T
= R
N k
k k
, , 2 , 1 , R
k
L = =
{ }
k
Exp. 2.5b
I A A AA A A
T T T
= = =
or
1
10
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 19/27
is block Toeplitz if is Toeplitz or
6. Block matrices and Kronecker products
Block matrices of size : each element is a matrix itself
(
(
(
(
(
=
n m m m
n
n
A A A
A A A
A A A
, 2 , 1 ,
, 2 2 , 2 1 , 2
, 1 2 , 1 1 , 1
,
,
,
L
L
L
L
where are matrices
j i
A
,
) ( , j i j i
A A
=
is block circulant if is circulant
n m A A
n j i j i
= =
,
) ulo mod ) (( ,
Exp. 2.6
Exp. 2.7
Kronecker products: A: M
1
xM
2
, B: N
1
xN
2
:
Separable operations: self-reading
{ } B n m a B A ) , ( =
q p
j i
A
,
j i
A
,
n m
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 20/27
is an Nx1 vector
7. Random signals
Definitions: given a sequence of random variables u(n)
Mean:
Variance:
Covariance:
Cross-covariance:
Auto-correlation:
Cross-correlation:
[ ] ) ( ) ( ) ( n u E n n
u
= =
[ ]
2
2 2
) ( ) ( ) ( ) ( n n u E n n
u
= =
[ ] [ ][ ] { } ) ' ( ) ' ( ) ( ) ( ) ' , ( ) ' ( ), (
* *
n n u n n u E n n r n u n u Cov
u
= =
[ ] [ ][ ] { } ) ' ( ) ' ( ) ( ) ( ) ' , ( ) ' ( ), (
* *
n n v n n u E n n r n v n u Cov
v u uv
= =
[ ] ) ' ( ) ( ) ' , ( ) ( ) ( ) ' , ( ) ' , (
* *
n n n n r n u n u E n n a n n a
uu
= = =
[ ] ) ' ( ) ( ) ' , ( ) ( ) ( ) ' , (
* *
n n n n r n v n u E n n a
v u uv uv
= =
For vector of size Nx1: u
[ ] { } ) (n E = = u
is an NxN matrix [ ] [ ] { } ) ' , ( ) )( (
* *
n n r E Cov
T
= = = = R R u u u
u
is an NxN matrix [ ] [ ] { } ) ' , ( ) )( ( ,
* *
n n r E Cov
uv
T
= = =
uv v u
R v u v u
11
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 21/27
Gaussian random process: the joint probability density of any finite sub-
sequence is a Gaussian distribution
Stationary process
Gaussian or normal distribution: for standard normal distribution
=
2
2
2
2
2
1
) (
u
u
e u p
1 and 0
2
= =
u(n) is strict-sense stationary: joint density of any partial
sequence is the same as that of the shifted
sequence for any integer m and any length k
{ } k l l u 1 ), (
{ } k l m l u + 1 ), (
u(n) is wide-sense stationary: [ ] const n u E = = ) (
[ ] ) ' ( ) ' , ( ) ' ( ) (
*
n n r n n r n u n u E = =
Symmetry:
Non-negativity:
' , ) ' , ( ) ' , (
*
n n n n r n n r =
n n x n x n n r n x
n n
, ) ( , 0 ) ' ( ) ' , ( ) (
'
*
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 22/27
Markov process
Markov-p or pth-order Markov
Gaussian Markov-p sequence
[ ] [ ] n p n u n u n u E n u n u n u E = ) ( , ), 1 ( | ) ( ) 2 ( ), 1 ( | ) ( L L
Covariance function of a first-order stationary Markov sequence u(n)
n n r
n
< = , 1 ) (
Toeplitz covariance matrix
(
(
(
(
(
(
1 , , ,
1 ,
, , 1
1
1 2
L
L
L
L
N
N
R
[ ] [ ] n p n u n u n u prob n u n u n u prob = ) ( , ), 1 ( | ) ( ) 2 ( ), 1 ( | ) ( L L
12
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 23/27
Orthogonality and independence
x and y are independent: n y p x p y x p
y x y x
< = , 1 ) ( ) ( ) , (
,
x(n) and y(n) random sequences are independent: x(n) and y(n) are
independent for every n and n
x and y are orthogonal: [ ] 0
*
= xy E
x and y are uncorrelated: or [ ] [ ] ( ) [ ] ( )
* *
E E E y x xy = [ ] 0 ) )( ( E
*
=
y x
y x
Karhunen-Loeve transform (KLT)
is a complex random sequence with { } N n n x 1 ), ( R
is an NxN unitary matrix, which reduces to its diagonal form R
KLT of : x x y
*T
=
Property of KLT: [ ] [ ] { } R xx yy = = =
T T * * * *
E E
[ ] ) ( ) ( ) ( E
*
l k l y k y
k
=
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 24/27
8. Some results from estimation theory
Mean square estimates (MSE)
is a real random sequence and x is a real random variable { } N n n y 1 ), (
is called the optimum mean square estimate of x if the MSE is minimized
x
(
=
2 2
) ( x x E
That means: ( ) ( ) [ ]
= = = d p N y y y x E y x E x
y x
) ( ) ( , ), 2 ( ), 1 ( | |
|
L
If x and y(n) are independent, then simply is mean of x because:
x
( ) [ ] ( ) x E y x E E x E = =
(
|
For zero mean Gaussian random variables, becomes linear in y(n)
=
N
n
n y n x
1
) ( ) (
13
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 25/27
The orthogonal principles
From MSE theory: error vector is orthogonal to every random variable
i.e for any ( ) ) ( , ), 2 ( ), 1 ( ) ( N y y y g y g L =
0 ) ( ) (
2
=
(
y g x x E
Orthogonality principle is useful in linear estimate since the cond. mean is
difficult to evaluate finding that minimizes the MSE. That is: ) (n
[ ] [ ] N n n xy E n y k y E k
N
k
, , 1 , ) ( ) ( ) ( ) (
1
L = =
In matrix form:
xy y
r R
1
=
The minimized MSE is given:
xy
T
x
r =
2 2
Prove
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 26/27
9. Some results from information theory
Information: There is a source generating a discrete set of independent
message (e.g gray level) r
k
, with probabilities p
k
. Since p
k
1
k k
p I
2
log =
Entropy: Defined as average info generated by the source (bit/message)
=
=
L
k
k k
p p H
1
2
log
For a given L, max entropy of a source is determined for uniform
distribution, i.e
L
L L
H
L
k
p
k
2
1
2
log
1
log
1
max = =
=
L k L p
k
, , 1 , / 1 L = =
Exp. 2.13
Binary source L=2. Then p
1
=p, p
2
=1-p, 1 0 p
Entropy is: H
max
=? ) 1 ( log ) 1 ( log ) (
2 2
p p p p p H H = =
14
Hanoi University of Science and Technology School of Electronics and Telecommunications
2D systems and mathematical preliminaries **0 1/2013 Introduction 27/27
Rate distortion function (RDF)
RDF of a random variable x gives min average rate R
D
required to
represent (or code) it for a fixed distortion D
x: Gaussian with ; y: reproduced value; MSE(x-y): Distortion measure
2
[ ]
2
) ( y x E D =
Then RDF of x is defined as:
( )
(
|
|
\
|
=
>
=
D
D
D D
R
D
2
2
2
2 2
2
log
2
1
, 0 max
0
/ log
2
1
Mean square distortion for Gaussian variables
and their reproduced values is defined as:
{ } ) 1 ( , ), 1 ( ), 0 ( N x x x L
{ } ) 1 ( , ), 1 ( ), 0 ( N y y y L
[ ]
=
=
1
0
2
) ( ) (
1
N
k
k y k x E
N
D