Second Order Diagonal Recurrent Neural Network

Second Order Diagonal Recurrent Neural Network Ali Kazemy†, Seyed Amin Hosseini† and Mohammad Farrokhi†,‡ † Faculty of Electrical Engineering Center of Excellence for Power System Automation and Operation Iran University of Science and Technology Tehran, Iran ali_kazemy@ee.iust.ac.ir, amin_hosseini@ee.iust.ac.ir, farrokhi@iust.ac.ir ‡ Abstract—In this paper a new diagonal recurrent neural network that contains two recurrent weights in hidden layer is proposed. Since diagonal recurrent neural networks have simpler structure than the fully connected recurrent neural networks, they are easier to use in real-time applications. On the other hand, all diagonal recurrent neural networks in literature use one recurrent weight in hidden neurons, while the proposed network takes advantage of two recurrent weights. It will be shown, by simulations, that the proposed network can approximate nonlinear functions better than the existing diagonal recurrent neural networks. After deriving the training algorithm, the convergence stability and adaptive learning rate will be presented. The performance of the proposed network in model identification shows the accuracy of this network against the diagonal recurrent neural networks. Moreover, this network will be applied to realtime control of an image stabilization platform. I. INTRODUCTION Neural networks are effective tools for modeling and control of nonlinear systems [1]-[3]. Most researchers use feedforward neural networks (FNNs), combined with tapped delays and the backpropagation training algorithm (BP) [4] to identify dynamical systems. The problem is that FNNs provide a static mapping and without the aid of tapped delays, it cannot approximate nonlinear dynamic systems with good accuracy. Recurrent Neural Networks (RNNs), on the other hand, have dynamic ability; hence, more suitable for dynamic systems. In Fully connected Recurrent Neural Networks (FRNNs) [5],[6], where all neurons are connected to each other, it is difficult to train the network due to massive number of weights; hence, it takes more time for weights to converge. Alternatively, Diagonal Recurrent Neural Networks (DRNNs) are a special kind of recurrent networks, which have fewer weights and shorter training time as compare to FRNNs. DRNN was first established by Ku [7]-[11]. Later on, other researchers developed the performance of this network [12]. DRNNs have been widely applied in system identification and controller design [13]-[20]. In the past decade, many recurrent neural network architectures were introduced in literatures. The structures of these networks are similar except for the recurrent neurons and their synaptic weights. Higher order DRNNs [21] use different combinations of recurrent neurons. Most of these networks are tailored for special plants and generally may not be able to identify and/or control a large class of systems. Also block-diagonal recurrent neural networks (BDRNN) [22]-[26], 1-4244-0755-9/07/$20.00 '2007 IEEE which were introduced recently, use a combination of a pair of neurons, called block, in their network structure, along with one tapped delay in hidden neurons. The main purpose of the taped delay is to retain the history of states of neurons. However, the recurrent weights are updated only using the previous state and cannot use other states directly. In this paper, two tapped delays are used in the hidden neurons of DRNN, hence calling it Second-order Diagonal Recurrent Neural Network (SDRNN). With this structure, more history of states of neurons can be incorporated directly into the training algorithm. A generalized dynamic backpropagation (DBP) algorithm will be derived, and it will be shown that the proposed SDRNN not only provides more accurate identification, but also a shorter training time as compare to DRNN. Moreover, convergence of the proposed dynamic backpropagation is developed and the maximum learning rate that guarantees the convergence of the algorithm is derived. This paper is organized as follows. In section II, a comparison between the numbers of weights in the abovementioned recurrent neural networks will be given. Section III represents the training algorithm for the proposed neural network, followed by convergence and stability analysis of the algorithm in section IV. In section V a simulation example for system identification will be given. Section VI shows the ability of the proposed neural network to control a highly nonlinear system in an experimental setup. Finally, section VII concludes the paper. II. THE NUMBER OF WEIGHTS IN SDRNN, FRNN AND DRNN Let N T = {I p , H q ,O r } represent a T-type neural network with p inputs ( I p ) , q sigmoid neurons in the hidden layer ( H q ) , and r linear neurons in the output layer (O r ) . N R , N D and N S represent FRNN, DRNN and SDRNN, respectively. Let G T be the total number of weights for a T-type neural network. Therefore, the total number of weights (including q bias weights), for the N R , N D and N S neural networks are G R = ( p + r + 1) q + q 2 , (1) G D = ( p + r + 2)q , (2) G = ( p + r + 3) q . S (3) For instance, if p = 3 , q = 10 , and r = 1 , then G = 150 , G D = 60 and G S = 70 . Even in this small neural network, the 251 R number of weights in FRNN is far more than that of DRNN or SDRNN; but the number of weights in SDRNN is just q weights more than that of DRNN. linear neuron W III. DYNAMIC TRAINING ALGORITHM FOR SECOND ORDER DIAGONAL RECURRENT NEURAL NETWORK W jO u1 y ( k ) = O ( k ) = ∑W Z j ( k ) (4) Z j ( k ) = ρ ( H j ( k )) (5) ≡ y (k ) u2 Fig. 1 shows the structure of the proposed neural network. Mathematical model for this network is given by O j I ij TD ui W jD 2 j H j ( k ) = ∑W u i +W I ij D1 j Z j ( k − 1) +W D2 j Z j ( k − 2 ), qj Fig. 1. Second order diagonal recurrent neural network structure Q ij ( k ) = (7) The gradient of error in equation (7), with respect to an arbitrary weight vector W, is represented by ∂y ( k ) ∂O ( k ) ∂E = −e ( k ) . = −e ( k ) . ∂W ∂W ∂W ∂O ( k ) ∂W jO ∂O ( k ) ∂W jD 1 ∂O ( k ) ∂W jD 2 ∂O ( k ) ∂W ijI = Z j (k ) ∂O ( k ) ∂Z j ( k ) . =W jO .P j ( k ) ∂Z j ( k ) ∂W jD 1 (10) = ∂O ( k ) ∂Z j ( k ) =W jO .G j ( k ) . ∂Z j ( k ) ∂W jD 2 (11) ∂O ( k ) ∂Z j ( k ) . = =W jO .Q ij ( k ) ∂Z j ( k ) ∂W ijI (12) ∂H j ( k ) ∂Z j ( k − 2 )   ∂Z j ( k − 2 ) ∂W ijI  . (15) = ρ ′ ( H j ( k ) ) . (u i +W jD 1.Q ij ( k − 1) +W jD 2 .Q ij ( k − 2 ) ) with the following initial values (9) = ∂Z j ( k )  ∂H j ( k ) ∂H j ( k ) ∂Z j ( k − 1) + . . I  ∂H j ( k )  ∂W ij ∂Z j ( k − 1) ∂W ijI + (8) where e ( k ) = y d ( k ) − y ( k ) is the error between the plant and the network response. The derivatives of the output with respect to the neural network weights are ρ sigmoid function i 2 1 ( y d ( k ) − y ( k )) . 2 TD ρ (6) where ρ ( ⋅) is the sigmoid function. Let y ( k ) and y d ( k ) be the real and desired outputs, respectively. The error cost function is defined by E (k ) = W jD 1 P j ( 0 ) = 0, P j (1) = 0, (16) G j ( 0 ) = 0, G j (1) = 0, (17) Q ij ( 0 ) = 0, Q ij (1) = 0. (18) Therefore, the weights are adjusted by the following equation:  ∂E  W ( n + 1) =W ( n ) + η  −   ∂W  (19) IV. CONVERGENCE AND STABILITY where Pj (k ) = The following theorem is based on reference [27] with some modifications. Theorem1: Let g max := max k g ( k ) ,where g ( k ) = ∂O ( k ) / ∂W , ∂Z j ( k )  ∂H j ( k ) ∂H j ( k ) ∂Z j ( k − 1) + . . D1  ∂H j ( k )  ∂W j ∂Z j ( k − 1) ∂W jD 1 ∂H j ( k ) ∂Z j ( k − 2 )  + .  ∂Z j ( k − 2 ) ∂W jD 1  = ρ ′ ( H j ( k ) ) . ( Z j ( k − 1) +W D1 j .P j ( k − 1) +W D2 j (13) and W is a weight vector composed of all the weight values in SDRNN and ⋅ is the usual Euclidean norm in R n . Then, the convergence of the identifier is guaranteed if ηm is chosen as .P j ( k − 2 ) ) 0 < ηm < ∂Z j ( k )  ∂H j ( k ) ∂H j ( k ) ∂Z j ( k − 1) + . . G j (k ) = D2  ∂H j ( k )  ∂W j ∂Z j ( k − 1) ∂W jD 2 ∂H j ( k ) ∂Z j ( k − 2 )  + .  ∂Z j ( k − 2 ) ∂W jD 2  = ρ ′ ( H j ( k ) ) .( Z j ( k − 2 ) +W jD 1.G j ( k − 1) +W jD 2 .G j ( k − 2 ) ) (14) 2 2 g max (20) Note that ηm changes adaptively during learning process of the network. Proof: Assume there are p inputs in the input layer, q neurons in the hidden layer and one neuron in the output layer. Given a Lyapunov function as 252 1 V (k ) = e 2 (k ) 2 (21) Thus, the change of the Lyapunov function in two consecutive samples due to the training process is obtained by 1 2 e ( k + 1) − e 2 ( k )  2 1   = ∆e ( k ) e ( k ) + ∆e ( k )  2   ∆V ( k ) =V ( k + 1) −V ( k ) = Let λ = η1 ∂e ( k ) ∂e ( k ) ∂e ( k ) 2 + η3 ∂W D 1 2 + η4 ∂W D 2 ∂e ( k ) ∂W O 2 (31) then 1 ∆V ( k ) = − e 2 ( k ) ( 2λ − λ 2 ) . 2 (32) According to the Lyapunov stability theory, if convergence must be guaranteed, then ∆V ( k ) < 0 , thus 0 < λ < 2 , that is consecutive error samples ∆e ( k ) = e ( k + 1) − e ( k ) , which can be defined as 0 < η1  ∂e ( k )  ∆e ( k ) =   ∆W  ∂W  + η2 ∂W I (22) where ∆e ( k ) is defined as the difference between two 2 T ∂e ( k ) ∂W 2 + η2 I ∂e ( k ) ∂W D1 ∂e ( k ) 2 + η3 ∂W 2 + η4 D2 ∂e ( k ) ∂W 2 < 2 (33) O (23) 4 Let ηm = max {ηi } ; thus, as long as Putting all weights into one vector as i =1 T T T T T W =  W I  W D 1  W D 2  W O     (24) η1 in which ∂e ( k ) ∂W I ∂e ( k ) 2 + η2 ∂W D 1 ∂e ( k ) 2 + η3 ∂e ( k ) 2 + η4 ∂W D 2 2 <2 ∂W O (34) T T T T W I =  W1I  W 2I  L W pI     (25) T W D1 T T T =  W1D 1  W 2D 1  L WqD 1     (26) Equation (33) must be satisfied. So the convergence condition can be written as 0 < ηm < T T T T W D 2 =  W1D 2  W 2D 2  L WqD 2     (27) T and W O = W1O  . In (25)-(27) W i Z represents the weight vector corresponding to the ith neuron in the z layer (W I ∈ R pq , (W D 1 , W D 2 , W O ) ∈ R q ). Also, let η I  η=    η D1 ηD2      O η  2 ∂e ( k ) ∂e ( k ) ∂e ( k ) + + ∂W I ∂W D 1 ∂W D 2 2 2 2 ∂e ( k ) + ∂W O (35) 2 Note that ⋅ is the Euclidean norm, therefore ∂e ( k ) ∂W I 2 + ∂e ( k ) ∂W D1 2 + ∂e ( k ) ∂W 2 + D2 ∂e ( k ) ∂W 2 O = ∂e ( k ) ∂W 2 (36) (28) Now let g (k ) = where η I , η D 1 , η D 2 and η O represent the learning rate matrix corresponding to W I , W D 1 , W D 2 and W O , respectively, and η I = η1I I , η D 1 = η 2 I D 1 , η D 2 = η3I D 2 , η O = η 4 I O . Moreover, ηi (i=1, …, 4) is a positive constant, and I Z is the identity matrix with z representing I, D1, D2, and O, respectively. Then η  ∂V ( k ) ∂e ( k ) e ( k ) = −e ( k ) .  ∆W = −η = −η  ∂W ∂W   I η D1 ηD2      O η  ∂W = ∂O ( k ) ∂W and let g max = max k g ( k ) , then (20) follows. Theorem2: Let g max := max k g ( k ) (37) o and S max := max k S ( k ) where g ( k ) = ∂O ( k ) / ∂W and S ( k ) = ∂y ( k ) / ∂u ( k ) = y u ( k ) , and W is a weight vector composed of all the weight in the SDRNN, and ⋅ is the Euclidean norm in R n . Then, the convergence of the controller is guaranteed if η mc is chosen as 0 < η mc < T   ∂e ( k ) T  ∂e ( k ) T  ∂e ( k ) T  ∂e ( k ) T  .          ∂W I   ∂W D 1   ∂W D 2   ∂W O      2 2 2 S max g max (38) Note that S ( k ) is the sensitivity of the plant output with (29)  ∂e ( k )  ∆e ( k ) =   ∆W = −e ( k )  ∂W  ∂e ( k ) respect to its input. T 2  ∂e ( k ) 2 ∂e ( k ) ∂e ( k ) η .η1 + + η3 2  ∂W I ∂W D 1 ∂W D 2  Proof: Same as in Theorem 1, it can be written 2 + η4 ∂e ( k ) ∂W O 2     ∆W = −η y u ( k ) (30) ∂e ( k ) ∂W and the rest of proof is straight forward. 253 e (k ) (39) o Comparison between DRNN and SDRNN identification (Offline) V. COMPARISON BETWEEN SDRNN AND DRNN ON SYSTEM IDENTIFICATION 6 Consider the following plant model Plant SDRNN DRNN 4 y ( k + 1) = 0.2 y ( k ) + 0.2 y ( k − 1) + 0.2 y ( k − 2 ) + sin 0.5 ( y ( k ) + y ( k − 1) + y ( k − 2 ) )  2 (40) Output ⋅ cos 0.5 ( y ( k ) + y ( k − 1) + y ( k − 2 ) )  The proposed neural network in this paper and the DRNN are employed to identify the system in (40). There are one input, 10 neurons in the hidden layer and one output for both networks. Fig. 2 shows the sum of squared error for both networks. Fig. 3 and Fig. 4 show the comparison on the model identification and the adaptive learning rate ( η m ), respectively. 0 -2 -4 -6 0 150 200 250 Sample Fig. 3. Comparison of SDRNN and DRNN in system identification MSE along Epochs 50 100 2 Adaptive learning rate DRNN SDRNN 1.8 0.08 1.6 Learning rate value 1.4 MSE 1.2 1 0.8 0.6 0.06 0.05 0.04 0.03 0.4 0.02 0.2 0.01 0 0 DRNN SDRNN 0.07 500 1000 1500 2000 2500 Epoch Fig. 2. MSE along learning 3000 3500 4000 0 0 500 1000 1500 2000 2500 3000 Epoch Fig. 4. Adaptive learning rate VI. EXPERIMENTAL RESULTS The proposed SDRNN is used to control a submarine periscope mirror. Fig. 5 shows the experimental setup of this system. Fig. 6 shows the schematic diagram of a periscope, where θ1 and θ 2 are the motions imposed from the sea waves on the submarine and the periscope, along the roll and the pitch axes, respectively. It is assumed that gyroscopes measure these angles. Image stabilization equation maps these angles to the three dimensional space. In this paper, the mirror is controlled for the line of sight (LOS) stabilization. Fig. 7 shows the proposed control block diagram. The proposed scheme is based on [12], in which the identifier is replaced by the proposed SDRNN. This identifier estimates the required ∂y / ∂u in the NN controller. Figs. 8 to 10 show the reference tracking on θ 4 and θ5 , and the error between the reference and the plant output, respectively. 254 Fig. 5. Experimental setup of periscope 3500 4000 Tracking error 1.2 1 Mirror θ4 θ5 0.8 Height of the Periscope Camera Degree Mirror micro motors d θ3 Main Servomotor 0.4 0.2 0 θ2 θ1 0.6 -0.2 Periscope movements 0 1 2 3 4 5 6 7 8 Time(sec) Fig. 9. Tracking Error on θ 4 Fig. 6. Schematic of the Periscope Reference Tracking 10 y r (k ) + Track Reference 8 - 6 θ2 Image Stabilization Equation θ 3r θ4r SDRNN controller θ5r 4 u (k ) Plant y ( k ) = θ4 em TD + yu u ( k − 1) y ( k − 1) 2 Degree θ1 ec - SDRNN Identifier 0 -2 -4 -6 y m (k ) -8 TD -10 Fig. 7. Control block diagram Reference Tracking 2 3 4 5 6 7 Time(sec) Fig. 10. Reference and model output response on θ 5 Tracking error 1 2 3 8 0.8 2 0.6 0 0.4 Degree Degree 1 1 Track Reference 4 0 -2 0.2 0 -4 -0.2 -6 -0.4 0 1 2 3 4 5 6 7 Time(sec) Fig. 8. Reference and model output response on θ 4 8 255 0 4 5 Time(sec) Fig. 11. Tracking Error on θ 5 6 7 8 VII. CONCLUSION In this paper, a new diagonal recurrent neural network that contains two recurrent weights for every hidden neuron was proposed. It was shown that using two recurrent weights could help improve the estimation property of recurrent networks while the computation burdens are not as much. After deriving the learning law for this proposed network, convergence stability and adaptive learning rate were presented. The performance of the proposed network in model identification showed the accuracy of this network against the DRNN. At the end, this network was applied to real time control of an experimental image stabilization platform. REFERENCE [1] K. S. Narendra and K. Parthasarathy, “Identification and control of dynamic systems using neural networks,” IEEE Trans. on Neural networks, vol. 1, no. 1, pp. 4-27, march 1990. [2] Y. Zhang and J. Wang, “Obstacle avoidance for kinematically redundant manipulators using a dual neural network,”IEEE Trans. Syst., Man and Cybernetics, vol. 34, no. 1, pp. 752-759, 2004. [3] S. Mukhopadhyay and K. S. Narendra, “Disturbance Regection in Nonlinear Systems Using Neural Networks,” IEEE Trans. Neural Nets., vol. 4, No. 1, pp. 63-72, 1993. [4] D. Rumelhart, G.E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” Parallel Distributet Processing, D. Rumelhart and J. McClelland (Eds.), vol. 1, MIT Press, Cambridge, pp. 318-362, 1986. [5] D. Luongvinh and Y. Kwon, “Behavioral modeling of power amplifiers using fully recurrent neural networks,” IEEE MTT-S International Microwave Symposium Diqest, June 2005. [6] G. Kechriotis and E. S. Manolakos, “Training fully recurrent neural networks with complex weights,” IEEE Trans. Circuits and Systems-II: Analog and Digital Signal Processing, vol. 41, no. 3, pp. 235-238, March 1994. [7] C. C. Ku and K. Y. Lee, “Nonlinear system identification using diagonal recurrent neural networks,” IEEE International Joint Conference on Neural Networks, vol. 3, pp. 839-844, 1992. [8] C. C. Ku and K. Y. Lee, “Diagonal recurrent neural networks for nonlinear system control,” IEEE International Joint Conference on Neural Networks, vol. 1, pp. 315-320, 1992. [9] C. C. Ku and K. Y. Lee, “Diagonal recurrent neural network-based control: convergence and stability,” American Contr. Conf., pp. 3340-3345, 1994. [10] C. C. Ku and K. Y. Lee, “Diagonal recurrent neural networks for controller designs,” IEEE International Forum on Neural Networks to Power Systems, pp. 87-92, 1993. [11] C. C. Ku and K. Y. Lee, “Diagonal recurrent neural network based control using adaptive learning rates,” IEEE Conf. Decision and Contr., pp. 3485-3490, Tucson, Arizona, USA, 1992. [12] C. C. Ku and K. Y. Lee, “Diagonal recurrent neural networks for dynamic systems control,” IEEE Trans. Neural Nets., vol. 6, no. 1, pp. 144-156, 1995. [13] S. Liu and Y. Du, “Sonar array servo system based on diagonal recurrent neural network,” IEEE Intern. Conf. Mechatronics & Automat., pp. 19121917, Niagara Falls, Ontario, Canada, 2005. [14] X. Wang and G. Peng, “Modeling and control for pneumatic manipulator based on dynamic neural network,” IEEE International Conference on Systems, Man and Cybernetics, vol. 3, pp.2231-2236, 2003. [15] X. Wang, G. Peng and Y. Xue, “Internal model controller with diagonal recurrent neural network for pneumatic robot servo system,” IEEE Intern. Symp. Comput. Intelligence in Robotics and Automat., pp. 10641069, Japan, 2003. [16] T. G. Barbounis, J. B. Theocharis, M. C. Alexiadis and P. S. Dokopoulos, “Long-term wind speed and power forecasting using local recurrent neural network models,” IEEE Trans. Energy Conversion, vol. 21, no. 1, pp. 273-284, 2006. [17] F. Shaosheng and X. Hui, “Diagonal recurrent neural network based predictive control for active power filter,” Intern. Conf. Power Syst. Tech. – POWERCON, pp. 759-762, Singapore, 2004. [18] R. T. Bambang, R. R. Yacoub and K. Uchida, “Identification of secondary path in ANV using diagonal recurrent neural networks with EKF algorithm,” 5th Asian Contr. Conf., pp. 665-673, Melbourne, Australia, 2004. [19]C. C. Ku, K. Y. Lee and R. M. Edwards, “Improved nuclear reactor temperature control using diagonal recurrent neural nerworks,” IEEE Trans. Nuclear Science, vol. 39, no. 6, pp. 2298-2308, 1992. [20] B. Jayawardhana, L. Xie and S. Yuan, “Active control of sound based on diagonal recurrent neural network,” Conference of the Society of Instrument and Control Engineers (SICE), pp. 2666-2671, Osaka, 2002. [21] J. S. Cho, Y. W. Kim and D. J. Park, “Identification of nonlinear dynamic systems using higher order diagonal recurrent neural network,” IEE Electron.Letters Online, vol. 33, no. 25, pp. 2133-2135, 1997. [22] P. A. Mastorocostas and J. B. Theocharis, “On stable learning of blockdiagonal recurrent neural networks, part I: the RENNCOM algorithm,” IEEE International Joint Conference on Neural Networks, vol. 2, pp. 815-820, 2004. [23] P. A. Mastorocostas and J. B. Theocharis, “On stable learning of blockdiagonal recurrent neural networks, part II: application to the analysis of lung sounds,” IEEE International Joint Conference on Neural Networks, vol. 2, pp. 821-826, 2004. [24] P. A. Mastorocostas and J. B. Theocharis, “A stable learning algorithm for block-diagonal recurrent neural networks: application to the analysis of lung sounds,” IEEE Trans. Syst., Man, Cybern. – Part B: Cybernetics, vol. 36. no. 2, pp. 242-254, 2006. [25] S. C. Sivakumar, W. Robertson and W. J. Philips, “On-line stabilization of block-diagonal recurrent neural networks,” IEEE Trans. Neural Nets., vol. 10, no. 1, pp. 167-175, 1999. [26] P. A. Mastorocostas, D. Varsamis, C. Mastorocostas and I. Rekanos, “An accelerating learning algorithm for block-diagonal recurrent neural networks,” Intern. Conf. Comput. Intelligence for Modelling, Contr. and Automat., Vienna, AUSTRIA, 2005. [27] P. Wang, Y. Li, S. Feng and W. Wei, “Convergence of diagonal recurrent neural networks' learning,” 4th Congress on Intelligent Contr. and Automat., pp. 2365-2369, Shanghai, China, 2002. 256