Comparison of Four Neural Network Learning Methods Based on Genetic Algorithm for Non-linear Dynamic Systems Identification

Non-linear dynamical systems are difficult to control due to the model uncertainties and external disturbances that may occur in these systems. This paper addresses the problem of identification using dynamic neural networks (DNNs) based on genetic algorithm (GA) for nonlinear dynamic systems. Four different dynamic neural networks are used for identification of the same nonlinear dynamic system, using the genetic algorithm (GA) to train the LayerRecurrent Network (LRN), Focused Time-Delay Neural Network (FTDNN), the Elman Network, and Nonlinear Autoregressive Network with Exogenous inputs (NARX). The simulation results show the generalization ability of the four dynamic neural networks which provide the high precision of model of the nonlinear dynamic system. Also this paper illustrates the advantages and disadvantages of the different dynamic neural networks trained by GA.


Introduction
Artificial neural networks (ANNs) are the computing architecture that consists of massively parallel interconnections of simple computing elements.ANNs have been implemented in various fields such as, image processing, speech recognition, and system identification etc.The fundamental and core property of ANNs for their superiority over other approximation methods are based on the fact that ANNs are able to universally approximate smooth but otherwise arbitrary nonlinear functions on a wide range of complex nonlinear functions on a compact set, using fewer parameters and requiring less computation time [1].
The ANNs are commonly used as a data-based technique in performing non-linear system identification of complex processes.For this purpose, models with adequate memory are required.Therefore, the ANNs have to be provided with dynamic elements and appropriate learning methods [2].A first approach refers to neural networks with external dynamics, e.g.static ANNs equipped with tapped delay lines, the latter increasing the dimensions of the ANN's input space.A better approach is achieved by providing the ANN with internal dynamics.This kind of network processes multi-inputs and does not require past values of process measurements as current inputs [3].There are several ways to deal with nonlinear control design for plants subject to uncertainty and disturbances; among them are robust and adaptive control methods.While neural adaptive control is being intensively developed, so as the applications of neural networks to nonlinear control [2].For example, Narendra et.al. [4], introduced multilayer neural networks for identification and adaptive control of nonlinear systems.A number of studies such as [5][6][7][8][9][10], presented an adaptive control of unknown feedback linearizable system for achieving guaranteed performance of the neural network.It has excellent capabilities of nonlinear mapping, learning ability, and parallel computations.
This paper first provides background information on Dynamic Neural Networks architectures, learning algorithm based on genetic algorithm (GA), and a theoretical basis for their use.Then it presents simulations for these architectures for identifying nonlinear dynamic systems or plant.Finally, it shows some new results and discussion for these results.

Dynamic Neural Networks (DNNs) for Systems Identification
Artificial neural networks (ANNs) can be classified into dynamic and static categories.Static feed-forward networks have no feedback elements and contain no delays; the output is calculated directly from the input through feed-forward connections.In dynamic networks, the output depends not only on the current input to the network, but also on the current or previous inputs, outputs, or states of the network [11].Dynamic networks can also be divided into two categories: those that have only feed forward connections, and those that have feedback, or recurrent, connections.Dynamic networks are generally more powerful than static networks (although slightly more difficult to train).Because dynamic networks have memory, they can be trained to learn sequential or time-varying patterns.This has applications in such different areas as channel equalization in communication systems, phase detection in power systems, fault detection, speech recognition [3].ANNs are used to model the dynamics of a system or plant.Usually the ANN and the plant are feed with the same input signal (or input signals for multi-input systems) for training, then neural network used to model the plant behavior [12].The network output is compared with the output from the plant and the error is used to update the weights of the synapses (see Fig. 1)..A simple feed-forward neural network can be used, if the ANN is feed with the current input, the previous inputs and the previous outputs.Using this configuration an input-output model of the plant is obtained, instead of a state space model.The advantage of this model is that a series-parallel model can be used while training the network.In a series-parallel model (Fig. 2) the previous outputs from the plant are used as inputs to the ANN instead of the outputs from the proper neural network.
Once the model is obtained then the input scan is taken from the previous outputs of the ANN obtaining and autonomous system.
This work demonstrates how to create, train ANN based on GA, and apply certain dynamic neural networks in identification nonlinear dynamic system.Some of the networks require dynamic back propagation for computing the gradients and others do not.This section, presents the following four dynamic neural networks.

a. Layer-Recurrent Network (LRN)
The dynamic network to be introduced is the Layer-Recurrent Network (LRN) [5]

b. Focused Time-Delay Neural Network (FTDNN)
The next network is the most straightforward dynamic network, which consists of a feed-forward network with a tapped delay line at the input.This is called the focused time-delay neural network (FTDNN) [5].This is part of a general class of dynamic networks, called focused networks, in which the dynamics appear only at the input layer of a static multilayer feed-forward network.The following Fig. ( 4), illustrates a twolayer FTDNN.

c. Elman Network
The Elman network commonly is a two-layer network with feedback from the firstlayer output to the first-layer input [5][6].This recurrent connection allows the Elman network to both detect and generate time-varying patterns.A two-layer Elman network is shown Fig. (5).The Elman network has tansig neurons in its hidden (recurrent) layer, and purelin neurons in its output layer.This combination is special in that two-layer networks with these transfer functions can approximate any function (with a finite number of discontinuities) with arbitrary accuracy.The only requirement is that the hidden layer must have enough neurons.More hidden neurons are needed as the function being fitted increases in complexity.

d. NARX Network
The nonlinear autoregressive network with exogenous inputs (NARX) is a recurrent dynamic network, with feedback connections enclosing several layers of the network [5,13].The NARX model is based on the linear ARX model.The defining equation for the NARX model is: where a two-layer feedforward network is used for the approximation.This implementation also allows for a vector ARX model, where the input and output can be multidimensional.You can consider the output of the NARX network to be an estimate of the output of some nonlinear dynamic system that you are trying to model.The output is fed back to the input of the feed-forward neural network as part of the standard NARX architecture, as shown in the Fig. (7a).because the true output is available during the training of the network, you could create a series-parallel architecture [5,14] ], in which the true output is used instead of feeding back the estimated output, as shown in the Fig. (7b).This has two advantages.The first is that the input to the feedforward network is more accurate.The second is that the resulting network has a purely feed-forward architecture, and backpropagation (BP) can be used for training [12].

Learning Algorithm of DNN
There are some researches in the dynamic neural network (DNN).Such as the static ANN with integrator, NARX network, Jordan network [3].The learning algorithm of the recurrent neural network (RNN) is usually the backpropagation (BP) algorithm.
Because it tends to convergent to the local minimum and flexibility is not so good.

Simulation Results of DNN
In this simulation, the nonlinear dynamic plant to be identified is expressed as follows: (5) The train signal is the random sequence which is submitted to the uniform range between 1 and -1, and the test input chosen:

Results Discussion
In all four cases, the normalized MSE, between the desired output and the output from the network, decreases, while learning, to mean squared error

Conclusion
The use of DNNs have shown encouraging results, and been useful for the identification of nonlinear dynamic systems.As classical methodologies, our structure allows the extraction of a mathematical model of the system, but it adds the advantages of GA with neural networks to learn and generalize.The training time of the four DNN is shorter and the convergence speed is faster.At the same time have high precision identification.Although the training error of the LRN network is the smallest among the four DNNs , the generalization ability of the Elman network and the NARX network are better than the FTDNN , and the LRN networks.The LRN, the NARX and the Elman network can depict the complex dynamic system due to the back-forward in the DNN.So the order of the system need not to be set in advance.Because the FTDNN network is the feed-forward multilayer NN.The order of the system will be set before the training.
[9].An earlier simplified version of this network was introduced by Elman[6].In the LRN, there is a feedback loop, with a single delay, around each layer of the network except for the last layer.The original Elman network had only two layers, and used a tansig transfer function for the hidden layer and a purelin transfer function for the output layer.The network is trained using an approximation to the BP gradient-based algorithm.The network has an arbitrary number of layers and to have arbitrary transfer functions in each layer.Fig.(3) Illustrates a two-layer LRN (according to Matlab abbreviated notation[5]).

Fig. ( 5 )
Fig. (5) Elman network architecture.Note that the Elman network differs from conventional two-layer networks in that the first layer has a recurrent connection.The delay in this connection stores values from the previous time step, which can be used in the current time step.Thus, even if two Elman networks, with the same weights and biases, are given identical inputs at a given time step, their outputs can be different because of different feedback states.Because the network can store information for future reference, it is able to learn temporal patterns as well as spatial patterns.The Elman network can be trained to respond to, and to generate, both kinds of patterns.The default BP training algorithm is used.You might use BP, but it tends to proceed so rapidly that it does not necessarily do well in the Elman network.The backpropagation weight/bias learning function default is, and the default performance function is mean squared error (mse).When the network is created, each layer's weights and biases are initialized with the Nguyen-Widrow layer initialization method.

1
The stop criteria of the algorithm is tags the optimization solution is not evolution after 30 generation.The sketch map is shown in Fig.(8).

( 6 )
The data was collected at a sampling interval of 1.0 seconds to for four time series.The result is displayed in the following plot.You can see that the errors are very small.The structure of LRN is decided by modeling.There is one node in the input layer, five nodes in the hidden layer, and one node in the output layer in this research.The simulation figure of the testing is shown in Fig.(9):
(MSE) values between 0.01 and 0.005.If when tested, the error increases in an important way then more coefficients were added and also new training signals were used.In Fig 9 to Fig 12 the outputs from the real system (black trace) and the outputs from the trained ANN (blue trace) are compared.In table 1 , it can be known that all four kinds of DNNs can provide the high precesion of model of nonlinear dynamic system after the training based on the GA.When the test signal is used to test the model, there are different test error in these DNNs.The test errors of the LRN and FTDNN networks are the smaller.But they are bigger in the Elman and NARX neural networks.The train time of LRN is the shortest.

Table 1 .
Performance Comparison of the four DNNs