A New Deep Neural Network Based on Multi-Layer Echo State Network

Title:A New Deep Neural Network Based on Multi-Layer Echo State Network

Volume: 11 Issue: 1

Author(s): Ronghui Liu*Junmin Zhao

Affiliation:

• School of Computer and Data Science, Henan University of Urban Construction, Pingdingshan, 467036,China

Keywords: Echo state network, auto-encoder, penalty factor, deep neural network, pattern identification, back propagation.

Abstract: Background: Deep Neural Network (DNN) has attracted great attention in regression and classification problems. However, the traditional DNN fails to provide favorable prediction performance owing to their inherent shortcomings of the Back Propagation (BP) algorithm, such as slow convergence and local optimum. To solve this problem, a novel DNN algorithm called Multi-Layer Echo State Network (ML-ESN) is proposed.

Method: This method utilizes Echo State Network (ESN) based unsupervised and supervised learning process. After discussing some related patents and methods, the ML-ESN structure is given. In the ML-ESN with M+1 hidden layers, ESN based Auto-Encoder (ESN-AE) is employed by the front M hidden layers for feature extraction in the unsupervised learning process, which can effectively extract abstract features from the original data. To overcome the under-fitting and over-fitting problem of ESN based classifier, the Penalty Factor ESN (PF-ESN) is presented to act as classifier by the M+1th hidden layer in the supervised learning process.

Results: The experiments demonstrate that ML-ESN outperforms ESN, Tikhonov regularization ESN (TR-ESN) and some DNN models such as Deep Belief Network (DBN) and stacked auto encoder (SAE). Owing to the fast learning speed of ESN, the training time consumtion of ML-ESN is also shorter than DBN and SAE.

Conclusion: The method provides references for fast and high accuracy classification and feature extraction.

Cite this article as:

Liu Ronghui*, Zhao Junmin, A New Deep Neural Network Based on Multi-Layer Echo State Network, Recent Patents on Computer Science 2018; 11 (1) . https://dx.doi.org/10.2174/2213275911666180507112411