Direct Error-Driven Learning for Deep Neural Networks with Applications to Big Data
In this brief, heterogeneity and noise in big data are shown to increase the generalization error for a traditional learning regime utilized for deep neural networks (deep NNs). To reduce this error, while overcoming the issue of vanishing gradients, a direct error-driven learning (EDL) scheme is proposed. First, to reduce the impact of heterogeneity and data noise, the concept of a neighborhood is introduced. Using this neighborhood, an approximation of generalization error is obtained and an overall error, comprised of learning and the approximate generalization errors, is defined. A novel NN weight-tuning law is obtained through a layer-wise performance measure enabling the direct use of overall error for learning. Additional constraints are introduced into the layer-wise performance measure to guide and improve the learning process in the presence of noisy dimensions. The proposed direct EDL scheme effectively addresses the issue of heterogeneity and noise while mitigating vanishing gradients and noisy dimensions. A comprehensive simulation study is presented where the proposed approach is shown to mitigate the vanishing gradient problem while improving generalization by 6%.
R. Krishnan et al., "Direct Error-Driven Learning for Deep Neural Networks with Applications to Big Data," IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 5, pp. 1763 - 1770, Institute of Electrical and Electronics Engineers (IEEE), May 2020.
The definitive version is available at https://doi.org/10.1109/TNNLS.2019.2920964
Mathematics and Statistics
Center for High Performance Computing Research
Second Research Center/Lab
Intelligent Systems Center
Keywords and Phrases
Error-driven; exploratory learning; generalization error; neural network
International Standard Serial Number (ISSN)
Article - Journal
© 2020 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
01 May 2020