Direct Error-Driven Learning for Deep Neural Networks with Applications to Big Data

Abstract

In this brief, heterogeneity and noise in big data are shown to increase the generalization error for a traditional learning regime utilized for deep neural networks (deep NNs). To reduce this error, while overcoming the issue of vanishing gradients, a direct error-driven learning (EDL) scheme is proposed. First, to reduce the impact of heterogeneity and data noise, the concept of a neighborhood is introduced. Using this neighborhood, an approximation of generalization error is obtained and an overall error, comprised of learning and the approximate generalization errors, is defined. A novel NN weight-tuning law is obtained through a layer-wise performance measure enabling the direct use of overall error for learning. Additional constraints are introduced into the layer-wise performance measure to guide and improve the learning process in the presence of noisy dimensions. The proposed direct EDL scheme effectively addresses the issue of heterogeneity and noise while mitigating vanishing gradients and noisy dimensions. A comprehensive simulation study is presented where the proposed approach is shown to mitigate the vanishing gradient problem while improving generalization by 6%.

Department(s)

Mathematics and Statistics

Research Center/Lab(s)

Center for High Performance Computing Research

Second Research Center/Lab

Intelligent Systems Center

Keywords and Phrases

Error-driven; exploratory learning; generalization error; neural network

International Standard Serial Number (ISSN)

2162-237X

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2020 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

01 May 2020

PubMed ID

31329564

Link to Full Text

Share

 
COinS