Distributed Learning of Deep Sparse Neural Networks for High-Dimensional Classification
Abstract
While analyzing high dimensional data-sets using deep neural network (NN), increased sparsity is desirable but requires careful selection of "sparsity parameters." In this paper, a novel distributed learning methodology is proposed to optimize the NN while addressing this challenge. To address this challenge, the optimal sparsity in the NN is estimated via a two player zero-sum game in the paper. In the proposed game, sparsity parameter is the first player with the aim of increasing sparsity in the NN while NN weights is the second player with the goal of improving its performance in the presence of increased sparsity. To solve the game, additional variables are introduced into the optimization problem such that the output at every layer in the NN depends on this variable instead of the previous layer. Using these additional variables, layer wise cost-functions are derived that are then independently optimized to learn the additional variables, NN weights and the sparsity parameters. To implement the proposed learning procedure in a parallelized and distributed environment, a novel computational algorithm is also proposed. The efficiency of the proposed approach is demonstrated using a total of six data-sets.
Recommended Citation
S. Garg et al., "Distributed Learning of Deep Sparse Neural Networks for High-Dimensional Classification," Proceedings of the 2018 IEEE International Conference on Big Data (2018, Seattle, WA), pp. 1587 - 1592, Institute of Electrical and Electronics Engineers (IEEE), Dec 2018.
The definitive version is available at https://doi.org/10.1109/BigData.2018.8621888
Meeting Name
2018 IEEE International Conference on Big Data, Big Data 2018 (2018: Dec. 10-13, Seattle, WA)
Department(s)
Electrical and Computer Engineering
Second Department
Mathematics and Statistics
Research Center/Lab(s)
Intelligent Systems Center
Second Research Center/Lab
Center for High Performance Computing Research
Keywords and Phrases
Big data; Clustering algorithms; Cost functions; Computational algorithm; Distributed environments; Distributed learning; High dimensional data; Learning procedures; Optimal sparsities; Optimization problems; Sparse neural networks; Deep neural networks
International Standard Book Number (ISBN)
978-1-5386-5035-6
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2018 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
01 Dec 2018
Comments
This research was supported in part by an NSF I/UCRC award IIP 1134721 and Intelligent Systems Center.