Configurable Model for Sigmoid and Hyperbolic Tangent Functions

Khaled Salah, Mona Safar, Mohamed Taher, Ashraf Salem

Abstract

Recurrent neural networks (RNNs) are considered to be among the most important types of neural networks especially for the applications where processing of a sequence of data comes to place. RNNs are in general computationally expensive and need a lot of processing time and power. Therefore, there is a strong need to reduce the processing time to be able to use them in an embedded environment with limited resources. In this work, we present an accelerated field programmable gate array (FPGA) model for RNNs with an emphasis on long short-term memory neural networks (LSTMs). A new configurable block capable of calculating Tanh and Sigmoid activation functions is proposed and analyzed. The solution is based on a look-up table and additional simple math operations, which leads to a speedup of the proposed model of the neural network. Results are obtained and compared with other work by the simulation tool ISE Xillinx.

Keywords

Activation functions; Deep learning; FPGA; LSTM; Recurrent neural network.

Article Metrics

Abstract view : 339 times
PDF - 89 times

Full Text:

PDF

References

S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Computation, 9(8), 1997, 1735-1780.

Zachary Chase Lipton, A critical review of recurrent neural networks for sequence learning, arXiv: 1506.00019, 2015.

Shiv Ram Dubey, Satish Kumar Singh and Bidyut Baran Chaudhuri, Activation functions in deep learning: A comprehensive survey and benchmark, Neurocomputing, 503, 2022, 92-108.

A. D. Rasamoelina, I. Cík, P. Sincak, M. Mach and L. Hruška, A large-scale study of activation functions in modern deep neural network architectures for efficient convergence, Inteligencia Artificial, 25(70), 2022, 95-109.

A. D. Jagtap and G. E. Karniadakis, How important are activation functions in regression and classification? A survey, performance comparison,and future directions, arXiv: 2209.02681v6, 2022.

Tomasz Szandała, Review and comparison of commonly used activation functions for deep neural networks, Bio-inspired Neurocomputing, 2021, 203-224.

B. Liang, S. Wang, Y. Huang, Y. Liu and L. Ma, F-LSTM: FPGA-based heterogeneous computing framework for deploying LSTM-based algorithms, Electronics, 12, 2023, 1139.

S. Wang, Z. Li, C. Ding, B. Yuan, Q. Qiu, Y. Wang and Y. Liang, C-LSTM: Enabling efficient LSTM using structured compression techniques on FPGAs, ACM/SIGDA International Symposium on Field-programmable Gate Arrays, California, USA, 2018.

W. Meiqi, W. Zhisheng, L. Jinming, L. Jun and W. Zhongfeng, E-LSTM: An efficient hardware architecture for long short-term memory, IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(2), 2019, 280-291.

T. Mealey and T. M. Taha, Accelerating inference in long short-term memory neural networks, IEEE National Aerospace and Electronics Conference, Dayton, USA, 2018, 382-390.

S. Wang, P. Lin, R. Hu, H. Wang, J. He, Q. Huang and S. Chang, Acceleration of LSTM with structured pruning method on FPGA, IEEE Access, 7, 2019, 62930-62937.

J. He, D. He, Y. Yang, J. Liu, J. Yang and S. Wang, An LSTM acceleration engine for FPGAs based on caffe framework, IEEE 5th International Conference on Computer and Communications (ICCC), Chengdu, China, 2019.

K. Basterretxea, J. M. Tarela and I. Del Campo, Approximation of sigmoid function and the derivative for hardware implementation of artificial neurons, IEE Proceedings - Circuits, Devices and Systems, 151(1), 2004, 18-24.

K. Chen, L. Huang, M. Li, X. Zeng and Y. Fan, A compact and configurable long short-term memory neural network hardware architecture, 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018, 4168-4172.

J. C. Ferreira and J. Fonseca, An FPGA implementation of a long short-term memory neural network, IEEE International Conference on ReConFigurable Computing and FPGAs (ReConFig), Cancun, Mexico, 2016, 1-8.

A. H. Namin, K. Leboeuf, R. Muscedere, H. Wu and M. Ahmadi, Efficient hardware implementation of the hyperbolic tangent sigmoid function, IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, 2009.

C. W. Lin and J. S. Wang, A digital circuit design of hyperbolic tangent sigmoid function for neural networks, Proceedings of IEEE International Symposium on Circuits and Systems, Seattle, USA, 2008, 856-859.

J. Schmidhuber, Deep learning in neural networks: An overview, Neural Networks, 61, 2015, 85-117.

F. A. Gers and J. Schmidhuber, Recurrent nets that time and count, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Network (IJCN-N), Como, Italy, III, 2000, 189-194.

K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, LSTM: A search space odyssey, IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2017, 2222-2232.

Refbacks

  • There are currently no refbacks.