Abstract
The paper aims to introduce and illustrate a novel kind of nonlinearity to be used as an activation function for the neurons of a Neural Network (NN). It is shown how the proposed function can be regarded as a generalization of the standard sigmoid. For this reason, it is referred to as quasi-sigmoid. The features of a backpropagation algorithm based on quasi-sigmoids are illustrated and commented. The properties of quasi-sigmoidal networks are compared to basic sigmoidal models on benchmark test cases as well as on sample function approximation problems in electromagnetics. The performance of quasi-sigmoidal networks as function approximators are shown to be generally superior to sigmoidal one by reason of the flexibility introduced and/or in terms of degrees of freedom of the model. As a by-product, the proposed activation function allows to carry out data-driven detections of nonlinearity in the data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Fahlman, S. E. (1989). Faster-learning variations on back-propagation: An empirical study. In D. Touretzky, G. Hinton, & T. Sejnowski (Eds.), Proceedings of 1988 Connectionist Models Summer School (pp. 38–51). San Marco, CA: Morgan Kaufmann.
Kruschke, J. K., & Movellan, J. R. (1991). Benefits of gain: Speeded learning and minimal hidden layers in back-propagation networks. IEEE Transactions, on Systems, Man, and Cybemetics, 21(1), 273–280.
Humpert, B. K. (1994). Improving back propagation with a new error function. Neural Networks, 7, 1191–1192.
Kung, S. Y. (1993). Digital neural networks. Prentice Hall.
Azzerboni, B., Finocchio, G., Ipsale, M., La Foresta, F., & Morabito, F. C. (2002). A New approach to the detection of muscle activation by ICA and wavelet transform. Lecture Notes in Computer Science, Springer-Verlag, Berlin-Heidelberg, 2846, 109–116.
Morabito, F. C. (2000). Independent component analysis and feature extraction for NDT data. Materials Evaluation, 58(1), 85–92.
Baldi, P. F., & Hornik, K. (1995). Learning in linear neural networks: A survey. IEEE Transactions on Neural Networks, 6(4), 837–858.
Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2, 359–366.
Coccorese, E., Martone, R., & Morabito, F. C. (1994). Identification of non-circular plasma equilibria using neural network approach. Nuclear Fusion, 34(10), 1349–1363.
Campolo, M., La Foresta, F., Labate, D., Morabito, F. C., Lay-Ekuakille, A., & Vergallo, P. (2011). ECG-derived respiratory signal using empirical mode decomposition. In Medical Measurements and Applications Proceedings (MeMeA), 2011 IEEE International Workshop on, IEEE, Piscataway (NJ) (Vol. 1, pp. 399–403).
Morabito, F. C. (1996) Function modeling in sigmoidal and quasi-sigmoidal backpropagation. In Proceedings of the World Congress on Neural Networks, WCNN’96, San Diego, CA (pp. 229–233). New York: Lawrence Erlbaum Associates.
Morabito, F. C. (1995, November). Equilibrium parameters recovery for experimental data in Asdex-Upgrade elongated plasmas. In Proceedings of the 1995 IEEE International Conference on Neural Networks, Perth, Western Australia (Vol. 2, pp. 937–941). New York, NY, USA: IEEE Press.
Dubey, S. R., Kumar Singh, S., & Chaudhuri, B. B. A comprehensive survey and performance analysis of activation functions in Deep Learning. arXiv:2109.14545.
Molina, A., Schramowski, P., & Kersting, K. P. (2020). Activation units: End-to-end learning of flexible activation functions in deep networks. In International conference on learning representations.
Macedo, D., Zanchettin, C., Oliveira, A. L., & Ludermir, T. (2019). Enhancing batch normalized convolutional networks using displaced rectifier linear units: A systematic comparative study. Expert Systems with Applications, 124, 271–328.
Acknowledgements
This theoretical contribution is presented in honor of Professor Janusz Kacprzyk, who has been working for many decades on finding suitable engineering representations for inverse problems mainly based on fuzzy logic and the related membership functions. The engineering community working on computational intelligence is grateful to him for spreading knowledge and friendship.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Morabito, F.C., Campolo, M., Ieracitano, C. (2023). On the Use of Quasi-Sigmoids in Function Approximation Problems with Neural Networks. In: Kondratenko, Y.P., Kreinovich, V., Pedrycz, W., Chikrii, A., Gil-Lafuente, A.M. (eds) Artificial Intelligence in Control and Decision-making Systems. Studies in Computational Intelligence, vol 1087. Springer, Cham. https://doi.org/10.1007/978-3-031-25759-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-25759-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25758-2
Online ISBN: 978-3-031-25759-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)