Abstract
It is well known that the estimator that minimizes the mean square error loss is the mean. However, the latter is prune to outliers and additionally, in many practical use cases, such as regression confidence bounds for risk, one would like to predict other statistics such as the median, or a certain percentile. In this chapter we will introduce the concept of Quantile Regression, a method based on a unique loss function which allows predicting quantiles of the conditional distribution of the target given the input. Albeit the method is useful and the loss is differential, thus fits modern deep neural network approach, most data science practitioners are not aware for its existence. We hope this chapter will reveal the theory and the know-hows of this methods and will allow building better machine learning systems. In addition, we discuss some recently developed approaches of how to optimize a general loss function under a quantile constraint.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abeywardana, S.: Deep quantile regression. https://towardsdatascience.com/deep-quantile-regression-c85481548b5a. Accessed 19 Sept 2021
Bottou, L.: Stochastic gradient descent tricks. In: Neural networks: Tricks of the trade, pp. 421–436. Springer, Berlin (2012)
Cannon, A.J.: Quantile regression neural networks: Implementation in r and application to precipitation downscaling. Comput. Geosci. 37(9), 1277–1284 (2011)
Chiu, C.-H., Wen, T.-H., Chien, L.-C., Yu, H.-L.: A probabilistic spatial dengue fever risk assessment by a threshold-based-quantile regression method. PloS One 9(10), e106334 (2014)
Elad, M.: Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, Berlin (2010)
Evangelou, M., Adams, N.M.: An anomaly detection framework for cyber-security data. Comput. Secur. 97, 101941 (2020)
Fazlyab, M., Robey, A., Hassani, H., Morari, M., Pappas, G.: Efficient and accurate estimation of Lipschitz constants for deep neural networks. In: Advances in Neural Information Processing Systems, pp. 11427–11438 (2019)
Gass, S.I.: Linear Programming: Methods and Applications. Courier Corporation (2003)
He, Y., Qin, Y., Wang, S., Wang, X., Wang, C.: Electricity consumption probability density forecasting method based on lasso-quantile regression neural network. Appl. Energy 233, 565–575 (2019)
He, Y., Xu, Q., Wan, J., Yang, S.: Short-term power load probability density forecasting based on quantile regression neural network and triangle kernel function. Energy 114, 498–512 (2016)
Koenker, R., Hallock, K.F.: Quantile regression. J. Econ. Perspect. 15(4), 143–156 (2001)
Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1), 503–528 (1989)
Meinshausen, N., Ridgeway, G.: Quantile regression forests. J. Mach. Learn. Res. 7(6) (2006)
Or, D.B., Kolomenkin, M., Shabat, G.: DL-DDA–Deep learning based dynamic difficulty adjustment with ux and gameplay constraints. In: IEEE Conference on Games (2021). https://ieee-cog.org/2021/assets/papers/paper_142.pdf
Pradeepkumar, D., Ravi, V.: Forecasting financial time series volatility using particle swarm optimization trained quantile regression neural network. Appl. Soft Comput. 58, 35–52 (2017)
Rodrigues, F., Pereira, F.C.: Beyond expectation: deep joint mean and quantile regression for spatiotemporal problems. In: IEEE Transactions on Neural Networks and Learning Systems (2020)
Ruder, S.: An overview of gradient descent optimization algorithms (2016). arXiv preprint arXiv:1609.04747
Silverman, B.W.: Some aspects of the spline smoothing approach to non-parametric regression curve fitting. J. R. Stat. Soc. Series B (Methodological) 47(1), 1–21 (1985)
Taylor, J.W.: A quantile regression neural network approach to estimating the conditional density of multiperiod returns. J. Forecasting 19(4), 299–311 (2000)
Wang, H., Ruan, J., Zhou, B., Li, C., Wu, Q., Raza, M.Q., Cao, G.-Z.: Dynamic data injection attack detection of cyber physical power systems with uncertainties. IEEE Trans. Ind. Inform. 15(10), 5505–5518 (2019)
Xu, Q., Deng, K., Jiang, C., Sun, F., Huang, X.: Composite quantile regression neural network with applications. Expert Syst. Appl. 76, 129–139 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Or, D.B., Kolomenkin, M., Osokin, T., Shabat, G., Shteingart, H. (2023). Deep Learning Quantile Regression for Robustness, Confidence and Planning. In: Sipola, T., Kokkonen, T., Karjalainen, M. (eds) Artificial Intelligence and Cybersecurity. Springer, Cham. https://doi.org/10.1007/978-3-031-15030-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-15030-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15029-6
Online ISBN: 978-3-031-15030-2
eBook Packages: Computer ScienceComputer Science (R0)