research-article

Vulnerability severity scoring and bounties: why the disconnect?

Authors:

Nuthan Munaiah,

Andrew MeneelyAuthors Info & Claims

SWAN 2016: Proceedings of the 2nd International Workshop on Software Analytics

Pages 8 - 14

https://doi.org/10.1145/2989238.2989239

Published: 13 November 2016 Publication History

Abstract

The Common Vulnerability Scoring System (CVSS) is the de facto standard for vulnerability severity measurement today and is crucial in the analytics driving software fortification. Required by the U.S. National Vulnerability Database, over 75,000 vulnerabilities have been scored using CVSS. We compare how the CVSS correlates with another, closely-related measure of security impact: bounties. Recent economic studies of vulnerability disclosure processes show a clear relationship between black market value and bounty payments. We analyzed the CVSS scores and bounty awarded for 703 vulnerabilities across 24 products. We found a weak (Spearman's ρ = 0.34) correlation between CVSS scores and bounties, with CVSS being more likely to underestimate bounty. We believe such a negative result is a cause for concern. We investigated why these measurements were so discordant by (a) analyzing the individual questions of CVSS with respect to bounties and (b) conducting a qualitative study to find the similarities and differences between CVSS and the publicly-available criteria for awarding bounties. Among our findings were that the bounty criteria were more explicit about code execution and privilege escalation whereas CVSS makes no explicit mention of those. We also found that bounty valuations are evaluated solely by project maintainers, whereas CVSS has little provenance in practice.

References

[1]

Coders’ Rights Project Vulnerability Reporting FAQ. https://www.eff.org/issues/coders/ vulnerability-reporting-faq. Accessed: 2016-04-03.

[2]

Common Vulnerability Scoring System (CVSS-SIG). https://www.first.org/cvss. Accessed: 2016-03-12.

[3]

Severity Ratings - Red Hat Customer Portal. https://access.redhat.com/security/updates/ classification. Accessed: 2016-08-14.

[4]

IEEE Standard Classification for Software Anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993), pages 1–25, Jan 2010.

[5]

A. Algarni and Y. Malaiya. Software Vulnerability Markets: Discoverers and Buyers. Int. Journal Computer, Information, Systems and Control Engineering, 8(3):449–459, 2014.

[6]

L. Allodi and F. Massacci. Comparing Vulnerability Severity and Exploits Using Case-Control Studies. Trans. Inf. Syst. Secur., 17(1):1:1–1:20, Aug 2014.

Digital Library

[7]

N. Bettenburg and S. Just. What Makes a Good Bug Report? In 16th Int’l Symp. Foundations of Software Engineering, pages 308–318, 2008.

Digital Library

[8]

R. Böhme. Vulnerability Markets: What is the economic value of a zero-day exploit? Chaos Communication Congress, pages 27–30, Dec 2005.

[9]

R. Böhme. A Comparison of Market Approaches to Software Vulnerability Disclosure. Emerging Trends in Information and Communication Security, 3995:298–311, 2006.

Digital Library

[10]

M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker. Beyond Heuristics: Learning to Classify Vulnerabilities and Predict Exploits. In Proc. 16th Int. Conf. Knowledge Discovery and Data Mining, pages 105–114, New York, NY, USA, 2010. ACM.

Digital Library

[11]

H. Cavusoglu, H. Cavusoglu, and S. Raghunathan. Emerging Issues in Responsible Vulnerability Disclosure. In 4th Workshop Economics of Information Security, pages 1–31, 2004.

[12]

M. Finifter, D. Akhawe, and D. Wagner. An Empirical Study of Vulnerability Rewards Programs. In 22nd Security Symp., pages 273–288, Washington, D.C., 2013.

Digital Library

[13]

USENIX.

[14]

S. Frei and F. Artes. International Vulnerability Purchase Program. Technical report, NSS Labs, 2013.

[15]

S. Frei, D. Schatzmann, B. Plattner, and B. Trammell. Modeling the Security Ecosystem - The Dynamics of (In)Security. pages 79–106, 2010.

[16]

K. Kannan and R. Telang. Market for Software Vulnerabilities? Think Again. Management Science, 51(5):726–740, 2005.

Digital Library

[17]

I. V. Krsul. Software Vulnerability Analysis. PhD thesis, Purdue University, 1998.

Digital Library

[18]

J. Lazar, J. H. Feng, and H. Hochheiser. Research methods in human-computer interaction. John Wiley & Sons, 2010.

Digital Library

[19]

Q. Liu and Y. Zhang. VRSS: A new system for rating and scoring vulnerabilities. Computer Communications, 34(3):264–273, 2011.

Digital Library

[20]

Q. Liu, Y. Zhang, Y. Kong, and Q. Wu. Improving VRSS-based vulnerability prioritization using analytic hierarchy process. Journal of Systems and Software, 85(8):1699–1708, 2012.

Digital Library

[21]

D. McKinney. Vulnerability Bazaar. Security & Privacy, 5(6):69–73, 2007.

Digital Library

[22]

P. Mell and K. Scarfone. Improving the common vulnerability scoring system. IET Information Security, 1(3):119, 2007.

[23]

P. Mell, K. Scarfone, and S. Romanosky. A complete guide to the common vulnerability scoring system version 2.0. pages 1–23, 2007.

[24]

C. Miller. The Legitimate Vulnerability Market: Inside the Secretive World of 0-day Exploit Sales. In 6th Workshop Economics of Information Security. Citeseer, 2007.

[25]

N. Munaiah and A. Meneely. Vulnerability Bounty Data Set. https://gist.github.com/nuthanmunaiah/ fe6a7e8e3cc4dbae84dc. Accessed: 2016-03-11.

[26]

A. Ozment. Bug Auctions: Vulnerability Markets Reconsidered. In 3rd Workshop Economics of Information Security, pages 19–26, 2004.

[27]

PCI SSC. Requirements and Security Assessment Procedures v3.1. (Apr):1–115, 2015.

[28]

S. E. Schechter. Computer Security Strength & Risk : A Quantitative Approach. PhD thesis, 2004.

Digital Library

[29]

Y. Yang, S. Jin, and X. He. Software Vulnerability Severity Evaluation Based on Economic Losses. In Int. Conf. Trustworthy Computing and Services, pages 144–151. Springer, 2014.

[30]

A. Younis, Y. K. Malaiya, and I. Ray. Assessing vulnerability exploitability risk using software properties. Software Quality Journal, 2015.

Digital Library

[31]

A. A. Younis and Y. K. Malaiya. Comparing and Evaluating CVSS Base Metrics and Microsoft Rating System. In Int. Conf. Software Quality, Reliability and Security, pages 252–261, Aug 2015.

Digital Library

Cited By

Gal‐Or EHydari MTelang R(2024)Merchants of Vulnerabilities: How Bug Bounty Programs Benefit Software VendorsSSRN Electronic Journal10.2139/ssrn.4808742Online publication date: 2024
https://doi.org/10.2139/ssrn.4808742
Mazloomzadeh IUddin GKhomh FSami A(2024)Reputation Gaming in Crowd Technical Knowledge SharingACM Transactions on Software Engineering and Methodology10.1145/369162734:1(1-41)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3691627
Sparks HGhosh K(2024)Discovery of Evolving Relationships of Software Vulnerabilities2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA)10.1109/TPS-ISA62245.2024.00045(332-340)Online publication date: 28-Oct-2024
https://doi.org/10.1109/TPS-ISA62245.2024.00045
Show More Cited By

Index Terms

Vulnerability severity scoring and bounties: why the disconnect?
1. Security and privacy
  1. Software and application security
    1. Software security engineering
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software

Recommendations

Predicting Severity of Software Vulnerability Based on Grey System Theory
Proceedings of the ICA3PP International Workshops and Symposiums on Algorithms and Architectures for Parallel Processing - Volume 9532

Vulnerabilities usually represents the risk level of software, therefore, it is of high value to predict vulnerabilities so as to evaluate the security level of software. Current researches mainly focus on predicting the number of vulnerabilities or the ...
Vulnerability scoring for security configuration settings
QoP '08: Proceedings of the 4th ACM workshop on Quality of protection

The best-known vulnerability scoring standard, the Common Vulnerability Scoring System (CVSS), is designed to quantify the severity of security-related software flaw vulnerabilities. This paper describes our efforts to determine if CVSS could be adapted ...
Examining the factors that impact the severity of cyberattacks on critical infrastructures
Abstract
In light of the rising threats of cyberattacks on critical infrastructures, cybersecurity has become a high priority for government agencies worldwide. In particular, the severity of cyberattacks could lead to devastating consequences for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SWAN 2016: Proceedings of the 2nd International Workshop on Software Analytics

November 2016

53 pages

ISBN:9781450343954

DOI:10.1145/2989238

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13, 2016

WA, Seattle, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
433
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)2

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gal‐Or EHydari MTelang R(2024)Merchants of Vulnerabilities: How Bug Bounty Programs Benefit Software VendorsSSRN Electronic Journal10.2139/ssrn.4808742Online publication date: 2024
https://doi.org/10.2139/ssrn.4808742
Mazloomzadeh IUddin GKhomh FSami A(2024)Reputation Gaming in Crowd Technical Knowledge SharingACM Transactions on Software Engineering and Methodology10.1145/369162734:1(1-41)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3691627
Sparks HGhosh K(2024)Discovery of Evolving Relationships of Software Vulnerabilities2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA)10.1109/TPS-ISA62245.2024.00045(332-340)Online publication date: 28-Oct-2024
https://doi.org/10.1109/TPS-ISA62245.2024.00045
Kim HKim D(2024)Methodological Advancements in Standardizing Blockchain AssessmentIEEE Access10.1109/ACCESS.2024.337257812(35552-35570)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3372578
Rahman ARedino CNandakumar DCody TShetty SRadke D(2024)Motivation for Model‐driven Penetration TestingReinforcement Learning for Cyber Operations10.1002/9781394206483.ch4(89-104)Online publication date: 27-Dec-2024
https://doi.org/10.1002/9781394206483.ch4
Sparks HGhosh K(2023)NUVER: Network Based Vulnerability Visualizer2023 IEEE 30th Annual Software Technology Conference (STC)10.1109/STC58598.2023.00010(16-19)Online publication date: 25-Sep-2023
https://doi.org/10.1109/STC58598.2023.00010
Caldeira JBrito e Abreu FCardoso JSimões ROliveira TPereira dos Reis J(2023)Software Development Analytics in Practice: A Systematic Literature ReviewArchives of Computational Methods in Engineering10.1007/s11831-022-09864-y30:3(2041-2080)Online publication date: 10-Jan-2023
https://doi.org/10.1007/s11831-022-09864-y
Algarni A(2022)The Historical Relationship between the Software Vulnerability Lifecycle and Vulnerability Markets: Security and Economic RisksComputers10.3390/computers1109013711:9(137)Online publication date: 14-Sep-2022
https://doi.org/10.3390/computers11090137
Paul R(2022)ASTOR: An Approach to Identify Security Code ReviewsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3559509(1-3)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3559509
Cody T(2022)A Layered Reference Model for Penetration Testing with Reinforcement Learning and Attack Graphs2022 IEEE 29th Annual Software Technology Conference (STC)10.1109/STC55697.2022.00015(41-50)Online publication date: Oct-2022
https://doi.org/10.1109/STC55697.2022.00015
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten