Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

How to Think Like a Vulnerability Assessor

•
•
Roger Johnston
Roger Johnston

If you can't think like a bad guy to get good security, at least try to think like a vulnerability assessor.

Related slideshows

Vulnerability Assessment Myths
Vulnerability Assessment MythsVulnerability Assessment Myths
Vulnerability Assessment Myths
 
Vulnerability Assessment, Physical Security, and Nuclear Safeguards
Vulnerability Assessment, Physical Security, and Nuclear SafeguardsVulnerability Assessment, Physical Security, and Nuclear Safeguards
Vulnerability Assessment, Physical Security, and Nuclear Safeguards
 
Relating Risk to Vulnerability
Relating Risk to Vulnerability Relating Risk to Vulnerability
Relating Risk to Vulnerability
 
1 of 19
Download to read offline
1 Talk for the 58th Annual ASIS Meeting Philadelphia, PA, September 10-13, 2012 Roger G. Johnston, Ph.D., CPP Jon S. Warner, Ph.D. Vulnerability Assessment Team Argonne National Laboratory 630-252-6168 rogerj@anl.gov Vulnerability Assessment Team (VAT)! Sponsors • DHS • DoD • DOS • IAEA • Euratom • DOE/NNSA • private companies • intelligence agencies • public interest organizations The VAT has done detailed vulnerability assessments on over 1000 different security devices, systems, & programs. The greatest of faults, I should say, is to be conscious of none. -- Thomas Carlyle (1795-1881) A multi-disciplinary team of physicists, engineers, hackers, & social scientists. Check us out on YouTube: keywords = argonne break into 2
2 Terminology! Threat: Who might attack, why, when, how, with what probability, and with what resources. (Includes information on goals and attack modes.) Threat Assessment (TA): Attempting to identify threats. Terminology! Vulnerability: Flaw or weakness that could be exploited to cause undesirable consequences. Vulnerability Assessment (VA): Discovering and demonstrating ways to defeat a security device, system, or program. Should include suggesting countermeasures and security improvements.
3 Threat vs. Vulnerability! Threat: Adversaries might try to steal PII information (SSNs, credit card numbers, etc.) from our computer systems to commit crimes. Vulnerability: We don’t keep our anti-malware software up to date. _____________________________________________ Threat: Adversaries could dump toxic chemicals on our property, then blame us to try to get us in trouble with environmental officials and the public. Vulnerability: We don’t have good access control or video monitoring of our grounds. 5 Why VAs Trump TAs! (especially for catastrophic security incidents)! 6 Threats Vulnerabilities reactive, focused on the past proactive, focused on the future speculative right in front of you (if you’re willing to see them) hard to test testable not usually fixable often easy to fix often generic specific to the ground level details If you get the threats wrong but understand and (at least partially) fix the vulnerabilities, you may be ok. If you get the vulnerabilities wrong (or ignore them), you are probably in trouble despite how well you understand the threats.
4 Security Risk Management - An Optimization Problem! Inputs: ü assets to protect ü overall security goals ü asset valuation/prioritization ü consequences of successful attack(s) ü threat assessment ü vulnerability assessment ü available resources & possible security measures ü general security philosophy/strategy ü various estimated/guessed probabilities *often vague, incomplete, or missing **often under-estimated * Outputs: Ø What to protect and at what level Ø How to deploy resources optimally ** * * ** ** 7 * Purpose! The purpose of a VA is to improve security & minimize risk, not to: • Pass a test • Test security • Generate metrics • Justify the status quo • Praise or accuse anybody • Check against some standard • Claim there are no vulnerabilities • Engender warm & happy feelings • Determine who gets salary increases • Rationalize the research & development • Apply a mindless, bureaucratic stamp of approval • Endorse a security product/program or Certify it as “good” or “ready to use” 8
5 A VA is Not…! § auditing § quality control § reliability testing § efficiency testing § compliance testing § acceptance testing § ergonomics testing § performance testing § response time testing § operational assessment § environmental robustness testing 9 Techniques Often Confused with VAs! q feature analysis q threat assessment q Design Basis Threat q CARVER Method (DoD) q software assessment tools q security survey (walking around with a checklist) q security audit (are the rules known & being followed?) q fault or event tree analysis (from safety engineering) q Delphi Method (method for getting a decision from a panel of experts) 10
6 Vulnerability Assessment (VA) Blunders! These assumptions are wrong: • A vulnerability assessment should be done at the end. • There are a small number of vulnerabilities. • Most or all can be found & eliminated. • A VA should ideally find zero vulnerabilities. • Vulnerabilities are bad news. Vulnerability Assessment (VA) Blunders! • Not using creative people with a hacker mentality who want to find problems and suggest solutions • Conflicts of interest (economic & psychological) • Shooting the messenger • Sham rigor • The fallacy of precision • Fear of NORQ analysis 12 NORQ = Non-Objective Non-Reproducible Non-Quantifiable
7 Vulnerability Assessment (VA) Blunders! • Focusing on high-tech attacks • Letting attack methods define the vulnerabilities, not the other way around • Arbitrarily constrained VAs (scope, time, effort, by modules or components) • Limiting the VA to the lower part of the Vulnerability Pyramid Where Vulnerability Ideas Come From! The Vulnerability Pyramid 14
8 Safety & Security are 2 Relatively Unrelated Problems! Example: March 2012 Recall of 900,000 Safety 1st Push N’ Snap Cabinet Locks 140 reports of babies/toddlers defeating the locks, resulting in 3 poisonings Security: All about nefarious adversaries. Safety: No adversaries. 15 16 Working with Outside VAers! • Seek creative, hands-on assessors with a history of finding problems and suggesting solutions, and who are psychologically pre-disposed to doing so. • At least be sure at the end you understand what subtle attacks & insider attacks look like! • You don’t have to mitigate all discovered vulnerabilities or accept all suggestions, but be sure you have good reasons (not just ego, arrogance, denial, laziness, or wishful thinking).
9 17 Assembling Your Own VA Team:! Seek…! q hackers q narcissists q trouble makers q hands-on types q creative people q loop-hole finders q independent thinkers q questioners of authority q people curious about how things work Blunder: Thinking Engineers Understand Security" Engineers... • ...work in solution space, not problem space • …make things work but aren't trained or mentally inclined to figure out how to make things break • ...view Nature or economics as the adversary, not the bad guys • …think of technologies as failing randomly, not by deliberate, intelligent, malicious, opportunistic intent • …are not typically predisposed to think like bad guys • …focus on user friendliness—not making things difficult for the bad guys • ...like to add lots of extra features that open up new attack vectors • …want products to be simple to maintain, repair, and diagnose—which usually makes them easy to attack 18
10 19 “White Box” vs. “Black Box” VA! White Box: Full details, specifications, and technical disclosures are given to the Vulnerability Assessors at the start. [Most time/cost effective & closest to reality.] Black Box: The Vulnerability Assessors reverse engineering or discover all or most of the details on their own. [Interesting & illuminating, but usually not realistic or time/ cost effective.] Adversarial Vulnerability Assessments! • Perform a mental coordinate transformation and pretend to be the bad guys (or VAers). (This is much harder than you might think.) • Be much more creative than the adversaries. They need only stumble upon 1 vulnerability, the good guys have to worry about all of them. 20
11 Adversarial Vulnerability Assessments! • Don’t let the good guys & the existing security infrastructure and tactics define the problem. • Gleefully look for trouble, rather than seeking to reassure yourself that everything is fine. 21 We need to be more like fault finders. They find problems because they want to find problems, and because they are skeptical: • bad guys • therapists • movie critics • computer hackers • scientific peer reviewers • mothers-in-law 22
12 * AVA Steps 1. Fully understand the device, system, or program and how it is REALLY used. Talk to the low-level users and frontline personnel. 2. Play with it. 3. Brainstorm--anything goes! (Effective brainstorming is the key!) 4. Play with it some more. 23 * AVA Steps 5. Edit & prioritize potential attacks. 6. Partially develop some attacks. 7. Determine feasibility of the attacks. 8. Devise countermeasures. 9. Perfect attacks. 10. Demonstrate attacks. 11. Rigorously test attacks. 12. Rigorously test countermeasures.
13 Delaying Judgment! Nothing can inhibit and stifle the creative process more— and on this there is unanimous agreement among all creative individuals and investigators of creativity—than critical judgment applied to the emerging idea at the beginning stages of the creative process. ... More ideas have been prematurely rejected by a stringent evaluative attitude than would be warranted by any inherent weakness or absurdity in them. The longer one can linger with the idea with judgment held in abeyance, the better the chances all its details and ramifications [can emerge]. -- Eugene Raudsepp, Managing Creative Scientists and Engineers (1963). Keep the possibility phase completely separate from the practicality phase! 25 The Creative VA Process! • Individuals must be given ownership of their original idea & should be personally recognized for their creativity. • The group environment needs to be: + diverse + high-energy + people tired + urgent but not stressful + free of authority figures + humorous, joyful, & fun + cohesive but not too cohesive + competitive in a friendly & respectful way + enthusiastic about individual differences & eccentricities • Every idea, no matter how wacky or seemingly stupid, gets written down & treated as a gem, at least initially. 26
14 The Creative VA Process! Be skeptical! Pay close attention to explicit or unstated assumptions, and to security features that are widely praised or admired. These are often the source of serious vulnerabilities. Concentrate on the 2nd and 3rd best attacks or countermeasures. You are likely overlooking something that would make them the best solutions. If there is widespread agreement about the efficacy of an attack or countermeasure, re-examine. Something important was probably overlooked. The Creative VA Process! Quantity breeds quality. With all ideas: elaborate, expand, modify, subvert, exaggerate, & combine with other ideas. Pursue hunches & intuition. The best ideas come late, and when you are not thinking about the problem. Pursue what is interesting, controversial, contrarian, exciting, or silly. 28
15 The Creative VA Process! Develop and explore models, metaphors, & analogies. Terminology constrains our thinking. Rename everything in your own (and/or silly) words, and think about them in light of the new terminology. Consider different verbs for what the bad guys might want to accomplish: attack, steal, demolish, embarrass, tag, terminate, uncover, purify, whistleblow, poison, etc. Ridicule existing security measures & strategies. Avoid the fear of the NORQ! 29 Video!
16 arrogance * 32
17 Slacker Donuts! 33 You want like…um…a donut, dude?TM 34 Elements of the Slacker Donuts Security Program • No checks • There’s a safe for cash but $50 is immediately available to hand robbers • Cash taken to local bank at 11 AM • Not open 24/7 but bright illumination 24/7 • Periodic rounds by shared private security • Good relations with local community, businesses, police, street people • Shared slacker culture with employees and clientele • Secret recipes known to only a few
18 In Summary! * There are advantages to thinking like a Vulnerability Assessor when you think about your security. * Don’t get confused about what a VA is or its role in overall Risk Management. * To go into “Vulnerability Assessor Mode”, step outside yourself, be creative & irreverent+, and & try humor (which can be very mentally liberating). * You must want to find problems—or else find people who do. 35 * Special Thanks to: * Christopher Folk (for helping to develop the Fear of NORQ model) * Security Theater 3000 “Commercial” * Mitch Farmer.....Investment Banker * Jim Regis…..Former Security Officer * Roy Lindley…..Arthritis Patient * Veronica Manfredi…..Wife (& Tech Support/Graphics) * Christopher Folk…..Husband * Marrissa Faler…..Homemaker (& Tech Support) * Buddy the Dog…..As Himself * Greg Byslma…..Tech Support 36
19 For More Information...! Additional information is available from: rogerj@anl.gov and http://www.ne.anl.gov/capabilities/vat http://www.youtube.com/watch?v=frBBGJqkz9E

More Related Content

How to Think Like a Vulnerability Assessor

  • 1. 1 Talk for the 58th Annual ASIS Meeting Philadelphia, PA, September 10-13, 2012 Roger G. Johnston, Ph.D., CPP Jon S. Warner, Ph.D. Vulnerability Assessment Team Argonne National Laboratory 630-252-6168 rogerj@anl.gov Vulnerability Assessment Team (VAT)! Sponsors • DHS • DoD • DOS • IAEA • Euratom • DOE/NNSA • private companies • intelligence agencies • public interest organizations The VAT has done detailed vulnerability assessments on over 1000 different security devices, systems, & programs. The greatest of faults, I should say, is to be conscious of none. -- Thomas Carlyle (1795-1881) A multi-disciplinary team of physicists, engineers, hackers, & social scientists. Check us out on YouTube: keywords = argonne break into 2
  • 2. 2 Terminology! Threat: Who might attack, why, when, how, with what probability, and with what resources. (Includes information on goals and attack modes.) Threat Assessment (TA): Attempting to identify threats. Terminology! Vulnerability: Flaw or weakness that could be exploited to cause undesirable consequences. Vulnerability Assessment (VA): Discovering and demonstrating ways to defeat a security device, system, or program. Should include suggesting countermeasures and security improvements.
  • 3. 3 Threat vs. Vulnerability! Threat: Adversaries might try to steal PII information (SSNs, credit card numbers, etc.) from our computer systems to commit crimes. Vulnerability: We don’t keep our anti-malware software up to date. _____________________________________________ Threat: Adversaries could dump toxic chemicals on our property, then blame us to try to get us in trouble with environmental officials and the public. Vulnerability: We don’t have good access control or video monitoring of our grounds. 5 Why VAs Trump TAs! (especially for catastrophic security incidents)! 6 Threats Vulnerabilities reactive, focused on the past proactive, focused on the future speculative right in front of you (if you’re willing to see them) hard to test testable not usually fixable often easy to fix often generic specific to the ground level details If you get the threats wrong but understand and (at least partially) fix the vulnerabilities, you may be ok. If you get the vulnerabilities wrong (or ignore them), you are probably in trouble despite how well you understand the threats.
  • 4. 4 Security Risk Management - An Optimization Problem! Inputs: ü assets to protect ü overall security goals ü asset valuation/prioritization ü consequences of successful attack(s) ü threat assessment ü vulnerability assessment ü available resources & possible security measures ü general security philosophy/strategy ü various estimated/guessed probabilities *often vague, incomplete, or missing **often under-estimated * Outputs: Ø What to protect and at what level Ø How to deploy resources optimally ** * * ** ** 7 * Purpose! The purpose of a VA is to improve security & minimize risk, not to: • Pass a test • Test security • Generate metrics • Justify the status quo • Praise or accuse anybody • Check against some standard • Claim there are no vulnerabilities • Engender warm & happy feelings • Determine who gets salary increases • Rationalize the research & development • Apply a mindless, bureaucratic stamp of approval • Endorse a security product/program or Certify it as “good” or “ready to use” 8
  • 5. 5 A VA is Not…! § auditing § quality control § reliability testing § efficiency testing § compliance testing § acceptance testing § ergonomics testing § performance testing § response time testing § operational assessment § environmental robustness testing 9 Techniques Often Confused with VAs! q feature analysis q threat assessment q Design Basis Threat q CARVER Method (DoD) q software assessment tools q security survey (walking around with a checklist) q security audit (are the rules known & being followed?) q fault or event tree analysis (from safety engineering) q Delphi Method (method for getting a decision from a panel of experts) 10
  • 6. 6 Vulnerability Assessment (VA) Blunders! These assumptions are wrong: • A vulnerability assessment should be done at the end. • There are a small number of vulnerabilities. • Most or all can be found & eliminated. • A VA should ideally find zero vulnerabilities. • Vulnerabilities are bad news. Vulnerability Assessment (VA) Blunders! • Not using creative people with a hacker mentality who want to find problems and suggest solutions • Conflicts of interest (economic & psychological) • Shooting the messenger • Sham rigor • The fallacy of precision • Fear of NORQ analysis 12 NORQ = Non-Objective Non-Reproducible Non-Quantifiable
  • 7. 7 Vulnerability Assessment (VA) Blunders! • Focusing on high-tech attacks • Letting attack methods define the vulnerabilities, not the other way around • Arbitrarily constrained VAs (scope, time, effort, by modules or components) • Limiting the VA to the lower part of the Vulnerability Pyramid Where Vulnerability Ideas Come From! The Vulnerability Pyramid 14
  • 8. 8 Safety & Security are 2 Relatively Unrelated Problems! Example: March 2012 Recall of 900,000 Safety 1st Push N’ Snap Cabinet Locks 140 reports of babies/toddlers defeating the locks, resulting in 3 poisonings Security: All about nefarious adversaries. Safety: No adversaries. 15 16 Working with Outside VAers! • Seek creative, hands-on assessors with a history of finding problems and suggesting solutions, and who are psychologically pre-disposed to doing so. • At least be sure at the end you understand what subtle attacks & insider attacks look like! • You don’t have to mitigate all discovered vulnerabilities or accept all suggestions, but be sure you have good reasons (not just ego, arrogance, denial, laziness, or wishful thinking).
  • 9. 9 17 Assembling Your Own VA Team:! Seek…! q hackers q narcissists q trouble makers q hands-on types q creative people q loop-hole finders q independent thinkers q questioners of authority q people curious about how things work Blunder: Thinking Engineers Understand Security" Engineers... • ...work in solution space, not problem space • …make things work but aren't trained or mentally inclined to figure out how to make things break • ...view Nature or economics as the adversary, not the bad guys • …think of technologies as failing randomly, not by deliberate, intelligent, malicious, opportunistic intent • …are not typically predisposed to think like bad guys • …focus on user friendliness—not making things difficult for the bad guys • ...like to add lots of extra features that open up new attack vectors • …want products to be simple to maintain, repair, and diagnose—which usually makes them easy to attack 18
  • 10. 10 19 “White Box” vs. “Black Box” VA! White Box: Full details, specifications, and technical disclosures are given to the Vulnerability Assessors at the start. [Most time/cost effective & closest to reality.] Black Box: The Vulnerability Assessors reverse engineering or discover all or most of the details on their own. [Interesting & illuminating, but usually not realistic or time/ cost effective.] Adversarial Vulnerability Assessments! • Perform a mental coordinate transformation and pretend to be the bad guys (or VAers). (This is much harder than you might think.) • Be much more creative than the adversaries. They need only stumble upon 1 vulnerability, the good guys have to worry about all of them. 20
  • 11. 11 Adversarial Vulnerability Assessments! • Don’t let the good guys & the existing security infrastructure and tactics define the problem. • Gleefully look for trouble, rather than seeking to reassure yourself that everything is fine. 21 We need to be more like fault finders. They find problems because they want to find problems, and because they are skeptical: • bad guys • therapists • movie critics • computer hackers • scientific peer reviewers • mothers-in-law 22
  • 12. 12 * AVA Steps 1. Fully understand the device, system, or program and how it is REALLY used. Talk to the low-level users and frontline personnel. 2. Play with it. 3. Brainstorm--anything goes! (Effective brainstorming is the key!) 4. Play with it some more. 23 * AVA Steps 5. Edit & prioritize potential attacks. 6. Partially develop some attacks. 7. Determine feasibility of the attacks. 8. Devise countermeasures. 9. Perfect attacks. 10. Demonstrate attacks. 11. Rigorously test attacks. 12. Rigorously test countermeasures.
  • 13. 13 Delaying Judgment! Nothing can inhibit and stifle the creative process more— and on this there is unanimous agreement among all creative individuals and investigators of creativity—than critical judgment applied to the emerging idea at the beginning stages of the creative process. ... More ideas have been prematurely rejected by a stringent evaluative attitude than would be warranted by any inherent weakness or absurdity in them. The longer one can linger with the idea with judgment held in abeyance, the better the chances all its details and ramifications [can emerge]. -- Eugene Raudsepp, Managing Creative Scientists and Engineers (1963). Keep the possibility phase completely separate from the practicality phase! 25 The Creative VA Process! • Individuals must be given ownership of their original idea & should be personally recognized for their creativity. • The group environment needs to be: + diverse + high-energy + people tired + urgent but not stressful + free of authority figures + humorous, joyful, & fun + cohesive but not too cohesive + competitive in a friendly & respectful way + enthusiastic about individual differences & eccentricities • Every idea, no matter how wacky or seemingly stupid, gets written down & treated as a gem, at least initially. 26
  • 14. 14 The Creative VA Process! Be skeptical! Pay close attention to explicit or unstated assumptions, and to security features that are widely praised or admired. These are often the source of serious vulnerabilities. Concentrate on the 2nd and 3rd best attacks or countermeasures. You are likely overlooking something that would make them the best solutions. If there is widespread agreement about the efficacy of an attack or countermeasure, re-examine. Something important was probably overlooked. The Creative VA Process! Quantity breeds quality. With all ideas: elaborate, expand, modify, subvert, exaggerate, & combine with other ideas. Pursue hunches & intuition. The best ideas come late, and when you are not thinking about the problem. Pursue what is interesting, controversial, contrarian, exciting, or silly. 28
  • 15. 15 The Creative VA Process! Develop and explore models, metaphors, & analogies. Terminology constrains our thinking. Rename everything in your own (and/or silly) words, and think about them in light of the new terminology. Consider different verbs for what the bad guys might want to accomplish: attack, steal, demolish, embarrass, tag, terminate, uncover, purify, whistleblow, poison, etc. Ridicule existing security measures & strategies. Avoid the fear of the NORQ! 29 Video!
  • 17. 17 Slacker Donuts! 33 You want like…um…a donut, dude?TM 34 Elements of the Slacker Donuts Security Program • No checks • There’s a safe for cash but $50 is immediately available to hand robbers • Cash taken to local bank at 11 AM • Not open 24/7 but bright illumination 24/7 • Periodic rounds by shared private security • Good relations with local community, businesses, police, street people • Shared slacker culture with employees and clientele • Secret recipes known to only a few
  • 18. 18 In Summary! * There are advantages to thinking like a Vulnerability Assessor when you think about your security. * Don’t get confused about what a VA is or its role in overall Risk Management. * To go into “Vulnerability Assessor Mode”, step outside yourself, be creative & irreverent+, and & try humor (which can be very mentally liberating). * You must want to find problems—or else find people who do. 35 * Special Thanks to: * Christopher Folk (for helping to develop the Fear of NORQ model) * Security Theater 3000 “Commercial” * Mitch Farmer.....Investment Banker * Jim Regis…..Former Security Officer * Roy Lindley…..Arthritis Patient * Veronica Manfredi…..Wife (& Tech Support/Graphics) * Christopher Folk…..Husband * Marrissa Faler…..Homemaker (& Tech Support) * Buddy the Dog…..As Himself * Greg Byslma…..Tech Support 36
  • 19. 19 For More Information...! Additional information is available from: rogerj@anl.gov and http://www.ne.anl.gov/capabilities/vat http://www.youtube.com/watch?v=frBBGJqkz9E