Advanced Endpoint Protection

ENDPOINT PROTECTION
Q2 2020 COMPARATIVE RATINGS
False Positives
Management
Resistance to
Handcrafted
Block Rate
(Targeted)
Malware
Malware
Evasions
Drive-By
Exploits
Exploits
Attacks
Overall
Social
Email
HTTP
Score Rating Vendor/Product Price
713 AA AA AA AA AA AA AA AA A AAA $172,500

Cybereason Professional v19.1.86.0
696 AA AA AA AA AAA AAA AA AA CCC AAA $118,125

Check Point Software Technologies
SandBlast Agent v81.20.7425
2020 ADVANCED ENDPOINT PROTECTION
695 AA AA AAA AA AA AAA AA AA CCC AAA $73,250

Kaspersky Endpoint Security for
Business v11.1
695 AA AA AA AA AA AAA AAA AA CCC AAA $116,072

Elastic Endpoint Security v3.14.0
683 AA AA AA AA AA AA AA AA B AAA $170,775

Fortinet FortiEDR v3.1.3.15
680 AA AA AA AA AA AA AA BB A AAA $195,000

Palo Alto Networks Traps Cortex
XDR v6.2
647 A AA BBB A AA AA AA BBB BB AAA $85,773

Sophos Intercept X Advanced 10.8.3
601 A A A A AA AA AA BBB D AAA $139,500

F-Secure PSB CPP + RDR v19.9
600 A A AA A AA AA AA BB D AAA $291,575

Bitdefender GravityZone Ultra
v6.6.16.216
589 A A AA A AA AA AA B D AAA $64,935

Fortinet FortiClient v6.2.2
440 B BBB AAA CC BBB A AA B D BB $221,100

Malwarebytes Endpoint
Protection (EP) v1.2.0.717
— NR Panda Adaptive Defense

— — — — — — — — — —
360: 3.60.00 Windows
Protection: 8.00.15.0030
This report is confidential and expressly limited to NSS Labs clients.

Q2 2020 PRODUCT RATINGS ENDPOINT PROTECTION
Tuning and False Positives
0.0% 0.2% 0.4% 0.6% 0.8% 1.0% 1.2% 1.4% 1.6% 1.8% 2.0%
Bitdefender GravityZone Ultra v6.6.16.216 0.5%
Check Point Software Technologies SandBlast Agent v81.20.7425 0.8%
Cybereason Professional v19.1.86.0 0.9%
Elastic Endpoint Security v3.14.0 0.9%
Fortinet FortiEDR v3.1.3.15 0.9%
Fortinet FortiClient v6.2.2 0.6%
F-Secure PSB CPP + RDR v19.9 1.1%
Kaspersky Endpoint Security for Business v11.1 0.2%
Malwarebytes Endpoint Protection (EP) v1.2.0.717 0.3%
Palo Alto Networks Traps Cortex XDR v6.2 0.5%
Panda Adaptive Defense 360: 3.60.00 Windows Protection:

1.1%
8.00.15.0030
Sophos Intercept X Advanced 10.8.3 1.7%
Figure 1 – False Positive Rate
Samples Tested: 645
This test include a varied sample of legitimate application To ensure that the vendor did not deploy unrealistic (overly
traffic that may be falsely identified as malicious (also known aggressive) security policies that blocked access to legitimate
as false positives). As part of the initial setup, we tuned the software and websites, we tested the endpoint protection
endpoint protection as it would be by a customer. Every effort against 645 false positive samples, including but not limited to
was made to eliminate false positives while achieving optimal the following file formats: .exe, .jar, .xls, .xlsm, .accdb, .css,
security effectiveness and performance, as a typical customer .pdf, .doc, .docx, .zip, .DLL, .js, xls, .chm, .rar, .Ink, .cur, .xrc.,
deploying the device in a live network environment would do. .slk, .ppt, pptx, .iqy, .htm.
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
2
Resistance to Evasion
0% 20% 40% 60% 80% 100%
Bitdefender GravityZone Ultra v6.6.16.216 100%
Cybereason Professional v19.1.86.0 100%
Elastic Endpoint Security v3.14.0 100%
Fortinet FortiEDR v3.1.3.15 100%
Fortinet FortiClient v6.2.2 100%
F-Secure PSB CPP + RDR v19.9 100%
Kaspersky Endpoint Security for Business v11.1 100%
Palo Alto Networks Traps Cortex XDR v6.2 100%
Panda Adaptive Defense 360: 3.60.00 Windows Protection: 8.00.15.0030 100%
Sophos Intercept X Advanced 10.8.3 100%
Missed % Detected % Blocked %
Figure 2 – Resistance to Evasions
Samples Tested: 49
Threat actors apply evasion techniques to disguise and modify were due to the evasions and not the underlying (baseline)
attacks at the point of delivery in order to avoid detection by attacks.
security products. Therefore, endpoint protection must
For example, we applied an evasion technique called process
correctly handle evasions. If an endpoint protection platform
injection, where the original file is extracted from the binary
fails to detect a single form of evasion, an attack can bypass
and code is injected into a legitimate/trusted target process
protection.
(i.e., Google Chrome). The malicious execution then occurs
Our engineers verified that the endpoint protection was under the context of the target process (Chrome). Once these
capable of detecting and blocking malware when subjected to process injections techniques ran, we tried to further elude
numerous evasion techniques. To develop a baseline, we took the detection by introducing anti-sandbox/anti-discovery
several attacks that had previously been detected and evasions that employed techniques to determine whether the
blocked. We then applied evasion techniques to those malware was on a user’s machine; whether a security product
baseline samples and tested. This ensured that any misses was present; whether debugging or sandboxing was occurring;
and so on.
3
Malware Delivered over Email
0% 20% 40% 60% 80% 100%
Check Point Software Technologies SandBlast Agent v81.20.7425 100%
Panda Adaptive Defense 360: 3.60.00 Windows Protection: 8.00.15.0030 87.1%
Figure 3 – Malware Delivered over Email
Samples Tested: 1531
One of the most common ways in which users are the sender is trustworthy. The victim is tricked into opening
compromised is through malware delivered over email. For the email attachment, which then launches the malicious
several years, the use of social engineering has accounted for malware program. To test how well the endpoint protection is
the bulk of cyberattacks against consumers and enterprises. able to protect against this type of attack, malware was
Socially engineered malware attacks often use a dynamic emailed to the user. The desktop client then retrieved the
combination of social media, hijacked email accounts, false email and opened/executed the malware. If the malware was
notification of computer problems, and other deceptions to blocked, the corresponding time was recorded. We deployed
encourage users to download malware. One well-known social a CentOS 7.7.1908 Linux mail store with kernel 3.10.0-
engineering attack method is spear phishing. Cybercriminals 957.5.1.el7.x86_64 running Dovecot v2.2.36 for IMAP as the
use hijacked email accounts to take advantage of the implicit mail server. Victim machines consisted of a combination of 32-
trust between contacts and deceive victims into believing that bit and 64-bit Windows 7 endpoints and 64-bit Windows 10
endpoints.
4
Malware Delivered over HTTP

0% 20% 40% 60% 80% 100%
Check Point Software Technologies SandBlast Agent v81.20.7425 100%
Kaspersky Endpoint Security for Business v11.1 100%
Figure 4 – Malware Delivered over HTTP
Samples Tested: 424
One of the more widespread threats to the enterprise involves We tested the capability of the endpoint protection to protect
attackers using websites to deliver malware. In these web- against malware that was downloaded over HTTP and then
based attacks, the user is deceived into downloading and executed (if the download was not blocked) using 424
executing malware. For example, an employee may be tricked malware samples against live victim machines running a
into downloading and installing a malicious application that combination of 32-bit and 64-bit Windows 7 endpoints and
claims it will “speed up your PC.” In cases where an attacker is 64-bit Windows 10 endpoints, with various versions of Google
aiming for a large number of victims, the attacker may hijack Chrome, Mozilla Firefox, Microsoft Internet Explorer, and
widely used reputable websites to distribute the malware. Microsoft Edge. Browser reputation systems were disabled so
However, in cases where an attacker plans to target specific that the endpoint protection was not inadvertently credited
individuals, the attacker typically would use an industry- for protection offered by a web browser.
specific “watering hole” plus one or more social engineering
techniques to deceive a user into unknowingly installing
malware.
5
Drive-by Exploits
0% 20% 40% 60% 80% 100%
Figure 5 – Drive-by Exploits
Samples Tested: 456
While there are millions (or hundreds of millions) of malware running 32-bit Windows 7 (version 6.1 (Build 7601: SP1) and
samples in circulation at any given point in time, they are 64-bit Microsoft Windows 10 (version 1709 (Build: 16299.15)
frequently delivered by exploits that target consumer with Microsoft Office (Office 16.0.7341.2032) and various
desktops known as drive-by exploits. versions of Google Chrome, Mozilla Firefox, Microsoft Internet
Explorer, and Microsoft Edge. Depending on the victim
In a drive-by exploit, an employee visits a website containing
machine, one or more of the following applications was
malicious code that exploits the user’s computer and installs
installed: Java 8 Update 231, Microsoft Silverlight 5.1.20125,
malware without the knowledge or permission of the user. An
Adobe Flash Player 18.0.0.160, Adobe Reader DC
example of this would be where an employee visits a social
2017.012.20093, Adobe Reader 9.40, Java 6 Update 27, Adobe
media site which is inadvertently hosting an exploit. Another
Flash Player 32.0.0.238, Java 8 Update 221, Microsoft
example (that we frequently observe in the wild) is where a
Silverlight 5.1.50918, Adobe Flash Player 32.0.0.223, Java 8
user navigates to a URL and then is re-directed without
Update 211, Adobe Flash Player 32.0.0.207, Internet Explorer
interaction to a web page serving malicious content. Using this
11, Internet Explorer 10, and Internet Explorer 9. Browser
technique, a single exploit can silently deliver and install
reputation systems were disabled so that the endpoint
millions of malware samples to unsuspecting victims’
protection was not inadvertently credited for protection
computers. To test how well the solution was able to protect
offered by a web browser.
against drive-by exploits, victim machines were deployed
6
Social Exploits
0% 20% 40% 60% 80% 100%
Figure 6 – Social Exploits
Samples Tested: 50
Social exploits combine social engineering (manipulating handling, which provides attackers with a wide attack surface.
people into doing what you want them to do) and exploitation As such, sending social exploits through mass email (phishing),
(malicious code designed to take advantage of existing could be profitable, as the number of victims would be large,
deficiencies in hardware or software systems, such as albeit smaller than in the case of malware since exploits would
vulnerabilities or bugs). An example of this would be an email have technical dependencies.
with “Your Bonus” as a subject line and containing a malicious To test how well the product was able to protect against social
spreadsheet named “bonus.xlsx” (which the employee opens). exploits, we deployed 19 victim machines. All of the machines
As with drive-by exploits, these attacks are limited to specific were running Windows 10 version 1709 (OS Build 16299.15).
operating systems and/or applications. However, the exploits Machines were configured with Internet Explorer 11 (version
contained within Excel spreadsheets or Word documents may 11.15.16299.0 – Update Version 11.0.47) and Microsoft Office
target kernel functions or common functions such as object 2016 (version 16.0.7431.2032).
7
Handcrafted (Targeted) Attacks

0% 20% 40% 60% 80% 100%
Fortinet FortiClient v6.2.2
Figure 7 – Handcrafted (Targeted) Attacks
Samples Tested: 21
The aim of this test was to see which endpoint products were For the purposes of this test, handcrafted (targeted) malware
able to protect customers while under adverse conditions was created by modifying the source code of keyloggers,
dictated by the attacker. In this case, we wanted to find out ransomware, and destructoware, and then recompiling the
which products could block new handcrafted (unknown) binary so that it was new to the products being tested. We
malware while being prevented from accessing cloud services. then attempted to infect a host (e.g., a laptop) with the
malware and recorded whether the endpoint protection
What happens, for example, if an employee goes on a
blocked the attack. Because creating samples in this manner is
business trip to China where Internet traffic is tightly
a painstaking and time-consuming exercise, we tested only a
controlled? In such a scenario, access to the corporate VPN is
handful of targeted samples; results should be viewed with
likely blocked, and the security software on the employee’s
this in mind.
laptop may not be able to receive updates or communicate in
general. What happens if the employee’s laptop is attacked
with targeted malware?
8
Test Environment
• BaitNET™ (NSS Labs Proprietary)
• 32-bit Microsoft Windows 7 (Version 6.1 (Build 7601: SP1)
• 64-bit Microsoft Windows 7 (Version 6.1 (Build 7601: SP1)
• 64-bit Microsoft Windows 10 (version 1607 (Build: 14393.0)
• 64-bit Microsoft Windows 10 (version 1709 (Build: 16299.15)
• Adobe Acrobat Reader 19.021.20061
• Adobe Flash Player 18.0.0.160
• Adobe Reader 9.40
• Adobe Reader DC 2017.012.20093
• Google Chrome 78.0.3904.70
• Kali (Kernel release 4.19.0-kali1-amd64)
• Microsoft Internet Explorer 9.0.8112.16421
• Microsoft Office Professional 2013 version 15.0.5119.1000 (Microsoft Word, Excel, PowerPoint, Access, etc.)
• Microsoft Office Professional 2016 version 16.0.7341.2032 (Microsoft Word, Excel, PowerPoint, Access, etc.)
• Microsoft Silverlight 5.1.20125
• Microsoft Silverlight 5.1.50918
• Oracle Java 6 Update 27
• Rapid7 Metasploit (v5.0.46-dev)
• VMware vCenter (Version 6.7u2 Build 6.7.0.30000)
• VMware vSphere (Version 6.7.0.30000)
• VMware ESXi (Version 6.7u3 Build 14320388)
• Wireshark version 3.0.3
9
Appendix
NSS LABS RATINGS
RATING DEFINITION
A product rated ‘AAA’ has the highest rating assigned by NSS Labs. The product’s capacity to meet its
AAA
commitments to consumers is extremely strong.
A product rated ‘AA’ differs from the highest-rated products only to a small degree. The product’s capacity to
AA
meet its commitments to consumers is very strong.
A product rated ‘A’ is somewhat more susceptible to sophisticated attacks than higher-rated categories.
A
However, the product’s capacity to meet its commitments to consumers is still strong.
A product rated ‘BBB’ exhibits adequate protection parameters. However, sophisticated or previously unseen
BBB
attacks are more likely to negatively impact the product’s capacity to meet its commitments to consumers.
A product rated ‘BB,’ ‘B,’ ‘CCC,’ ‘CC,’ and ‘C’ is regarded as having significant risk characteristics. ‘BB’ indicates
the least degree of risk and ‘C’ the highest. While such products will likely have some specialized capability and
protective characteristics, these may be outweighed by large uncertainties or major exposure to adverse
conditions.
A product rated ‘BB’ is less susceptible to allowing a compromise than products that have received higher-risk
BB ratings. However, the product faces major technical limitations, which could be exposed by threats that would
lead to its inability to meet its commitments to consumers.
A product rated ‘B’ is more susceptible to allowing a compromise than products rated ‘BB’; however, it
B currently has the capacity to meet its commitments to consumers. Adverse threat conditions will likely expose
the product’s technical limitations and expose its inability to meet its commitments to consumers.
A product rated ‘CCC’ is currently susceptible to allowing a compromise and is dependent upon favorable
CCC threat conditions for it to meet its commitments to consumers. In the event of adverse threat conditions, the
product is not likely to have the capacity to meet its commitments to consumers.
A product rated ‘CC’ is currently highly susceptible to allowing a compromise. The ‘CC’ rating is used when a
CC failure has not yet occurred but NSS Labs considers a breach a virtual certainty, regardless of the anticipated
time to breach.
A product rated ‘C’ is currently highly susceptible to allowing a compromise. The product is expected to fail to
C
prevent a breach and to not have useful forensic information compared with products that are rated higher.
A product rated ‘D’ is actively being breached by known threats and is unable to protect consumers. For non-
specialized products, the ‘D’ rating category is used when protecting a consumer is unattainable without a
major technical overhaul. Unless NSS Labs believes that such technical fixes will be made within a stated grace
D
period (often 30-90 calendar days), the ‘D’ rating also is an indicator that it is a virtual certainty that existing
customers using the product have already experienced a breach—whether they know it or not—and should
take immediate action.
10
Authors
Rabin Bhattarai, Thomas Skybakmoen, Vikram Phatak
Test Methodology
NSS Labs Advanced Endpoint Protection (AEP) Test Methodology v4.0 is available at www.nsslabs.com.
Contact Information
NSS Labs, Inc.
3711 South Mopac Expressway

Building 1, Suite 400
Austin, TX 78746
info@nsslabs.com
www.nsslabs.com
This and other related documents are available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs.
© 2020 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, e-mailed
or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).
Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you
should not read the rest of this report but should instead return the report immediately to us. “You” or “your” means the person who accesses
this report and any entity on whose behalf he/she has obtained this report.
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and
reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever
arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO
EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS
OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the
hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or
that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective
owners.
11

Advanced Endpoint Protection

Uploaded by

Copyright:

Available Formats

Advanced Endpoint Protection

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Endpoint Protection

Uploaded by

Copyright:

Available Formats

What does the report evaluate?

What does the report evaluate?

How does the report assess resistance to evasions?

How does the report assess resistance to evasions?

ENDPOINT PROTECTION

Q2 2020 COMPARATIVE RATINGS

713 AA AA AA AA AA AA AA AA A AAA $172,500

696 AA AA AA AA AAA AAA AA AA CCC AAA $118,125

695 AA AA AAA AA AA AAA AA AA CCC AAA $73,250

695 AA AA AA AA AA AAA AAA AA CCC AAA $116,072

683 AA AA AA AA AA AA AA AA B AAA $170,775

680 AA AA AA AA AA AA AA BB A AAA $195,000

647 A AA BBB A AA AA AA BBB BB AAA $85,773

601 A A A A AA AA AA BBB D AAA $139,500

600 A A AA A AA AA AA BB D AAA $291,575

589 A A AA A AA AA AA B D AAA $64,935

440 B BBB AAA CC BBB A AA B D BB $221,100

— NR Panda Adaptive Defense

This report is confidential and expressly limited to NSS Labs clients.

Tuning and False Positives

Bitdefender GravityZone Ultra v6.6.16.216 0.5%

Check Point Software Technologies SandBlast Agent v81.20.7425 0.8%

Cybereason Professional v19.1.86.0 0.9%

Elastic Endpoint Security v3.14.0 0.9%

Fortinet FortiEDR v3.1.3.15 0.9%

Fortinet FortiClient v6.2.2 0.6%

F-Secure PSB CPP + RDR v19.9 1.1%

Kaspersky Endpoint Security for Business v11.1 0.2%

Malwarebytes Endpoint Protection (EP) v1.2.0.717 0.3%

Palo Alto Networks Traps Cortex XDR v6.2 0.5%

Panda Adaptive Defense 360: 3.60.00 Windows Protection:

Sophos Intercept X Advanced 10.8.3 1.7%

Figure 1 – False Positive Rate

Samples Tested: 645

Bitdefender GravityZone Ultra v6.6.16.216 100%

Check Point Software Technologies SandBlast Agent v81.20.7425 95.9%

Cybereason Professional v19.1.86.0 100%

Elastic Endpoint Security v3.14.0 100%

Fortinet FortiEDR v3.1.3.15 100%

Fortinet FortiClient v6.2.2 100%

F-Secure PSB CPP + RDR v19.9 100%

Kaspersky Endpoint Security for Business v11.1 100%

Malwarebytes Endpoint Protection (EP) v1.2.0.717 91.8%

Palo Alto Networks Traps Cortex XDR v6.2 100%

Panda Adaptive Defense 360: 3.60.00 Windows Protection: 8.00.15.0030 100%

Sophos Intercept X Advanced 10.8.3 100%

Missed % Detected % Blocked %

Figure 2 – Resistance to Evasions

Malware Delivered over Email

0% 20% 40% 60% 80% 100%

Bitdefender GravityZone Ultra v6.6.16.216 99.7%

Check Point Software Technologies SandBlast Agent v81.20.7425 100%

Cybereason Professional v19.1.86.0 99.9%

Elastic Endpoint Security v3.14.0 99.8%

Fortinet FortiEDR v3.1.3.15 99.3%

Fortinet FortiClient v6.2.2 99.9%

F-Secure PSB CPP + RDR v19.9 99.7%

Kaspersky Endpoint Security for Business v11.1 99.7%

Malwarebytes Endpoint Protection (EP) v1.2.0.717 88.4%