(Credit: René Ramos; mangpor2004/Shutterstock.com)
Most of the software you install and use visibly does what you expect. Video editors let you tune up your videos. Games respond to your button-mashing. Chat programs connect with your friends. But when you install the protection of an antivirus utility or security suite, there’s no immediate way to know how well it works. Unless you get a notification that the utility foiled an attack, you don’t really know what it’s doing. That’s where PCMag can help. When evaluating these products for review, we put their claims to the test in many ways. Each review reports the results of our tests, along with hands-on experience with the product. This article will fill you in on just how these tests work.
Of course, not every test is appropriate for every product. Many antivirus utilities include protection against phishing, for example, but some don't. Many suites include parental control, but others omit this feature. Some nominally standalone antivirus products add firewall protection as a bonus. Whatever features a given product offers, we put them to the test.
We Test Antivirus Protection Using Actual Malware
Every full-powered antivirus tool includes two core features. The on-demand scanner seeks out and destroys existing malware infestations, and the real-time monitor fends off new attacks. There’s only one way to be sure these protective features work: to hit them with real-world malware. We use virtual machines for this testing, so there’s no risk of spreading any missed infections.
We Collect Real-World Malware
Each year in the spring, when most security vendors have finished their yearly update cycle, we gather a new collection of malware samples for this test. We start with a feed of the latest malware-hosting URLs, download many thousands of samples, and winnow them down to a manageable number.
It's Surprisingly Easy to Be More Secure Online
We analyze each sample using various hand-coded tools. Some of the samples detect when they're running in a virtual machine and refrain from malicious activity; we simply don't use those. We look for a variety of different types, and for samples that make changes to the file system and Registry.
During this process, we check the identifying hash value of each sample against a database maintained by Google subsidiary VirusTotal. This database scans files using about 70 antivirus engines and reports how many (and which) of them identified a file as malicious. If fewer than half the engines consider a sample to be malicious, we discard it. Once we’ve pared the collection down to something reasonable, we record exactly what system changes each sample makes.
We Challenge Each Antivirus to Defend Against Malware
To test a product's malware-blocking abilities, we start by opening the folder of samples. Real-time protection in some products kicks in immediately, wiping out known malware. If necessary to trigger real-time protection, we single-click each sample, or copy the collection to a new folder, or download the samples from cloud storage—whatever it takes. We note which samples the antivirus eliminates on sight.
Next, we launch each surviving sample and watch how the antivirus handles it. We record the total percentage detected, regardless of when detection happened.
Detection of a malware attack is necessary but insufficient; the antivirus must also actively prevent the attack. A small hand-coded program checks the system to determine whether the malware managed to make any Registry changes or install any of its files. In the case of executable files, it also checks whether any of those processes are running. As soon as the measurement is complete, we shut down the virtual machine.
If a product prevents installation of all executable traces by a malware sample, it earns 8, 9, or 10 points, depending on how well it prevented cluttering the system with non-executable traces. Detecting malware but failing to prevent the installation of executable components gets half-credit, 5 points. Finally, if we find one or more malware processes running despite the antivirus's attempt at protection, that's worth a mere 3 points. The average of all these scores becomes the product's final malware protection score.
We Evaluate Web-Level Malware Protection
The best time to frustrate a malware attack is before it ever reaches your computer. Many antivirus products integrate with your browsers and steer them away from known malware-hosting pages. A few operate below the browser level, with no extension required. If protection doesn't kick in at that level, there's always an opportunity to wipe out the malware payload during or immediately after download.
While our basic malware protection test uses the same set of samples for months, the malware-hosting URLs we use to test Web-based protection are different every time. We get those links from a feed of the very newest malicious URLs detected by London-based MRG-Effitas and typically use URLs that are no more than a few days old.
The Best Antivirus Utilities We've Tested
Using a small purpose-built utility, we go down the list, launching each URL in turn. We discard any of them that don't point to a malware download and any that return error messages. For the rest, we note whether the antivirus prevents access to the URL, wipes out the download, or sits by idly doing nothing. After recording the result, the utility jumps to the next URL in the list that isn't at the same domain. We do skip any files larger than 10MB or so, and we skip files that have already appeared in the same test. We keep at it until we've accumulated data for a hundred verified malware-hosting URLs.
The score in this test is simply the percentage of URLs for which the antivirus prevented downloading malware, whether by cutting off access to the URL completely or by wiping out the downloaded file. Perfect 100% scores are not uncommon, and most tested antivirus tools manage 90% or better protection.
We Use Actual Phishing Frauds for Testing
Why go to the effort of creating an elaborate data-stealing Trojan when you can just trick people into giving up their secrets? That's the mindset of malefactors who create and manage phishing websites. These fraudulent sites mimic banks and other sensitive sites. If you enter your login credentials, you've just given away the keys to the kingdom. And phishing is platform-independent; it works on any operating system that supports browsing the Web.
These fake websites typically get blacklisted not long after their creation, so for testing we use only the very newest phishing URLs. We gather these from phishing-oriented websites, trying for an equal split between verified phishing pages and those that have been reported as frauds but not yet verified. The best security programs detect these new fakes using real-time analysis. Those that just rely on simple-minded blacklists typically earn lower scores.
We use four virtual machines for this test, one protected by the product under testing, and one each using the phishing protection built into Chrome, Edge, and Firefox. A small utility program launches each URL in the four browsers. If any of the four returns an error message, we discard that URL. If the resulting page doesn't actively attempt to imitate another site or doesn't attempt to capture username and password data, we discard it. For the rest, we note whether each product detected the fraudulent page.
Unlike the web-level protection test, antiphishing scores vary widely. Some products achieve 100% detection, while others can’t even outscore the protection built into the three browsers.
We’ve Simplified Antispam Testing
These days email accounts for most consumers have spam vacuumed out of them by the email provider, or by a utility running on the email server. In fact, the average consumer’s need for spam filtering is slim to none. Austrian test lab AV-Comparatives tested antispam functionality back in 2016, finding that even Microsoft Outlook alone blocked almost 90 percent of spam, and most suites did better—some of them much better. But the lab hasn’t published another such test. According to one of the principals there, “end consumers use hosted mail solutions like Gmail, so no spam filter is needed anymore.”
Years ago, we ran our own antispam tests using a real-world account that we carefully tuned so it would receive plenty of spam, along with plenty of valid email. To test a spam filter, we would download thousands of messages and manually check whether any spam slipped into the inbox or, worse, valid mail got marked as spam. This test took more time and effort than any of our other hands-on tests. Expending maximal effort on a feature of minimal importance no longer makes sense.
There are still important points to report about a suite's spam filter. What email clients does it support? Can you use it with an unsupported client? Is it limited to POP3 email accounts, or does it also handle IMAP, Exchange, or even Web-based email? At present, we carefully consider each suite's antispam capabilities, but we longer suffer through downloading and analyzing thousands of emails
Security Suite Performance Doesn’t Require Testing
When your security suite is busily watching for malware attacks, defending against network intrusions, preventing your browser from visiting dangerous websites, and so on, it uses your system's CPU and other resources to do its job. Many, many years ago, security suites deservedly earned a reputation for sucking up system resources so much that they’d affect your activities on the computer. If users turn off protection because it’s a performance drag, that’s no kind of protection.
The Best Security Suites We've Tested
Over the years, security companies steadily tuned up their products to eliminate any noticeable slowdown. Until recently we would run some simple tests to measure each suite’s effect on system performance. We’ve dropped that test because almost all recent suites pass with no more than a tiny performance impact. In some cases, the tests run faster after installing the suite.
We Test Firewall Protection Several Ways
The typical personal firewall has two jobs, protecting the computer from outside attack and ensuring that programs don't misuse the network connection. In the past, we used a physical computer that used the router's DMZ port to effectively get a direct connection to the internet. More recently, routers and ISPs have made that connection more difficult. In any case, a computer that's connected through a router (as almost all are) is effectively invisible to the internet at large, making our port-scan tests and other web-based tests less relevant. In any case, the built-in Windows firewall handles stealthing all ports, so we've dropped this test.
We Evaluate the Firewall’s Program Control
Program control in the earliest personal firewalls was painfully interactive. Every time a new program tried to access the network, the firewall popped up a query asking the user whether to allow or block access. This approach isn't very effective since the user generally has no idea what action is correct. Most will just allow everything. Others will click Block every time until they break some important program; after that, they allow everything.
It's becoming more and more common for suite firewalls to include this kind of detailed program control without making it the default. For those products, we turn it on before testing. Also, many firewalls come preconfigured with access permissions for known programs. For our hands-on check of program control functionality, we use an app that’s guaranteed to be unknown—a tiny browser utility coded in-house and not found anywhere else.
At the other end of the spectrum, the best firewalls automatically configure network permissions for known good programs, eliminate known bad programs, and step up surveillance on unknowns. Only if an unknown program attempts a suspicious connection does the firewall kick in to stop it. That’s not something we can test, since we don’t write zero-day malware, but we can observe this functionality during malware protection testing.
We Check Protection Against Exploits
Software isn't and can't be perfect, so the bad guys work hard to find security holes in popular operating systems, browsers, and applications. They devise exploits to compromise system security using any vulnerabilities they find. Naturally, the maker of the exploited product issues a security patch as soon as possible, but until you apply that patch, you're vulnerable.
The smartest firewalls intercept these exploit attacks at the network level, so they never even reach your computer. Even for those that don't scan at the network level, in many cases the antivirus component wipes out the exploit's malware payload. We use the CORE Impact penetration tool to hit each test system with about 30 recent exploits and record how well the security product fended them off.
We Probe the Firewall’s Defenses
Finally, we run a sanity check to see whether a malware coder could easily disable security protection. We look for an on/off switch in the Registry and test whether it can be used to turn off protection. We attempt to terminate security processes using Task Manager and third-party task killer utilities. And we check whether it's possible to stop or disable the product's essential Windows services.
We Verify Parental Control Features
At present, PCMag doesn’t rate or recommend standalone parental control and monitoring services. Rather, we suggest making use of the free screen time and other parental features built right into modern operating systems. But when a security suite includes parental control as one of its components, we still put it through its paces, to be sure it does what it promises.
The typical parental control utility keeps kids away from unsavory sites, monitors their Internet usage, and lets parents determine when and for how long the kids are allowed to use the Internet each day. Other features range from limiting chat contacts to patrolling Facebook posts for risky topics.
We always perform a sanity check to make sure the content filter correctly blocks inappropriate websites. As it turns out, finding porn sites for testing is a snap. Just about any URL composed of a size adjective and the name of a normally covered body part is already a porn site. Very few products fail this test.
Recommended by Our Editors
We use a tiny in-house browser utility to verify that content filtering is browser independent. We issue a three-word network command (no, we're not publishing it here) that disables some simple-minded content filters. And we check whether we can defeat the filter by using a secure anonymizing proxy website.
Imposing time limits on the children's computer or Internet use is only effective if the kids can't interfere with timekeeping. We verify that the time-scheduling feature works, then try foiling it by resetting the system date and time. The best products don't rely on the system clock for their date and time.
After that, it's simply a matter of testing the features that the program claims to have. If it promises the ability to block the use of specific programs, we engage that feature and try to break it by moving, copying, or renaming the program. If it says it strips out bad words from email or instant messaging, we add a random word like “fnord” to the block list and verify that it doesn't get sent. If it claims it can limit instant messaging contacts, we set up a conversation between two of our accounts and then ban one of them. Whatever control or monitoring power the program promises, we do our best to put it to the test.
We Monitor and Interpret Antivirus Lab Tests
We don't have the resources to run the kind of exhaustive antivirus tests performed by independent labs around the world, so we pay close attention to their findings. We follow four labs that release scored test results on a regular basis, using their results to help inform our reviews.
AV-Test’s Three-Way Evaluation
Based in Magdeburg, Germany, the AV-Test Institute continuously puts antivirus programs through a variety of tests. The one we focus on is a three-part test that awards up to 6 points in each of three categories: Protection, Performance, and Usability. To reach certification, a product must earn a total of 10 points and have no zero scores in any of the categories. That’s a fairly low bar. The very best products take home a perfect 18 points in this test.
To test protection, the researchers expose each product to AV-Test's reference set of over 100,000 samples, and to several thousand extremely widespread samples. Products get credit for preventing the infestation at any stage, be it blocking access to the malware-hosting URL, detecting the malware using signatures, or preventing the malware from running based on its behavior. The best products often reach 100 percent success in this test.
Performance is important—if the antivirus noticeably puts a drag on system performance, some users will turn it off. AV-Test's researchers measure the difference in time required to perform more than a dozen common system actions with and without the security product present. Among these actions are downloading files from the Internet, copying files both locally and across the network, and running common programs. Averaging multiple runs, they can identify just how much impact each product has.
The Usability test isn't necessarily what you'd think. It has nothing to do with ease of use or user interface design. Rather, it measures the usability problems that occur when an antivirus program erroneously flags a legitimate program or website as malicious or suspicious. Researchers actively install and run an ever-changing collection of popular programs, noting any odd behavior by the antivirus. A separate scan-only test checks to make sure the antivirus doesn't identify any of over 600,000 legitimate files as malware.
Multiple Tests From AV-Comparatives
We track results from three of the many tests regularly released by AV-Comparatives, which is based in Austria and works closely with the University of Innsbruck. Security tools that pass a test receive Standard certification; those that fail are designated as merely Tested. If a program goes above and beyond the necessary minimum, it can earn Advanced or Advanced+ certification.
This lab’s file detection test is a simple, static test that checks each antivirus against about 100,000 malware samples. A parallel false-positives test ensures accuracy—too many false positives can knock a program’s rating down by one or even two ranks. Much like the AV-Test, the performance test measures any impact on system performance. We consider the dynamic whole-product test to be the most significant. This test aims to simulate as closely as possible an actual user's experience, allowing all components of the security product to participate in fighting the malware.
SE Labs Replays Actual Malware Attacks
Where AV-Test and AV-Comparatives typically include a couple of dozen products in testing, SE Labs generally reports on no more than 10. That's in large part because of the nature of this lab's test. Researchers capture real-world malware-hosting websites and use a replay technique so that each product encounters precisely the same drive-by download or other Web-based attacks. It's extremely realistic but arduous.
A program that totally blocks one of these attacks earns three points. If it sprang into action after the attack began but managed to remove all executable traces, that's worth two points. And if it merely terminated the attack without full cleanup, it still gets one point. In the unfortunate event that the malware runs free on the test system, the product under testing loses five points. Because of this, some products (notably early editions of Windows Defender) have received below-zero scores.
In a separate test, the researchers evaluate how well each product refrains from erroneously identifying valid software as malicious, weighing the results based on each valid program's prevalence and on how much of an impact the false-positive identification would have. They combine the results of these two tests and certify successful products at one of five levels: AAA, AA, A, B, and C.
Tests by MRG-Effitas Are Pass/Fail
As noted above, we use a feed of samples supplied by MRG-Effitas in our hands-on malicious URL blocking test. We also follow quarterly results from two of this lab’s ongoing tests. The 360 Assessment & Certification test simulates real-world protection against current malware, much like the dynamic real-world test used by AV-Comparatives. A product that completely prevents any infestation by the sample set receives Level 1 certification. Level 2 certification means that at least some of the malware samples planted files or other traces on the test system, but these traces were eliminated later. Any product that doesn’t achieve one of these levels simply fails. The Online Banking Certification specifically tests for protection against financial malware and botnets on a similar pass/fail basis.
We Calculate an Aggregate Testing Score
Coming up with an overall summary of lab results isn't easy, since the labs don't all test the same collection of programs and they all use different scoring systems. We've devised an algorithm that normalizes each lab's scores to a value from 0 to 10. Our aggregate lab results chart reports a weighted average of these scores, as well as the number of lab tests. The best possible result is a perfect 10 score based on results from all four labs. If just one lab includes a product in testing, we consider that to be insufficient information for an aggregate score.
What About VPNs?
You may have noted that this list of testing methods doesn't cover virtual private networks, or VPNs. Evaluating a VPN is very different from any other portion of a security suite, so we have a separate article for how we test VPN services.
Like What You're Reading?
Sign up for SecurityWatch newsletter for our top privacy and security stories delivered right to your inbox.
This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.
Thanks for signing up!
Your subscription has been confirmed. Keep an eye on your inbox!
Sign up for other newsletters
