16:05 CET, 17th October 2008 By Thomas Kristensen.
The test of the 12 Internet Security Suites published earlier this week has generated a lot of reactions, however, it appears that some have misinterpreted the purpose of the test.
It is very important to note that this in no way should be considered a comprehensive test of the security provided by these solutions. We only tested one specific aspect (exploitation of vulnerabilities) because too many users believe (and are lead to believe by the marketing material) that they only need a security suite to protect them against various threats including hackers.
Our point is not that Internet Security Suites are useless (they are quite useful for most users). Instead, our point is that they protect insufficiently against hackers and that it is better to prevent attacks by patching rather than relying on other security measures alone.
Also, it seems quite odd that the AV-vendors are so busy claiming that they can detect literally anything malicious when executed. If they can do that, why do they then have to push "signature" updates to their software so frequently?
With regards to the more serious comments like the one from Panda about why we didn't launch the file-based test cases: We didn't expect that to be necessary since we explicitly requested a manual scan of the files.
If a user receives e.g. an Office document, saves it, scans it, and it isn't detected as malicious, the user would (and should be able to) trust the document as it may be e.g sent to someone else or moved to a system without the same kind of protection.
It is obviously much better to be able to detect malicious content while it is passive instead of relying on (hopefully) being able to catch it once executed.
Some people also questioned whether we "remembered" to update the software - naturally this was the case (and seemed like stating the obvious):
* All security suites were fully updated (25th September)
* All security suites were fully patched (25th September)
* All security suites were installed with default settings
* All security suites were tested in the same way
* All security suites failed to prevent code execution for a number of exploits
NOTE: The test cases were split into three categories as described in the test paper: PoC, GameOver, and Exploits.
However, even though we did not execute every single file-based exploit, we did execute every single web-based exploit. Only one of the tested products managed to block some of the web-based exploits generically, but after a few modifications of the exploits we were able to bypass that detection.
Nonetheless, we find the criticism from Panda useful and if we do conduct another test of the file-based test cases then we will categorise their performance into: Unzipping, manual scan, and opening of test case with vulnerable application.
In my opinion it would serve the security industry well if AV-vendors would admit that the security provided by their products rely on a reasonably updated and well administrated system. If they really could protect systems without patches, then I'm quite confident that software vendors would stop making patches and instead provide these fabulous security solutions themselves.