Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
General Security Discussions
A Brief Critique of Professional AV Tests
Message
<blockquote data-quote="Trident" data-source="post: 1037905" data-attributes="member: 99014"><p>The samples containing anti-vm would alter any test, not just the AVC or AV-Test samples. Evasion of VM can be controlled and reduced down to minimum, the points malware checks to determine whether it’s running on a real system have been discussed many times in many different places.</p><p>For example Joe Sandbox, Anyrun, Hybrid Analysis/Falcon Sandbox, the CheckPoint emulation are all highly resistant to evasion.</p><p></p><p>There are other, deeper issues with these tests that Cruelsister discussed in the notepad files already.</p><p>This coin has multiple sides though.</p><p></p><p>On one side we have:</p><ul> <li data-xf-list-type="ul">Small malware scope. Many vendors claim they block about 400 new malware samples per second. 12K samples for a whole month in this case are nothing.</li> <li data-xf-list-type="ul">Potentially badly curated malware, not enough studied, not enough understood. I personally know how easy it is to go down a rabbit hole with one sample or group of samples and you can devote few days on that. It is impossible to research in-depth 12K samples.</li> <li data-xf-list-type="ul">Everyone produces high, sometimes inconsistent scores so in the process of choosing a product, the test becomes useless. I mean AV-Test continuously rates over 10 products as “TOP” with 6 on protection, 6 on performance and 6 on usability. Who do you go for in that case.</li> <li data-xf-list-type="ul">There are various other issues but if I keep discussing them I will write a whole book and we are not here for that.</li> </ul><p>But then there is the other side of the coin.</p><p></p><p>It’s easy to sit down and say “my javascript was not detected, my vbs script was not detected” so the antivirus is rubbish. But:</p><ul> <li data-xf-list-type="ul">Everybody knows antiviruses miss malware. By publishing such statements one is hardly re-discovering the hot water. If antivirus could detect and prevent everything, malware creation would become unprofitable, end and together with that, antivirus development will have to be suspended too. So far we see this hasn’t happened.</li> <li data-xf-list-type="ul">How much of this malware affects home users? Is it even relevant to test Avira Antivirus Free vs malware that’s targeting business environments? There is a ton of defences more that businesses use on top of “antivirus”. You can’t expect 1 layer out of the many that is offered to home users, to block such attacks.</li> <li data-xf-list-type="ul">Antivirus software is developed not only with detection and protection in mind. There is always the performance challenge which is extreme, you only got milliseconds to analyse a file and output the verdict. You always need to make sure false positives are not produced as this robs users from certain experiences and creates enormous overhead for the developers who are not at work all day to deal with anti-malware vendors. And finally, it is the automation. You can’t produce constant messages and prompts, decisions should be automated.</li> </ul><p>All this leads to anti-malware vendors researching, balancing and compromising, and producing solutions for real people experiencing real life problems. Such solutions can’t perform well on various tests.</p><p>It’s also difficult to come up with a strategy that properly tests their abilities. The majority of people criticising antivirus software haven’t actually developed their own alternatives, have they? If anyone thinks any antivirus is rubbish because it didn’t detect their JavaScripts and VBS files, they are welcome to point us to the antivirus they/their business develops, that detects and blocks 100% of everything.</p></blockquote><p></p>
[QUOTE="Trident, post: 1037905, member: 99014"] The samples containing anti-vm would alter any test, not just the AVC or AV-Test samples. Evasion of VM can be controlled and reduced down to minimum, the points malware checks to determine whether it’s running on a real system have been discussed many times in many different places. For example Joe Sandbox, Anyrun, Hybrid Analysis/Falcon Sandbox, the CheckPoint emulation are all highly resistant to evasion. There are other, deeper issues with these tests that Cruelsister discussed in the notepad files already. This coin has multiple sides though. On one side we have: [LIST] [*]Small malware scope. Many vendors claim they block about 400 new malware samples per second. 12K samples for a whole month in this case are nothing. [*]Potentially badly curated malware, not enough studied, not enough understood. I personally know how easy it is to go down a rabbit hole with one sample or group of samples and you can devote few days on that. It is impossible to research in-depth 12K samples. [*]Everyone produces high, sometimes inconsistent scores so in the process of choosing a product, the test becomes useless. I mean AV-Test continuously rates over 10 products as “TOP” with 6 on protection, 6 on performance and 6 on usability. Who do you go for in that case. [*]There are various other issues but if I keep discussing them I will write a whole book and we are not here for that. [/LIST] But then there is the other side of the coin. It’s easy to sit down and say “my javascript was not detected, my vbs script was not detected” so the antivirus is rubbish. But: [LIST] [*]Everybody knows antiviruses miss malware. By publishing such statements one is hardly re-discovering the hot water. If antivirus could detect and prevent everything, malware creation would become unprofitable, end and together with that, antivirus development will have to be suspended too. So far we see this hasn’t happened. [*]How much of this malware affects home users? Is it even relevant to test Avira Antivirus Free vs malware that’s targeting business environments? There is a ton of defences more that businesses use on top of “antivirus”. You can’t expect 1 layer out of the many that is offered to home users, to block such attacks. [*]Antivirus software is developed not only with detection and protection in mind. There is always the performance challenge which is extreme, you only got milliseconds to analyse a file and output the verdict. You always need to make sure false positives are not produced as this robs users from certain experiences and creates enormous overhead for the developers who are not at work all day to deal with anti-malware vendors. And finally, it is the automation. You can’t produce constant messages and prompts, decisions should be automated. [/LIST] All this leads to anti-malware vendors researching, balancing and compromising, and producing solutions for real people experiencing real life problems. Such solutions can’t perform well on various tests. It’s also difficult to come up with a strategy that properly tests their abilities. The majority of people criticising antivirus software haven’t actually developed their own alternatives, have they? If anyone thinks any antivirus is rubbish because it didn’t detect their JavaScripts and VBS files, they are welcome to point us to the antivirus they/their business develops, that detects and blocks 100% of everything. [/QUOTE]
Insert quotes…
Verification
Post reply
Top
