Why are Independant lab results unreliable?

[jackuars]

I'm not interested in claims that they grab money for publishing results, but what could be the other reasons that they are unreliable in testing the overall effectiveness of an antivirus software? Can you give me some examples?

Also in what ways are User Reviews at MalwareTips better or worse than the ones' published at Independent Labs?

Time and again this debate comes up in our forums so let's put and end to that?

 

[Deleted member 178]

These "guys" that can disable "most" security suite's, do they do so remotely, or as Admin on the system it is installed?

remotely, easier if you have an non-customized admin account.

Neither one of these problems, would be a problem if the attacker can not make it into the system to begin with..

Then enter the social engineering.

 

[illumination]

remotely, easier if you have an non-customized admin account.

My general assessment of this is simple but yet profound.
There are not as many skilled hackers as there are proclaimed ones. If they truly are endowed with a gift, it is generally from being a recluse, keeping to themselves, and very unlikely they would show many that gift, if at all.

Taking scenarios to the extreme will leave those bound for answers with a feeling of helplessness, thinking anything they do to secure their system will not be enough. This, is untrue. It takes a great deal of knowledge to remotely attack an individual's system and gain access to perform malicious actions, and generally needing something to gain from it to make it worth taking the chances, which leaves home users a lot less likely to become targeted from such an attack. That is where your "social engineering" comes into play...

Then enter the social engineering.

I agree safe surfing habits are indeed needed with a good solid baseline security.

 

[Kiwimike]

@Kiwimike

Where did you get those statistics from?

How should they do that? They would need zero-day exploits for this kind of test and those go for quite a bit of money, more that they make every year with their tests.

Killtesting AV software is useless, if it's on the system the protection of your AV is the least of your worries.

There are no absolutes in software, it can and will fail, so based on this every security software can and will fail.

Basing the security of a software on an impractical and outdated test?

Are we talking about home users here or businesses? ARP spoofing protection for home users is unnecessary.

Shameless self quote Best Firewall: Kaspersky IS 2016, BitDefender IS, or Eset Smart Security, or Emsisoft IS

They never have and never will block everything, they are okay for what they are supposed to be doing ,blocking known and common stuff, but that's it.


Trashing three of the biggest security companies out there without any reason? Have you even tried their endpoint and/or server protection?

1. If you want sources, a simple google search will do the trick:
Antivirus tools miss almost 70 percent of malware within the first hour
Antivirus is dead, says maker of Norton Antivirus

Those are the two highest links that come up for me, and while I will admit they do not say a mere 20% are detected. So while I prefer to stick to my claim, I'll give you that one.

2. True, they would. But if they are testing antivirus software they should live up to what they are actually doing. If you wanted to be really cheap you could probably find exploits on the internet or grab some from a company or find exploits that are zero day currently out there.

3. You do make a point, antivirus software's primary function is to prevent and not remove. However, if malware is on the system killing the antivirus software would indeed be bad, incase lets say the malware gets added to the signature database one day and the malware gets detected and removal gets attempted.
Although an even more better reason, is that it should just be done anyways because way to many antiviruses fail at it, and its a real shame.
If an antivirus can't protect itself, how it can it protect the system from exploitation? AVG and Kaspersky recently included ASLR in their products after a memory vulnerability could cause a peice of malware or attacker to manipulate the av. This is probably not the case with just these.

4. True, and yes. Which is why I repsect Comodo so much, they haven't failed. They have the protection that is deserved by the enterprise. But other than Comodo, all antiviruses fail yes.

5. Your right, it is old and impractal. Still doesn't mean its irrelevant, it very much still tests the antivirus protective measures against malware performing certain actions.
Spyshelter hasn't failed this test, or been overcome by a keylogger, or been killed. It's very good, if an anti-keylogger can imcorperate protection like this without interfereing the user. Why don't other antiviruses do the same?

6. We are actually talking about businesses and home users. I beileve it should be blocked on all circumstances. It's something a 5 year old could do, that calls for it being blocked on all platforms.

http://www.av-comparatives.org/wp-content/uploads/2014/04/avc_fw_201403_en.pdf
Saying that, I do agree with your post you linked me to. It is true, and I personally and quite lack with my security which is my own flaw. But, I can admit it at least, I have a firewall with ClamAV built in and then just use my own antivirus. But I do admit that Firewall in your operating system aren't nearly as needed as before.

But they still should have some standard, the point isn't that firewalls NEED to be the best. It's that if your gonna include a custom firewall make it better than what's already included.


And lastly, Yes. I have used Symantec Endpoint protection. I am good friends with a system administrator for Wells Fargo, who uses it all the time, so I have also used it and did not like the console. I also found its protection was quite mediocre.

I am a fan of Kaspersky's endpoint, advanced console that I think is good for any admin I also feel that its got great protection and don't think I'm putting down Mcafee's firewall, I'm putting down their antivirus thier firewalls are actually quite good at managing and dealing with network intrusions in a network as the function as both effective firewalls and load balancers allowing traffic to be balanced between each firewall so to prevent a firewall from going down, and if it does it will simply switch firewalls.

This was certainly a strict post on my part, but I'm happy and proud of it because its needed. All big antiviruses have their good points but they still all suck. At RSA confrence last year I beileve a presentation demonstrated a very efficient and good antivirus for the enterprise which blocked a reinactment of the Sony hack.

In the end though, I blame the corperate world as much as the antivirus companies themselves.

Lastly, I used to use Norton a whole lot. And while I also think antiviruses aren't as big of a deal as lets say a good router but I do think they still suck. I have and always will think antiviruses are still pretty useful and i use them but some of them really suck.

 

[Deleted member 178]

My general assessment of this is simple but yet profound.
There are not as many skilled hackers as there are proclaimed ones. If they truly are endowed with a gift, it is generally from being a recluse, keeping to themselves, and very unlikely they would show many that gift, if at all.

It takes a great deal of knowledge to remotely attack an individual's system and gain access to perform malicious actions, and generally needing something to gain from it to make it worth taking the chances, which leaves home users a lot less likely to become targeted from such an attack.

Expert hackers create sophisticated tools then resell them at high price to idiots they call mockingly "skiddies or script kiddies" so the tools can easily penetrate your system via surface attacks and open ports"

If an antivirus can't protect itself, how it can it protect the system from exploitation? AVG and Kaspersky recently included ASLR in their products after a memory vulnerability could cause a peice of malware or attacker to manipulate the av. This is probably not the case with just these.

indeed, all famous AVs can be disabled with the proper targeted attack, as you said Kaspersky was the latest one.

4. True, and yes. Which is why I repsect Comodo so much, they haven't failed. They have the protection that is deserved by the enterprise. But other than Comodo, all antiviruses fail yes.

i would say no, enterprises and corporation won't use it; they can't allow such buggy piece of software ruin their network.

But they still should have some standard, the point isn't that firewalls NEED to be the best. It's that if your gonna include a custom firewall make it better than what's already included.

just use an hardware one

And lastly, Yes. I have used Symantec Endpoint protection. I am good friends with a system administrator for Wells Fargo, who uses it all the time, so I have also used it and did not like the console. I also found its protection was quite mediocre.

The console is horrible yes ^^ for it's protection it depend largely on the admin tweaking skills; using it at default is like having no protections.

I am a fan of Kaspersky's endpoint, advanced console that I think is good for any admin I also feel that its got great protection and don't think I'm putting down Mcafee's firewall, I'm putting down their antivirus thier firewalls are actually quite good at managing and dealing with network intrusions in a network as the function as both effective firewalls and load balancers allowing traffic to be balanced between each firewall so to prevent a firewall from going down, and if it does it will simply switch firewalls.

i rather use Sophos or Symantec endpoint, Kaspersky and all the other are small players in the field.
anyway hardware anti-malware firewall are far better than any software ones.

In the end though, I blame the corperate world as much as the antivirus companies themselves.

Lastly, I used to use Norton a whole lot. And while I also think antiviruses aren't as big of a deal as lets say a good router but I do think they still suck. I have and always will think antiviruses are still pretty useful and i use them but some of them really suck.

In homeland security , you won't see much of the vendors we discuss in this forum on the top , except Malwarebytes and Blue Ridge Network (Appguard) and Invincea (Sandboxie). Those 3 are top products that REALLY stop malwares, it is why i use them after trying hundreds of products.

 

[FreddyFreeloader]

Variability in:

• malware sampling\collection methods
• actual malware samples used
• age of samples

has a lot to do with it.

When, where, and how the labs collect samples influences the test set.

Also, signature-based engine detection varies a bit over time - sometimes significantly.

Finally, hardware, operating system and version of AV does influence test results.

The overall trend over time is the best indicator of protection:

1. or 2. Kaspersky
1. or 2. Bitdefender
3. ... Panda
4. ...
5. ...

3, 4, and 5 always seem to be Avira, ESET and Trend Micro.

 

[AtlBo]

All of these things contribute to the total rating of an antivirus. I would also include personally anti-exploit tests, whether the firewall and behaviour system allows malware to update through a service started by task shceduler. Whether the antivirus is resiliant to attempts to shut the service down.

After all of these, in my eyes the best antivirus doesn't exist. Because of the simple fact every antivirus I have personally test cannot block exploits without signatures or behaviour analysis, can be shut down, can let malware update in the background, doesn't detect every threat through behaviour analysis and has a firewall barely better than Windows.

I understand that no antivirus is perfect, but having these severe issues is not good and I would consider every antivirus on the market a failure for these reasons. Zero day protection is a backup for incase the signature detection fails therefore it should be expected to live up to its expectation

I agree with this basically. After thinking through all of the issues best I am able, I do see hope, though. Products like Malwarebytes Anti-Exploit and the others like it are a step forward.

That said, there is much more to do, for example, securing against spoofing and command prompt based attacks. I saw a security analyst crack a PC in less than 20 seconds on I think it was the Today Show or one of those recently. He wasn't joking around about how easy it is to compromise PCs, especially mobile and wi-fi devices. All he did was inject a packet with his code, wait for it to unload and run, and boom he was looking at the directory of the PC he was attacking. At that point, he can do anything, even connect to remote desktop (although that would kick the user to the logon screen of their PC and alert them to the attack if they knew what the kick means). Anyway, this is what got me about command prompt attacks. That and a video by a self proclaimed hacker on YouTube that explained how he discovered buffer overflowing...all at a command prompt and with the help of a couple of programs.

There is alot of work to do, and there are alot of new standards to be set for security both by ISPs and by PC security software companies. However, I feel we should at least be happy that progress is being made. Baseline security is getting better slowly. The connections element of firewalls should be more configurable, intuitive, and understandable, and this could to a large degree strip the incentive from hackers when it comes to hacking PCs.

Otherwise, I am just sad that a really good hacker can single me out and attack me basically any time, and there isn't much I can do about it except hope things get better someday. I just want to feel like it isn't worth it to them to try it with normal everyday PC owners. Big business is another game, and they can look after themselves with their large security budgets...

Sorry, to the actual topic. a-v tests are one measure of a component of PC protection. I would say the tests are beneficial, but one component of security does not explain the value of a security product overall. This is the same notion mentioned a few times before in the thread...
.

 

[omidomi]

in fact MalwareTips users because we do't earn money from av company

 

[Kiwimike]

Expert hackers create sophisticated tools then resell them at high price to idiots they call mockingly "skiddies or script kiddies" so the tools can easily penetrate your system via surface attacks and open ports"



indeed, all famous AVs can be disabled with the proper targeted attack, as you said Kaspersky was the latest one.



i would say no, enterprises and corporation won't use it; they can't allow such buggy piece of software ruin their network.




just use an hardware one



The console is horrible yes ^^ for it's protection it depend largely on the admin tweaking skills; using it at default is like having no protections.



i rather use Sophos or Symantec endpoint, Kaspersky and all the other are small players in the field.
anyway hardware anti-malware firewall are far better than any software ones.



In homeland security , you won't see much of the vendors we discuss in this forum on the top , except Malwarebytes and Blue Ridge Network (Appguard) and Invincea (Sandboxie). Those 3 are top products that REALLY stop malwares, it is why i use them after trying hundreds of products.

Considering your reply to the second quote, I want to say I said the protection was good but never the software stability. The enterprise is always going to try and use the most stable software they can. I personally use Norton, because I find it to be stable. Kaspersky might be a small player although I would have to disagree but thats another story.
I do like Sophos, their XG firewall is VERY good and I have tried the home version and like it a lot, although they are starting to fluff it up a bit.

I do not like it when consoles have amazing visuals and tend to make it more basic. Norton is used by every fortune 500 company pretty much. So indeed it is advanced.
As I said before, I think ALL antiviruses suck. It's not me saying that I hate the companies, they have good technology and good software engineers and good analysts and researchers. But, its the antivirus themselves. They aren't bad. They just suck, saying they aren't good enough for the industry.

I do also agree with you in terms of using a hardware firewall. I beileve software firewalls are becoming more irrelevant but considering the growth of public wireless networks and the flaws inside of router's simply because companies leave a lot of older routers upatched is becoming more and more common. This means software firewalls should be considered a defense.

Windows firewall is sufficient, never said it wasn't. Just said that if your gonna include a firewall. If you are going to brag and include a firewall people pay for, make it better than whats already included.
I do infact beileve hardware firewalls are much more important and MUCH MORE EFFECTIVE.

I would rather spend $500 building a good firewall, putting pfsense, ipfire or Sophos XG firewall/UTM on it than spend $100 a year for an antivirus.

So in the end I beileve you proved my point, which is that antivirus software is irrelevant. I configure security software often, and have configured endpoint security in the past. I'm not saying the software is bad, I'm saying its not effective enough. I have ClamAV running on my firewall blocking websites and downloads that are malicious, I have Trend Micro built into my router (used for wifi) which is good too!

I'm just saying antivirus software isn't as good as it should be. I'm not saying they are horrible peices of software. I HIGHLY RESPECT THE SECURITY COMPANIES but I don't think their software is good enough.

You saying "use a hardware one" and "In homeland security , you won't see much of the vendors we discuss in this forum on the top , except Malwarebytes and Blue Ridge Network (Appguard) and Invincea (Sandboxie). Those 3 are top products that REALLY stop malwares, it is why i use them after trying hundreds of products"
Totally proves my point! Which is also the same point you are making! I too use a hardware firewall and even have my firewall's off on my linux box simply because it doesn't really matter.

 

[Deleted member 178]

Considering your reply to the second quote, I want to say I said the protection was good but never the software stability. The enterprise is always going to try and use the most stable software they can. I personally use Norton, because I find it to be stable. Kaspersky might be a small player although I would have to disagree but thats another story.
I do like Sophos, their XG firewall is VERY good and I have tried the home version and like it a lot, although they are starting to fluff it up a bit.

Sophos are generally good products, one of the few vendors you an rely on.

As I said before, I think ALL antiviruses suck. It's not me saying that I hate the companies, they have good technology and good software engineers and good analysts and researchers. But, its the antivirus themselves. They aren't bad. They just suck, saying they aren't good enough for the industry.

industry or consummer , they indeed suck. i keep saying it since a while even if i sometimes try some of them to see potential improvements but i often remove them after a while. not one satisfying me. i rather use Virtualization + anti-executables, far more effective than traditional AVs.

This means software firewalls should be considered a defense.

of course, you can't carry your hardware firewall and ask the guy of the coffee shop "can i reroute your traffic via my router " ^^

Windows firewall is sufficient, never said it wasn't. Just said that if your gonna include a firewall. If you are going to brag and include a firewall people pay for, make it better than whats already included.
I do infact beileve hardware firewalls are much more important and MUCH MORE EFFECTIVE.

WF needs a lot of tweakings and be adapted to the system...few people will took the time to do it.

I would rather spend $500 building a good firewall, putting pfsense, ipfire or Sophos XG firewall/UTM on it than spend $100 a year for an antivirus.

indeed me too, if my budget was enough a good hardware firewall like PFsense, Redsock would solve the weak chain in my security setup...

So in the end I beileve you proved my point, which is that antivirus software is irrelevant.

hence my signature's motto

I'm just saying antivirus software isn't as good as it should be. I'm not saying they are horrible peices of software. I HIGHLY RESPECT THE SECURITY COMPANIES but I don't think their software is good enough.

same here, i even ditched my favorite vendor (Emsisoft) because they took the "simplistic way" , i cant blame them they need customers.

You saying "use a hardware one" and "In homeland security , you won't see much of the vendors we discuss in this forum on the top , except Malwarebytes and Blue Ridge Network (Appguard) and Invincea (Sandboxie). Those 3 are top products that REALLY stop malwares, it is why i use them after trying hundreds of products"
Totally proves my point! Which is also the same point you are making! I too use a hardware firewall and even have my firewall's off on my linux box simply because it doesn't really matter.

i dont have a hardware one yet, since i move between 2 countries. and the one i like is far over my budget ;
trying to use PFsense in a VM to filter my traffic
Problem is that many traditionnal AV vendors have good marketing tricks that vary from fancy statements (we stop all threats) to scareware tactics (if you dont use us , you are a potential victim of..)

 

[jamescv7]

You can see the strong points of a product from an ordernary user test? This makes little sense to me considering they are using malware databases that have a high chance of being on most of the better antivirus' signatures.

@cruelsister @MrXidus and few others even though they are ordinary users that conducts reviews on Youtube, hence they provide in-depth analysis to make sure the product is worth convincing and accurately tested compare on independent-testing organizations.

The thing makes it well reliable to find the source is removal test, cause AV's today mainly focus on prevention and not on cure which why they develop separate product for that.

Sometimes better to have new outcome of a problem than the common results we saw everyday unless there's a consistency on performance.

 

[illumination]

Expert hackers create sophisticated tools then resell them at high price to idiots they call mockingly "skiddies or script kiddies" so the tools can easily penetrate your system via surface attacks and open ports"

Lets have a showing of hands by home users that have been directly attacked by a hacker that believed he/she had enough to gain from doing so. I mentioned once already in this thread, and will state it again, fear mongering is not healthy... What do i mean by this, well i will use one of those car analogies that you are fond of..

"There is a chance driving down the road at 50 MPH that my front right tire could blow out and cause me to crash my vehicle.
I should...
A: drive with care knowing this potential lies there although very uncommon..
B: Get a tire made out of solid rubber, which will diminish the driving performance, lower my gas mileage, make the ride very unjoyable..

Bottom line is, by stating no matter what a user does to secure their system ,a hacker "can potentially" bust through any thing a user does to protect their system, leave the user with a "why should i bother then" attitude, which in turn leaves them vulnerable for the potential pitfalls they really should be concerned with, as well as their family and friends.


Hackers are not going to risk going to prison for 20 years to bust into John Does home network to try and steal his $32 dollars from his bank account...

Now trying to add John Does system to a Botnet would be another matter, typically done through social engineering and malware infections, which do and can be stopped..

 

[Deleted member 178]

well i will use one of those car analogies that you are fond of..

"There is a chance driving down the road at 50 MPH that my front right tire could blow out and cause me to crash my vehicle.
I should...
A: drive with care knowing this potential lies there although very uncommon..
B: Get a tire made out of solid rubber, which will diminish the driving performance, lower my gas mileage, make the ride very unjoyable..

do A+ B

Now trying to add John Does system to a Botnet would be another matter, typically done through social engineering and malware infections, which do and can be stopped..

That is what i was thinking about.

also skiddies are idiots , so with idiots every motives are possible...

 

[illumination]

do A+ B

I prefer an enjoyable ride, and for my vehicle to sustain without unneeded wear and tear.

also skiddies are idiots

Indeed.. Lacking skills to code their own tools, 99% of the time not fully understanding tools they do get their hands on "chances of succeeding" very minimal at best.. Chances of being caught, quite high..

*Moral of this drawn out thread"
Baseline security and good surfing habits are enough for home users. A user with good enough surfing habits could use any of the security products list in the AV tests and be fine. Learning good surfing habits should become number one priority.

 

[LabZero]

Well,today it has become almost impossible to close all ways through which a malware can penetrate a network and on an endpoint through drive-by-downloads, or other techniques. The main reason is that in the era of social media, mobile and cloud, the surface area is expanded at a higher speed than with which many security vendors manage to provide countermeasures to their customers. The most common firewalls and Ips (or port-based) classify the types of traffic based on ports used usually by individual programs. Depending on the level of reliability or otherwise of a program, the firewall allows or inhibits the flow of data through the port used by the object. The Ips goes deeper. Assuming that an application can handle certain types of threats and not others, it administers a circumscribed set of signature to the port that is used by the application: it doesn't try all signature on all ports.

Complicating the situation is the fact that more and more applications are designed to evade traditional security systems through the port hopping. Change that is continually the ports to prevent the intervention of firewall, antivirus or Ips get worse the "user experience".

Thanks to social engineering, hackers manage to exploit applications using the port-hopping or those that use (often in combination with port hopping) encryption to penetrate the network malware targets the type of attack you want. This may consist, for example, in permitting, step by step, direct control of target by the attacker, or transform one or more machines in the bot intended to form part of a botnet, which is controlled by remote servers, with which to carry out actions against foreign users or businesses (spam, DDoS etc).

The first step of defense in this new scenario is realizing that, increasingly, malware do not act individually, but on the net and with coordinated actions. The new security soutions, therefore, must offer at the same time the ability to analyze every movement as a whole, and in case of need penetrate deep to fight back in a targeted and effective mode.

 

[Kiwimike]

Sophos are generally good products, one of the few vendors you an rely on.



industry or consummer , they indeed suck. i keep saying it since a while even if i sometimes try some of them to see potential improvements but i often remove them after a while. not one satisfying me. i rather use Virtualization + anti-executables, far more effective than traditional AVs.



of course, you can't carry your hardware firewall and ask the guy of the coffee shop "can i reroute your traffic via my router " ^^



WF needs a lot of tweakings and be adapted to the system...few people will took the time to do it.



indeed me too, if my budget was enough a good hardware firewall like PFsense, Redsock would solve the weak chain in my security setup...



hence my signature's motto




same here, i even ditched my favorite vendor (Emsisoft) because they took the "simplistic way" , i cant blame them they need customers.



i dont have a hardware one yet, since i move between 2 countries. and the one i like is far over my budget ;
trying to use PFsense in a VM to filter my traffic
Problem is that many traditionnal AV vendors have good marketing tricks that vary from fancy statements (we stop all threats) to scareware tactics (if you dont use us , you are a potential victim of..)

Agreed, Although saying that. I think Sophos antivirus home could use some extra feautres. But it is for home users, it functions and it has a basic but good looking management console so its better than most in my eyes still.

Running Pfsense in a vm isn't bad. The trend on my router is from my Asus router which is on, but I also have a router running openWRT. If you don't have a large budget thats a really awesome way to go. Or ddwrt, tomato etc. Openwrt is my favourite though because it's got full linux freedom that I love and crave.

One thing to note however, is it does allow ping and doesn't drop probes by default. It uses the IPsec firewall though, which is very good and awesome although it still does have a gui on the more stable and later releases which makes it easy to manage everything. But no https and self-signed certificate unfortunately.

I still use basic Firewall mechanics in my Windows Firewall, which is block everything unless its specified in the rules. This is also how Windows firewall comes by default which is actually very effective. This is unless a program or peice of malware creates a rule and doesn't remove it or abuses the rule to connect to the pc remotely.

I do agree with you on the tactics as well. I hate it how a lot of antivirus vendors use their blogs as ways to trick people into using antivirus software. I use Linux on all my computers so I don't feel myself needing security software just lots of patches and my software firewall is on and configured.
Simply because of the fact I'm in a home network and often get script-kid attacks thrown at me even though most times it's too outdated and can't affect my system, but it will sometimes lag out my system in the same way if your on someones server and you probe it or attack it mid-way it will indeed hiccup a little.

@cruelsister @MrXidus and few others even though they are ordinary users that conducts reviews on Youtube, hence they provide in-depth analysis to make sure the product is worth convincing and accurately tested compare on independent-testing organizations.

The thing makes it well reliable to find the source is removal test, cause AV's today mainly focus on prevention and not on cure which why they develop separate product for that.

Sometimes better to have new outcome of a problem than the common results we saw everyday unless there's a consistency on performance.

I shouldn't have made it so broad saying all users. I think as long as someone uses effective methods of testing security software it is fine. Just most of them use basic malware that is nothing much.

 

[jackuars]

I don't know if someone really read my post before. So I was asking about the effectiveness of testing for antivirus on a virtual machine like most of the MalwareTips reviewers does? Aren't there malware's that can bypass virtual machine testing?

 

[Deleted member 178]

i would never test an AV in VM, results are inaccurate; some products don't even work properly in them.

ask yourself:

"do car vendors do only crash tests on computer or in real situation?"

 

[cruelsister]

Kiwimike- I have to agree with you that some User testing methods are less than optimal. Frequently it seems to me that emphasis is placed on sample quantity instead of sample quality (and this goes for some Pro testing).

Personally I would rather see a smaller number of samples used that contain distinct (and confirmed) malware types instead of a couple of hundred unknowns that may consist of numerous variants of the same old thing. And as far as true zero-day testing is concerned, the only realistic way of doing this is by the tester coding malware herself.

Umbra- Malwarebytes??? Just don’t run into any Worms.

Jackuars-
VM aware malware don't bypass the VM as such- those malware that are VM aware essentially will query their environment and if they determine that they are either in a VM and/or a sandbox won’t run (there was a ransomware sample recently that ran fine within a VM but would shut down within a sandbox).
When a potential malware sample does not seem to run in a VM it is a pain to determine whether it is VM aware or just a dud. Easy way is just to exclude it from testing.

 

[CMLew]

"do car vendors do only crash tests on computer or in real situation?"

They do both That's what they called simulation vs reality. Haha...

 

[jackuars]

Kiwimike- I have to agree with you that some User testing methods are less than optimal. Frequently it seems to me that emphasis is placed on sample quantity instead of sample quality (and this goes for some Pro testing).

Personally I would rather see a smaller number of samples used that contain distinct (and confirmed) malware types instead of a couple of hundred unknowns that may consist of numerous variants of the same old thing. And as far as true zero-day testing is concerned, the only realistic way of doing this is by the tester coding malware herself.

Umbra- Malwarebytes??? Just don’t run into any Worms.

Jackuars-
VM aware malware don't bypass the VM as such- those malware that are VM aware essentially will query their environment and if they determine that they are either in a VM and/or a sandbox won’t run (there was a ransomware sample recently that ran fine within a VM but would shut down within a sandbox).
When a potential malware sample does not seem to run in a VM it is a pain to determine whether it is VM aware or just a dud. Easy way is just to exclude it from testing.

This is what I've been referring to.... Cant' they defeat our reviewer's tests and give inaccurate results?