Why are Independant lab results unreliable?

jackuars

Level 27
Thread author
Verified
Top Poster
Well-known
Jul 2, 2014
1,689
I'm not interested in claims that they grab money for publishing results, but what could be the other reasons that they are unreliable in testing the overall effectiveness of an antivirus software? Can you give me some examples?

Also in what ways are User Reviews at MalwareTips better or worse than the ones' published at Independent Labs?

Time and again this debate comes up in our forums so let's put and end to that?
 

Kiwimike

Level 1
Verified
Dec 15, 2015
29
I don't know if someone really read my post before. So I was asking about the effectiveness of testing for antivirus on a virtual machine like most of the MalwareTips reviewers does? Aren't there malware's that can bypass virtual machine testing?
There have been bugs that malware could exploit to bypass virtual machines, but I wouldn't worry about this in standard malware databases. This is more on a corporate level probably, as a lot of game servers are run inside of virtual machines in large quantity. So, if malware could be custom made to bypass lets say a VMware virtual machine on a Windows or Linux server, or possibly even Virtual Box on a Solaris system but I doubt that would be the case.

There is a pyramid. It goes from government made and developed malware and techniques down to consumer. Business, Corporate are in between. Business meaning companies like the now extinct Blackhole make exploits and the new ones that I don't really pay much attention to.

So, if you were downloading a fake crack over p2p that would have a higher chance of being something dangerous than on a malware database, that's my opinion at least.

I would never test an AV in VM, results are inaccurate; some products don't even work properly in them.

ask yourself:

"do car vendors do only crash tests on computer or in real situation?"

Car vendors do both, but that isn't really the point. But I do get what your trying to say. And I totally agree, malware can detect when its running inside of a virtual machine.

Also, I hate virtual machines although I run them every day. They are sluggish, and slow and heavy and have problems with drivers and compatibility and optimization. Especially had horrible experiences with VirtualBox.

So, the point is that if your gonna test out antiviruses build a virus lab.

It's not that hard to build a virus lab at your house, provided you have the hardware and software availible. Which is easy to get or build but can cost a bit which is why people tend not to do it.

I know the guy who owns ThePCSecurity Channel and sometimes help him with information regarding AVG products. He tells me he doesn't test professionally, he tests them as a broad spectrum, not to compare software.


Kiwimike- I have to agree with you that some User testing methods are less than optimal. Frequently it seems to me that emphasis is placed on sample quantity instead of sample quality (and this goes for some Pro testing).

Personally I would rather see a smaller number of samples used that contain distinct (and confirmed) malware types instead of a couple of hundred unknowns that may consist of numerous variants of the same old thing. And as far as true zero-day testing is concerned, the only realistic way of doing this is by the tester coding malware herself.

Umbra- Malwarebytes??? Just don’t run into any Worms.

Jackuars-
VM aware malware don't bypass the VM as such- those malware that are VM aware essentially will query their environment and if they determine that they are either in a VM and/or a sandbox won’t run (there was a ransomware sample recently that ran fine within a VM but would shut down within a sandbox).
When a potential malware sample does not seem to run in a VM it is a pain to determine whether it is VM aware or just a dud. Easy way is just to exclude it from testing.

I agree, I'll say that Malwarebytes isn't meant to be a full on antivirus.

But I'll be honest, I find Malwarebytes pretty slack in terms of detecting high-level malware saying that it makes a good scanner against the stuff antivirus software normally misses which is exactly what its designed to do.

And yes, I would like to see some actual exploit kits get tested primarily.

I don't care about someone downloading and running malicious binaries as much as I do an antivirus blocking a complicated exploit kit.

PEOPLE CAN SAY AS MUCH AS THEY WANT. BUT PEOPLE PAY FOR EXPLOIT KITS, DEPLOY THEM AND THEY GET PEOPLE ALL THE TIME. There are so many clients infected with these things that Norton can't detect!! It's sonar is useless against it!
And it adds the system to a botnet.

Before you know it some cracker could RDP through the network if they were able to which I doubt would be too hard. And then get the domain controller, get in somehow get themselves a damn golden ticket or permission elevation and THEN REROUTE THE WHOLE COMPANIES EMAILS THROUGH THEM!!

Apologies for my ranting. I get very angry at how some antiviruses are letting exploit kits bought for a mere $5,000 infect company networks lol. I'm saying lol because I'm actually incredibly stressed.


This is what I've been referring to.... Cant' they defeat our reviewer's tests and give inaccurate results?

It's about opinion and perspective. Both tests in reality are both useful. They just are different ones. This means none is unreliable, but both are kind of sucks. Meaning that you probably should take it both and just decide from that.

Or even better, find a friend or researcher with a malware lab and get them to test it out for you.

MBAM is a complement, that is true for their home user or Endpoint version. no one is foolish enough to use it as main protection.

I've seen it in businesses but never in corporate. Saying that, it wouldn't hurt considering how horrible antiviruses are doing at this point in time.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top