I apologize for not having a ton of time to dive into a deeper discussion, but it might help you to know that tests are helpful to an extent. It is unfortunately a bit simpler to evade a security tool like antivirus with a one-off or targeted distribution of malware, much harder after a wide deployment from an attacker. This is because there is herd protection in place...the endpoint companies are looking for trends, and trying to react accordingly.
I've learned in my career to stand firm on things I know, regardless of whether people say I'm wrong (until they learn I'm right, sometimes years later)...and to admit when I'm out of my specialty...so...
I suspect influenza vaccines are of similar tradecraft, where next year's strains and evolution are monitored, and protections prepared for the people that haven't been exposed yet (cue nCov19 coming out of nowhere causing issues). Again, I am not a virologist.
So, the point is, we can produce a demo that shows us evading even the most well marketed antivirus tools.
Is that realistic? To an extent. If we start attacking people, eventually we'll catch the antivirus vendors' attention, and now be in the same game of manipulating our code just like the malware authors, and monitoring things like Virustotal for hits.
Is it fair? I certainly see "scummy" tactics like a one-off the vendor has never seen to earn some salesperson a sale. Does that mean either tool (competitor or salesperson's tool) is being given a fair assessment? No.
Does it provide an accurate assessment of the tool's capabilities? Sort of. The tools largely use similar datasets, and different capabilities attack the malware tradecraft elements in different ways. If architects were held to the same level of marketing truthfulness as security products, I suspect we'd have a lot of dead people and collapsed buildings. That creates issues for us as product developers. I call it, "Cyber Jesus" sales pitches, and unfortunately they work a lot. Why wouldn't they? The clients don't know any better.
As a customer, if you are going to try to extend your coverage, it might be good to look at installing different products that approach malware from different perspectives. Or just install one that seems to work well for you, and rely on the herd mentality. It really is diminishing returns when installing multiple products of the same genre (antivirus, for example), and you are relying on variables out of your control more than anything, if you are hoping Vendor A catches a pieces of malware that Vendor B temporarily does not catch.
So, how has that affected us in our company, now that I switched to product development, from "just" malware researcher? (I *really* hope someone by now is saying, 'how is this jerk different then?')
Well, what we've done is say, "what is the place we can attack the malware, that requires the most resources for the attackers to pivot around us?"
What we see if that it takes a lot longer for a ransomware tool developer to totally strip out their encryption code and replace with an entirely new library, versus changing filenames. Yes, changing filenames work on a surprisingly large number of products....
I'll cut discussion of what we do in our company short, but hit us up sometime for a longer discussion if you like. You just gotta get in line behind the competitors and criminals trying to ask questions...sigh.
Anyway, I hope that helps provide some context.
