Do you agree?

  • Total voters

Deleted member 178

What is a real world test to me ? there is my take (simplified PoV , i don't want add much technical knowledge).

1- Take several average users
2- teach them the basis of how works the native security as you would give the driving license learning book to wannabe drivers (not necessary but will give an idea of the user behavior after being informed)
3- let them surf as they do at home.
4- sometimes send them some phishy emails with malicious links/samples or plugin a infected USB.
5- observe the behaviors of the users and how respond the software. this is the most important point.

Security softwares always reacts after the user behavior.

That is real world test and no labs could ever produce it because it is lot of time and wasted resources.
What youtesters/labs does is an extrapolation of this test , so it is inherently inaccurate by design.
Average users won't encounter hundreds of samples in their life , and surely not 5 in less than 10mn...

Some labs are using methodologies that seems more professional and neutral than others. It is your duty to read carefully the test then get a clear opinion to comment with.

But remember, all tests were made in an specific point in time. So just take it as information not as a forever rule because as you know things are always evolving.
I find funny some people bashing products by pointing tests months/years old...

feel free to add your opinion.
Last edited by a moderator:


Level 28
Top poster
Content Creator
Mar 1, 2014
I have a problem with the second criterion, about teaching the average users. I think, in the real world, majority of users have no formal knowledge of what computers are and what they should do to prevent security leaks.
Last edited:

Deleted member 178

I have a problem the second criterion, about teaching the average users. I think, in the real world, majority of users have no formal knowledge of what computers are and what they should do to prevent security leaks.

Help file of Windows? nobody read it... if i was MS i would do a mandatory security tutorial mode after installation of Windows (like in video games). but people will get angry because 5 mn of explanation is already too much...


Level 30
Top poster
Content Creator
Jul 28, 2014
I agree with @XhenEd I think teaching the users might skew the test a little, at the most I would just provide them a link to the AV's official site and let them have the choice of reading it or not.


Level 20
Top poster
Oct 16, 2015
Testing security software in the manner which you described could be equivalented to the "clinical trials" used for pharmaceutical approvals.

Security software developers could analyze the behavior of the user and the software, and the interaction thereof, both on a per case basis (e.g. how would this specific user, who has this specific bad browsing habit, for example, respond in this specific situation) or statistically (e.g. how did the majority of users respond to this potential threat).
This data could originate from such "real-life simulations" conducted by the security software companies, or push it a step further with larger scale experiments of this same nature.

Thing is, security software is not nearly as critical as pharmaceutics, and DEFINITELY not worth the cost (financial cost, and cost of time) of such large scale testing methods.

In other words, it sounds great in theory and all, but in practice it might be quite overkill. (not saying that there wouldn't be any benefits from it - actually the opposite - however the "marketing" and "financial" executives of security software vendors would probably disapprove of this method...)


Level 85
Mar 15, 2011
I like the point.

Although there is no more involvement on AV's performance, as it directs on how a person perform/aware on computer habits.

Still the test of independent organization on the other hand have different purpose, which overall it links to the person's knowledge on how things goes work and in what ways to prevent any threats.


Level 3
Jan 27, 2017
Because I...

1. had so many 'the light globe came on' moments of thinking after reading Umbra's take of a real world test in another thread
2. asked the original questions that were able to illicit the information from the deep corners of The Umbra Mind
3. took the risk of being seriously slapped :)
4. really, really, really (add several more reallys!) want to keep this discussion alive...

I just have to ask MT member's on this thread the following questions:

What would be your intention or ultimate purpose for conducting a "real world test"?
How would you improve Umbra's guidelines... how would you make them even better?

@Umbra if these questions take the thread too far away from the poll questions I promise to take my slapping like a man. ;)

Deleted member 178

My vision is just mine, not a perfect one but i felt it is realistic enough in term of procedure (not on resources and time involved), many will disagree for various reasons and i respect that.
  • Like
Reactions: Andy Ful