My take on "what should be a real world test"

Do you agree?


  • Total voters
    14
D

Deleted member 178

Thread author
What is a real world test to me ? there is my take (simplified PoV , i don't want add much technical knowledge).


1- Take several average users
2- teach them the basis of how works the native security as you would give the driving license learning book to wannabe drivers (not necessary but will give an idea of the user behavior after being informed)
3- let them surf as they do at home.
4- sometimes send them some phishy emails with malicious links/samples or plugin a infected USB.
5- observe the behaviors of the users and how respond the software. this is the most important point.

Security softwares always reacts after the user behavior.

That is real world test and no labs could ever produce it because it is lot of time and wasted resources.
What youtesters/labs does is an extrapolation of this test , so it is inherently inaccurate by design.
Average users won't encounter hundreds of samples in their life , and surely not 5 in less than 10mn...

Some labs are using methodologies that seems more professional and neutral than others. It is your duty to read carefully the test then get a clear opinion to comment with.

But remember, all tests were made in an specific point in time. So just take it as information not as a forever rule because as you know things are always evolving.
I find funny some people bashing products by pointing tests months/years old...

feel free to add your opinion.
 
Last edited by a moderator:

XhenEd

Level 28
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Mar 1, 2014
1,708
I have a problem with the second criterion, about teaching the average users. I think, in the real world, majority of users have no formal knowledge of what computers are and what they should do to prevent security leaks.
 
Last edited:
D

Deleted member 178

Thread author
I have a problem the second criterion, about teaching the average users. I think, in the real world, majority of users have no formal knowledge of what computers are and what they should do to prevent security leaks.

Help file of Windows? nobody read it... if i was MS i would do a mandatory security tutorial mode after installation of Windows (like in video games). but people will get angry because 5 mn of explanation is already too much...
 

Atlas147

Level 30
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Jul 28, 2014
1,990
I agree with @XhenEd I think teaching the users might skew the test a little, at the most I would just provide them a link to the AV's official site and let them have the choice of reading it or not.
 
D

Deleted member 178

Thread author
I agree with @XhenEd I think teaching the users might skew the test a little, at the most I would just provide them a link to the AV's official site and let them have the choice of reading it or not.
good point.

my point was more tell them what UAC or smartscreen does.
 
  • Like
Reactions: Andy Ful and XhenEd

DracusNarcrym

Level 20
Verified
Top Poster
Well-known
Oct 16, 2015
970
Testing security software in the manner which you described could be equivalented to the "clinical trials" used for pharmaceutical approvals.

Security software developers could analyze the behavior of the user and the software, and the interaction thereof, both on a per case basis (e.g. how would this specific user, who has this specific bad browsing habit, for example, respond in this specific situation) or statistically (e.g. how did the majority of users respond to this potential threat).
This data could originate from such "real-life simulations" conducted by the security software companies, or push it a step further with larger scale experiments of this same nature.

Thing is, security software is not nearly as critical as pharmaceutics, and DEFINITELY not worth the cost (financial cost, and cost of time) of such large scale testing methods.

In other words, it sounds great in theory and all, but in practice it might be quite overkill. (not saying that there wouldn't be any benefits from it - actually the opposite - however the "marketing" and "financial" executives of security software vendors would probably disapprove of this method...)
 

jamescv7

Level 85
Verified
Honorary Member
Mar 15, 2011
13,070
I like the point.

Although there is no more involvement on AV's performance, as it directs on how a person perform/aware on computer habits.

Still the test of independent organization on the other hand have different purpose, which overall it links to the person's knowledge on how things goes work and in what ways to prevent any threats.
 

reboot

Level 3
Verified
Well-known
Jan 27, 2017
139
Because I...

1. had so many 'the light globe came on' moments of thinking after reading Umbra's take of a real world test in another thread
2. asked the original questions that were able to illicit the information from the deep corners of The Umbra Mind
3. took the risk of being seriously slapped :)
4. really, really, really (add several more reallys!) want to keep this discussion alive...

I just have to ask MT member's on this thread the following questions:

What would be your intention or ultimate purpose for conducting a "real world test"?
How would you improve Umbra's guidelines... how would you make them even better?

@Umbra if these questions take the thread too far away from the poll questions I promise to take my slapping like a man. ;)
 
D

Deleted member 178

Thread author
My vision is just mine, not a perfect one but i felt it is realistic enough in term of procedure (not on resources and time involved), many will disagree for various reasons and i respect that.
 
  • Like
Reactions: Andy Ful

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top