Why Antivirus Software uses so Much RAM- It's a good Thing: Emsisoft
I found this an interesting article: A Memory Usage comparison with Protection, Cloaking of Actual Memory Utilization and the relevance of the Page File.
Good high memory usage vs. bad high memory usage
Let’s conclude what we have learned so far: RAM is fast, make use of it! Reducing memory usage from e.g. 70% down to 40% doesn’t get you any advantage, as free RAM is wasted dead material. It doesn’t save you any power nor does it provide any performance improvements. From that point of view: Make sure you’re using as much RAM as possible to get the best overall system performance.
But there’s a tipping point when it’s maxed out and Windows starts to use the page file. You can avoid Windows hitting this point frequently by making sure you have enough RAM installed. RAM is cheap to buy and a bigger RAM module is probably the easiest way to extend the lifetime of your old computer for another year or two. For example, I’m a heavy computer user but I rarely need more than 4 GB of RAM.
Why does antivirus/anti-malware software need so much RAM after all?
We often hear customers blaming our software for using too much RAM. Well, we want to detect malware. To do that, we need recognition/search patterns to compare files with our database of known threats. Those patterns (sometimes called fingerprints or signatures) are not really that big, but there is a really huge number of threats out there, and therefore we need many signatures too.
At present, the Emsisoft protection software uses more than 7 million malware signatures. To load them all into RAM, it needs a bit more than 200 megabytes. That sounds like a lot, but keep in mind that this equals a short sequence of 28 bytes on average that we can use to confirm whether a file is good or bad. To illustrate that: Imagine a text sequence of just 28 letters that must be found in a library of 1 billion books, and you are not allowed to come up with a single false detection. A malware scanner has to check 7 million signatures against each of roughly 300,000 files on your hard disk, – all within a fraction of a second!
Technically there is no way to make 7 million signatures suddenly disappear. They must be stored somewhere if you want a really good detection rate instead of an absolute minimum (as seen in Windows Defender). They also need to be accessed somewhere quickly, so they can scan every new and modified file that enters the computer. Fast enough, so you don’t even notice that something was scanned in the background. The place to do this is the RAM.
The challenge with RAM usage doesn’t only affect Emsisoft, it’s an industry-wide issue. All signature-based antivirus or anti-malware scanners naturally require a significant amount of RAM to protect your computer effectively.
An insider’s secret: Antivirus programs tend to hide their RAM usage
High memory usage is bad for marketing, but what do you do if you can’t avoid it? You hide it. There are two major techniques to make a big program look like a small one:
The good thing about malware is that many samples appearing in the real world (outside labs) are very similar. There is a limited number of malware families and often samples just differ in a few bytes of data. That means we can detect large numbers of threats with fewer, but smarter signatures. Using that method, the number of required signatures for best detection don’t grow as fast as the total number of threats out there in the wild.
There is a company who I believe uses the Page File as a source of a lot of memory. I won't mention the name, but a process of it frequently hangs upon shutdown.
Whether this is truth or hype, I cannot answer, but it does make some sense. It's news from a reputable vendor.
Source: Emsisoft Blog Why antivirus uses so much RAM – And why that is actually a good thing!
I found this an interesting article: A Memory Usage comparison with Protection, Cloaking of Actual Memory Utilization and the relevance of the Page File.
Good high memory usage vs. bad high memory usage
Let’s conclude what we have learned so far: RAM is fast, make use of it! Reducing memory usage from e.g. 70% down to 40% doesn’t get you any advantage, as free RAM is wasted dead material. It doesn’t save you any power nor does it provide any performance improvements. From that point of view: Make sure you’re using as much RAM as possible to get the best overall system performance.
But there’s a tipping point when it’s maxed out and Windows starts to use the page file. You can avoid Windows hitting this point frequently by making sure you have enough RAM installed. RAM is cheap to buy and a bigger RAM module is probably the easiest way to extend the lifetime of your old computer for another year or two. For example, I’m a heavy computer user but I rarely need more than 4 GB of RAM.
Why does antivirus/anti-malware software need so much RAM after all?
We often hear customers blaming our software for using too much RAM. Well, we want to detect malware. To do that, we need recognition/search patterns to compare files with our database of known threats. Those patterns (sometimes called fingerprints or signatures) are not really that big, but there is a really huge number of threats out there, and therefore we need many signatures too.
At present, the Emsisoft protection software uses more than 7 million malware signatures. To load them all into RAM, it needs a bit more than 200 megabytes. That sounds like a lot, but keep in mind that this equals a short sequence of 28 bytes on average that we can use to confirm whether a file is good or bad. To illustrate that: Imagine a text sequence of just 28 letters that must be found in a library of 1 billion books, and you are not allowed to come up with a single false detection. A malware scanner has to check 7 million signatures against each of roughly 300,000 files on your hard disk, – all within a fraction of a second!
Technically there is no way to make 7 million signatures suddenly disappear. They must be stored somewhere if you want a really good detection rate instead of an absolute minimum (as seen in Windows Defender). They also need to be accessed somewhere quickly, so they can scan every new and modified file that enters the computer. Fast enough, so you don’t even notice that something was scanned in the background. The place to do this is the RAM.
The challenge with RAM usage doesn’t only affect Emsisoft, it’s an industry-wide issue. All signature-based antivirus or anti-malware scanners naturally require a significant amount of RAM to protect your computer effectively.
An insider’s secret: Antivirus programs tend to hide their RAM usage
High memory usage is bad for marketing, but what do you do if you can’t avoid it? You hide it. There are two major techniques to make a big program look like a small one:
- Use the page file: As described earlier, Windows puts less frequently used parts of programs onto the slow hard disk. Programs can also force that process and ‘ask’ Windows to swap them to the pagefile in regular intervals. Then the Windows Task Manager shows a very low memory usage, but the price for that is regular 1-3 second ‘thinking-periods’ when you access the program. That’s the amount of time needed to read the data from the harddisk again.
Reduced memory usage
In Emsisoft Anti-Malware and Emsisoft Internet Security, you have full control over that feature. When you turn off the “Memory usage optimization” in main settings, the software doesn’t initiate swapping to the page file. This means overall system performance is likely to increase if you have enough RAM.
- Use system drivers: Windows Task Manager only shows active programs and services, but not drivers. Drivers are code modules that are loaded directly by the operating system for certain core functionality. Some anti-virus vendors load hundreds of megabytes of data in their drivers to create the illusion of low memory usage. You can spot these by summing up the memory usage of all active programs and compare that with the value of total used RAM. If there is a huge difference, something is probably hiding high memory usage from you.
The good thing about malware is that many samples appearing in the real world (outside labs) are very similar. There is a limited number of malware families and often samples just differ in a few bytes of data. That means we can detect large numbers of threats with fewer, but smarter signatures. Using that method, the number of required signatures for best detection don’t grow as fast as the total number of threats out there in the wild.
There is a company who I believe uses the Page File as a source of a lot of memory. I won't mention the name, but a process of it frequently hangs upon shutdown.
Whether this is truth or hype, I cannot answer, but it does make some sense. It's news from a reputable vendor.
Source: Emsisoft Blog Why antivirus uses so much RAM – And why that is actually a good thing!
