Tony- There is a misconception regarding VM aware malware, in that many think that it will break out of containment. This isn't the case at all- the malware instead will query the environment that it is running in, and if it is determined that it's in a VM it will just shut down. It does this in order to make the user think it is harmless so that he (and certainly not a "she"!) will then run it on a real system and get infected.
Malware can use numerous methods to detect VM's- from simple things like looking for running processes of stuff like Vbox or VMware to things like a Chimera variant did: it measures the time lapse (via GetTickCount) between a series of actions and if they are slower than expected (as VM's are less powerful than actual machines) it will terminate.
About FireEye- it is certainly extremely good but as it works through an appliance and one must pay for multiple seats it really is a Corporation only product (and quite expensive). For Home users Comodo is in my opinion the best choice as with the proper settings it will just "Do" and not "Ask". Make no mistake- Comodo can also be breached- but only with extreme difficulty and under certain circumstances. AV products are a great deal easier to bypass.
Done- Some Encryptors are poorly coded and actually store the encryption key on your computer where it can be fished out; but these are rare and you would have to know exactly what your infection was. But generally if you had System Restore turned off and don't have any external backups your files are lost. I'm really sorry.