Would enabling drag and drop from host to guest be a bad idea? I do that but disable the option before testing. I also have 16 gigs a RAM so I don't worry much about resources. The VM runs pretty fast.
All these Guest -> Host and Host -> Guest features are all large attack vectors because they actually provide functionality for the communication. Would not be surprised if a government agency already figured out how to exploit it with some sort of attack like exploitation of the ROP chain. A VM will never be bullet-proof but I doubt any single individual here will encounter a VM escape if its secure by disabling such features like the drag and drop, shared clipboard, shared folders, etc.
You can also use a Linux system with no hard-disk which boots off a USB as a precaution against hardware firmware hacking LOL. I've never done it but I know you definitely can.
The issue I think is vulnerabilities in the CPU chips which cause so much hassle with virtualisation security, have you seen the latest Intel vulnerability news? It's all over the News now. And Virtual Machines are only possible because of usage of technology which is implemented into the CPU hardware (Intel VT-x, AMD SVM, etc). However, AMD seems to be quite safe compared to Intel IMO.