It has pretty good detection now, it's absolutely not intrusive.
1. Microsoft have an automatic file upload feature in Windows Defender for their cloud.
2. SmartScreen naturally requires telemetry to function to the best of its ability.
3. Windows Update shares information about your Windows Defender installation to receive the latest update definitions.
Microsoft collect information about how Windows Defender is used the same way that most other vendors collect information about how their products are used, and they also have cloud features which involves file sharing.
Windows in general collects a lot of information - nowadays, Microsoft have the ability to see what applications you've recently ran and with what documents you've used those applications with, and they've made this visible with the recent Windows 10 cloud features (e.g. cloud clipboard, recently used, etc.). It definitely goes to Microsoft's servers because it supports sync across environments with the same Microsoft account.
Anyway, if you want to truly benefit from Windows Defender's modern features, you need to be sharing information with them. That information is later used internally to improve Windows Defender (fix bugs, add new signatures, etc.) and may even be shared with third-parties as part of the collaboration programs (e.g. Virus Information Alliance).
Vendors like Avast/AVG and numerous others also contribute to Windows Defender through the partnership programs. Microsoft help those vendors back by providing intelligence, telemetry information, documentation for various things, etc. It's a two-way street. Without vendors like Avast/AVG then Windows Defender would be missing out because Avast/AVG are very big contributors in these partnership programs, the same way that Microsoft is to them.
