But why does this happen as after removal, won't the restrictions placed by the antivirus be removed?
I think you're thinking of quarantine as a malware jail, but it's not.
Once a file is detected as malicious and "moved to quarantine", what really happens is that the antivirus deletes the file from its original location and then modifies it (call it permissions, strings) in order for it NOT to be able to run as a program/executable. This modified file is moved to a hidden folder aforementioned by
@roger_m that nobody can access. If you then choose to empty quarantine, this folder and its content is deleted for good.
If upon uninstall an antivirus doesn't let you choose what to do with quarantine objects (usually it should), then probably this folder will remain hidden full of modified, harmless objects.