There is a magical sweet spot of maximum efficiency for caches, as for most things. If you set your browser to delete your cache every time you quit, then closing your browser will be a little slower. And since building the cache took some time too, and then it's thrown away, it's just a waste of time. But if you leave it there longer, then when you reopen your browser, it will have less work, less downloading to do, and will be able to use cached components to speed things up. The sweet spot then starts becoming sour again when you have too much in your cache, because your browser will simply have too much data to process locally, which will end up taking more time than it saves; and then is the time to delete your cache to regain performance.
So, for example, doing it manually every day instead of every time you close the browser can be a good setting. Maybe weekly is better depending on how you browse. Or you can set a hard limit of a few megabytes and never worry about it again.