Security wise? They can be infected too. Although I haven't experienced running an infected one that can bleed infection into the system. In case they appear to be infected, I just delete everything and replace it with a zipped backup.
Performance? Running an app exclusively from a folder from an HDD is faster than one running from a USB drive. Although you can install a portable app on your HDD, the main purpose of portability is that you can carry it with you always.
Here's what on my USB drive