Think of it this way.
Imagine someone wants to break into your house: To do so, firstly, they need to actually know where your house -is-. If they don't know the location of your house, short for breaking into -every- house (which, in this context, would be the equivalent of just trying to just hack into multiple networks or computers systematically by their addresses or similar, as opposed to just targeting one, singular system), they won't be able to break into it, or at least, if they break into some random house somewhere, the chances of that house actually being yours, is rather slim. The address of your house, in this context, is your IP address: The address of your internet connection.
Now, if they have your IP address, they now know -where- to hack. However, that in itself, although a significant and required step of specifically targeting your machine, does not necessarily mean that they'll automatically just be able to hack into your machine and access your files.
If, using this metaphor, they want to break into your house, they still need a plan of attack: They could steal your front door key and just walk right in when you're not home (the equivalent, in this context, would be if you have, for example, some sort of remote access software on your machine which is password protected, and they somehow manage to steal your password, thus allowing them remote access to your machine), they could walk around the house to try find an open door or window that you forgot to close or lock (the equivalent of those port scans you mentioned), and from there, exploit that open door or open window to walk right in when you're not home, or something similar (you get the idea).
Take in mind, too, that having an open port, in itself, doesn't always represent some vector of attack that could be used: The target machine needs to actually possess some software or server that listens to that port, which could be used to commit an attack (RATS, or remote access trojans, are a type of malware used to do this). Case in point: Websites and webservers, by their very nature, (usually) require port 80 to be open and listened to by the server software, in order to receive and respond to requests from users to access the website. If a website or webserver closed this port, suddenly, the website would become unavailable.
There's nothing wrong with this port being open in those circumstances. Where this becomes a problem, however, is when there's some software which you won't want outsiders (ie, hackers) gaining access to, which isn't secured, and is listening on an open, unsecured port: In this situation, you've got an exploit on yours hands, which a hacker could use to attack your system.
As already suggested, if you've got a good firewall, and if you close off those ports that aren't required, you should be safe.
Of course, it's generally not a good idea to hand over your IP address to someone that you know to be a hacker, but overall, I wouldn't be overly paranoid about websites you visit having records of your IP address or anything like that.