Actually also forgot to mention that although formerly pretty much limited to the Search Engines, a bunch of private concerns use crawler (or spiders) to do Predictive Analysis. For instance, an Investment Banker like Goldman Sachs will scan various websites to quantify the amount of interest in a certain stock by determining the amount of chatter showing up on these sites.