MSN Crawling RSS Feeds for Breaking News

MSN announced on its blog that they are crawling certain feeds in order to index breaking news as quickly as possible.

One of the key elements of having a great index is having fresh content.  If someone searches for Vista on the day that we announced the name for the next version of Windows then they should be able to see relevant results. 

This means that we need to have the content in our index within minutes of these types of events happening. In order to do this we built a simple crawler to get News and Blogs that break this type of news.