That seems like a Penguin attack...Make sure you aren’t trading or spamming links, remove all unnatural links to your site, have variation in anchor text and get rid of any over-optimized onpage element.
Just in case it isn’t a Penguin affair, Here’s Matt Cutts helping you with other possible reasons:
http://www.youtube.com/watch?feature...&v=gx04qNrCbnk