From an interview to air this Sunday on This Week with George Stephanopoulos:

STEPHANOPOULOS: "Does it frustrate you when your own supporters see racism when you don’t think it exists."

OBAMA: "Look I think that race is such a volatile issue in this society, always has been that, uh, it becomes hard for people to separate out race being a sort of part of the backdrop of American society versus race being a predominant factor in any given debate. And what I’ve said, when we talked during the campaign, are there some people who don’t like me because of my race? I’m sure there are. Are there some people who voted for me only because of my race? There are probably some of those too. The overwhelming part of the American population, I think, is right now following this debate and they are trying to figure out, is this gonna help me. Is health care going to make me better off? Now there are some who are, setting aside the issue of race, actually I think are more passionate about the idea of whether government can do anything right. And I think that that’s probably the biggest driver of some of the vitriol."

(bold emphasis mine)

The entire interview can be viewed here:
http://blogs.abcnews.com/george/2009/09/obama-tk-interview.html

The emphasis is there to highlight that portion which I think is where most of the heated debate and extremism comes from. I think the President hit the nail on the head. (entire paragraph, not just that statement)