There are plenty of mainstream casual games that are well made, from angry birds to bejewelled to plants vs zombies to the sims to Mario right the way up to FIFA and CoD.
However there's a certain type of casual game, the result of a focus group coming up with the most 'friendly' least inspiring art style imaginable, combined with woeful production values and gimmicky controls, that aren't good for the industry. Nintendo went there and are back pedalling, Sony dipped their toe in the water but couldn't quite pull it off, meanwhile Microsoft look to be pursuing it full force, alongside their secondary goal of turning the 360 into an advert delivery platform.
These games are pulling funding and talent away from more worthy titles, and they pull casual players in short term while alienating them and putting them off when they realise how bad the games are. People who aren't heavy or experienced gamers aren't automatically idiots, and it would be much better for the industry as a whole to draw them in with better quality content so they might stick around for longer and not buy one or two rubbish casual games then leave their console gathering dust in the corner. Mobile games are doing it so much better, with easy to pick up non threatening games that are actually fun to play at the same time.