I agree and that is what I have done. I read an article a while back that the way to go is a TV plus set top box. I have 3 TV's with Roku and WDTV.
Another advantage, from the aforementioned article, is that a set top box is faster than a smart TV. I have no experience with a smart TV, but if a connected BluRay player is any indication, you can keep your smart TV's. My BluRay player, which I never use, is horribly slow.