I think you've forgotten how aggressive Letterman was when he started out. The censors were up in arroars. Also, keep in mind: Colbert was a staff writer on SNL. The Colbert Report was a foil for the Daily Show and Colbert was definitely playing a role, but prior to that role he showed some impressively diverse chops.
Don't get me wrong, I think Letterman is great. I think a lot of what he did was lost on a younger audience, but it was still respectable even to those people. I think CBS now though, as opposed to when Letterman took the job, will try to play it safe. Most of what they do now is playing it safe in the last five or so years, with Craig Ferguson as an exception, but that's a very late time slot. It will be interesting to see how it goes, and who knows, maybe they've give him some free reins to bring in a younger audience.
Craig Ferguson's late show is easily my favorite on TV and has been for a while now. It's a shame it's on so late and I'm either asleep or doing other things when his show is on.