Interesting view, and I half-agree with you. In an unfamiliar/professional setting it's definitely important to remain sensitive to these things, because, as you said, you don't know what people might take offense at.

I mean your friends might not have issues with all the race jokes but supposed someone from outside your group hears them? They might have a problem with it because some jokes hurt a lot more than you think to some people.

If someone from outside my friend group overhears us and finds it offensive... Then they find it offensive. They can have a problem with it all they like. However, I'd be hesitant to label it as 'casual racism'; doing so is subtly calling the speaker a racist. A person isn't a racist unless they believe that one race is inferior to the other. Unless the speaker genuinely believes that and their words reflect that belief, it shouldn't be labeled as 'racism'. Stripping away the intention and focusing on the words will always lead to misinterpretation.

For example, by focusing solely on the words, something as innocuous as "I love steak" can be interpreted as promoting animal cruelty. However, the speaker could really just be enjoying his dinner.

It's impossible to make sure that nobody is offended by what I say, so I shouldn't be branded a racist just because someone took offense. If anyone is offended at anything, they should think "Why am I offended?" and "What was that person's intention?" We shouldn't be so quick to jump the gun and call them racists.