It's a worthwhile discussion and one that we, as a culture, need to learn how to have. Unfortunately "you don't agree with my argument" doesn't mean "you don't understand my argument" and most of the discussions around the phrase "white privilege" conflate the two.

I know I certainly thought I understood prejudice as a "disadvantaged" ethnicity. It wasn't until spending time amongst African Americans that I realized that while I might have a better idea than most middle class white men my understanding was woefully incomplete. Unfortunately the whole "white privilege" discussion is usually some form of "you lack the moral authority to have an opinion" rather than "here's a perspective you haven't considered" and the Internet isn't helping that. Even perspectives like the one linked above tend to deprecate any individual experience by demonstrating that on average, white people have it better.

End result? Whoever deploys the argument is saying "I hereby nullify your individual experience through averaged demographics."