White privilege has white people as the focus because white people are the focus in the societies where white privilege matters. It's kind of poetic.
It's kind of circular reasoning, which most advocates of ideas that invoke special pleading (e.g. white privilege) are prone to commit. The idea of white privilege doesn't simply describe racially-based advantages and disadvantages -- any reasonable person can see the obvious disparities. It does more than that by assigning blame for such disparities to individuals of the advantaged group based not on individual circumstances but on group characteristics over which they had no choice or control. It's the same species of idea that asserts all white people are racist simply due to their race, and black people therefore cannot be racist. These types of ideas have very little (if any) appeal in terms of their logical or rational capacity to convince because they were never crafted on those grounds. Nevertheless, they seem to have a much broader appeal on rhetorical grounds, where convincing someone of a proposition is very much like a religious conversion experience. In short, these ideas tend to position themselves toward the unfalsifiable end of the spectrum and, as a result, become nothing more than doctrinal statements identifying the good guys from the bad guys -- you're either with us or against us.
Of course you haven't because how an ideology self-identifies versus how it applies its principles is rarely the same -- that's the nature of ideological thinking. It's like asking Fox News to be consistent with its claim to being "fair and balanced" while expecting them to compromise their core ideology for the sake of being "fair and balanced". That will never happen because what they preach and what they practice can never square with one another. Their goal has never been to convince but to win at all costs. And to suggest that the Left is in any sense less prone to this instrumentalist bullshit is pure fantasy. They engage in the exact same rhetorical slight-of-hand as do the right-wing fear mongers. It's really a shame. The Left (at least historically) has marshaled fact and reason as a basis for their social, economic, and political critiques. Nowadays, such requisites are seen as suspicious or status-quo capitulations that invariably compromise the party line. Truth is now a matter of beating the royal crap out of your opponent rather than demonstrating the persuasive merits of the better argument.
I'm neither defining nor forming an idea of what white privilege is or is not. I'm describing how that idea has been used by those who believe it to be true. It's an idea that pathologizes the behavior of a particular demographic group by categorically putting members of that group on the couch without any consideration for the diversity of behavior or beliefs among those within that group. It's more commonly known as stereotyping.