Is this trying to make the argument that expecting women to not wear yoga pants in a professional environment is wrong?

No, this is trying to make an argument that you can't let what other people are wearing be an excuse for how you treat them. There's a HUGE difference between thinking something and being seemingly unable to control your reaction to your hormones. Yes, as you suggest, this does apply to all sides, and there are people of all genders who get handsy when they see something they like and it's ALWAYS inappropriate, no matter who is doing it.

The problem is that there is a deeply-ingrained culture of acceptance in cis-male and socially conservative groups where, because feminine figures are seen as "inferior" somehow (often not even conciously, but sometimes it's encouraged), masculine figures are allowed to treat them as they please, and do so with impunity. This is not to say that all of the members of the aforementioned groups are like this, but there are enough of them that the problem needs to be pointed out and addressed.

This culture of impunity and superiority is the problem, and I think a lot of it comes from how we culturally view people. People who are attractive must not be intelligent, and must be relying on their good looks to get ahead in life (the opposite also being true of looking down on "overly intelligent" people). As a result we must be superior than them and look down on them, seeing them as deserving of whatever misfortune they come upon. This applies to the "Never misses Leg Day, Do-You-Even-Lift" muscular men as much as it applies to the "My bag matches my Dress which matches my Shoes" pink-loving Barbie Girl, or to the "I seem like an asshole but it's only because I can't figure out how to simplify I know enough for you to understand" Intelligent person (both obviously caricatures for the sake of example). Basically we are Crabs in a Bucket, pulling down anyone who attempts to rise above "Normal" in any way.

We're a really fucked up species.