That might be true if voters had the greatest influence on how congressmen vote. They don't. Lobbyists, party leaders, and other vote buying and intimidating groups do. If you don't know how people vote you can't buy a vote or respond harshly to a vote that wasn't the way you wanted it. Individual congressmen lose any incentive to appeal to lobbyists because there's no money in lobbying if you can't be sure you're buying votes.

Vote buying, in the form of lobbying, and intimidation by both party leaders and organizations are the two largest confounding factors in our democracy. Money buys votes, not constituent support.

Take a look at the graphs on page 10 of this study. You'll note that the policy preference of average citizens has almost no significance to the political outcomes, while the preferences of the economic elite and special interest groups are significantly impactful.

If voting for representatives we like were a remotely reliable method of ensuring policy that we like we ought to see a much more significant impact. Maybe not as significant as special interests and the economic elite, but it should at least move the line. Instead the line is flat, meaning that things the average citizen hates are as likely to pass as things the average citizen loves.

Meanwhile special interests and the wealthy dominate the political landscape. They do this because they're able to pay politicians who support them and run attack ads on those who don't. This wasn't always the case.

Before the 70s lobbyists were a joke. They were broke and ineffectual. Then the Legislative Reorganization Act of 1970 was passed.

On page 6 you'll see the following

PUBLIC ANNOUNCEMENT OF COMMITTEE VOTES SEC. 104. (b) Clause 27(b) of Rule XI of the Rules of the House of Representatives is amended by adding at the end thereof the following: "The result of each rollcall vote in any meeting of any committee shall be made available by that committee for inspection by the public at reasonable times in the offices of that committee. Information so avail- able for public inspection shall include a description of the amendment, motion, order, or other proposition and the name of each Member voting for and each Member voting against such amendment, motion, order, or proposition, and whether by proxy or in person, and the names of those Members present but not voting. With respect to each record vote by any committee on each motion to report any bill or resolution of a public character, the total number of votes cast for, and the total number of votes cast againstj the reporting of such bill or resolution shall be included in the committee report."

The result of this is lobbying as it is today. This allowed for it, because now you knew what you were buying. It also allowed for intimidation, because you know who to target. Strip it away and you take away the very thing that makes our democracy less representative than it once was.

James D'Angelo explains this at length if you've got the time. He's also got a couple shorter videos addressing the topic.