Self publishing has the potential to obliterate non fiction. Without a publishing house to fund actual research that takes months or years to complete I don't see a lot of people producing quality research based text. You make a good point about good books often not selling well but shouldering promotion and editing while also presumably working so you don't starve is unrealistic. The publishers are a necessary evil and so are record labels and movie studios. Strip out all the protections afforded by a big guy with money in your corner and you get 50 Shades of Grey and the YouPorn of erotica kleinbl00 mentioned. It's nice to have the gatekeepers weakened so a potential talent isn't overlooked but tossing them out completely is at the bare minimum foolish. Since I mentioned record labels, where are all the Internet musicians making it big with direct distribution and YouTube promotion? I can think of one and his name is Justin Bieber.
The non-fiction is an angle I'd never considered. Though as a response that I have no reasonable way of checking, don't university presses make a lot of the non-fiction for the academy? As I said, probably wrong, a legitimate question. And the difference I think that we're thinking of is scale of success. You mention Bieber as the scale of success that you are familiar with, but underground success can make you a living in the music industry, and I think it could be able to with publishing if authors weren't being paid like slaves. The without publishers people would still write, and without record labels the music industry survives because art isn't a commercial product that scales. The Odyssey was written before publishing.
Mozart wrote before record labels. The difference is the definitions of success...megahits versus supporting yourself. Consumerism in contrast to satisfaction.
I worry about Amazon because they're the #1 retailer of books in the world. They're so powerful they can push around traditional publishers on issues of price and those publishers are huge companies backed by other huge companies. Anyone with a reasonable working knowledge of antitrust laws couldn't argue that they aren't abusing their market share in monopolistic ways. This self publishing trend is nice but it produces abject shit for the most part. Publishers take incredible risks on books that go nowhere. Go to Goodwill or the remainder pile in Barnes and Noble to see how much money is spent on books no one reads. Traditional publishing wants to sell your book more than the record companies want to sign your band, your book just sucks. No one, not even publishers, get into the book business to get rich but Amazon by virtue of its market share can make publishing much worse if it ends up assuming end to end control of publishing which seems to be within their goals as a company.
Fair enough. But the phenominal thing about monopolies and large companies is that they are very difficult to keep in the position of market leaders. They eventually get out innovated by the market. Even when Microsoft was at its worst and either buying or crushing everything that opposed it in the 90s a company came back and beat them. I mean now Apples is getting pantsed, but beside the point. The problem isn't that there is something wrong with publishers putting money out marketing and printing books. But with digital distribution, the issue at stake here, what is the problem with a funding model that rewards better click rates? Isn't that the application of the internet funding model on books. We're all okay with bloggers getting paid that way, newspaper and magazine writers getting paid that way, but we've got a problem with self-published novelists? I know Amazon won't last because the strategy is flawed.
Bieber didn't really make it big until he got picked up. There's always this discussion of "gatekeepers" as if their job is to keep you out. It isn't. Their job is to find the stuff they think is valuable and then exploit it for a percentage. An agent who gets ten great books out of ten submissions is going to push all ten towards publication so that she can get ten points off of all ten. Unfortunately agents generally get zero out of ten so they have to say "sorry, no thanks, best of luck" as politely as they can. Trust me - the publishing industry has the nicest "gatekeepers" I've met.