I'm a fan of outsider music. A couple of favourites are The Shaggs and Daniel Johnston. Daniel Johnston gained some recognition thanks to Kurt Cobain. And the Shaggs were called "better than the Beatles" by Frank Zappa. They were also praised by Kurt Cobain.
I've always thought that the difference between "outsider" and "insider" music has gotten progressively smaller over time. A large component of the original punk movement dealt with the democratization of music, and more recently with lackadaisical rap gaining some degree of popularity as well as production software (particularly auto-tune) becoming both easy to come by and relatively easy to use, it is now very easy to make music that is acceptable, which can become exceptional if accompanied with a powerful message or emotion. Of course "outsider" can also just refer to music that comes from a different tradition or doesn't have a direct link from prior works--something that just sounds different. Speaking of Zappa, he also worked with Captain Beefheart, who was a rather interesting musician.