As a trans woman, I can make two points:

You may actually in fact have met many trans people. Trans men tend to "pass" very easily, and trans women who transition at a good age generally also "pass" quite well. I started hormone therapy in July 2012 and now work in retail, nobody knows. A co-worker actually made a comment the other day in a conversation regarding LGBT matters that they've "never met a transsexual." Him and I are decent acquaintances. Whoops. What a lot of people don't realize is that being trans isn't necessarily an identity to a lot of trans individuals. Frankly the use of the phrase "being trans" is a bit of a misappropriation to me. To me and many other trans individuals, "being trans" is having a medical condition. We don't say refer to someone with cancer as "being unregulated in their cell growth"

This thus leads into my second point, most people don't discuss their medical conditions very often. Homosexual individuals are quite prominent because signalling your sexual orientation is a rather obvious, natural tactic for being able to find a partner. If an individual's physical presentation doesn't match the common stereotypes for how a transsexual looks, you won't be able to know they're trans unless they specifically disclose this medical information (or, as is all too common, someone else does). Many transsexuals don't see being transsexual as some identity they need to disclose to people.

What transsexuals share with the LGB community is an underlying fight against human tendency to place prejudice on those who are different. It is unfortunate for transsexual individuals that they happen to have a condition which ties into very strong cultural and societal ideas. We know there are many conditions that lead to a human not having entirely congruent sexual dimorphism. Some of these can be quite minor such as hormonal problems during adolescence or in mature adults (however these problems can be severe, too) and some of these can be quite major such as hermaphroditism. It's commonly accepted that hermaphroditism is a medical intersex condition that one had any control over or any identity with, and yet it is something quite shunned and parodied in society. Transsexuals face a similar battlefront in society, and yet they have an extra disadvantage in that we don't even have an external or internal consensus on whether it's a medical condition or "identity."