That's one of the things Steven Hawking has been known to say: the lack of time travelers among us suggests there is no way to time travel, at least not into the past. Apparently, what he said was "I gave a party for time-travellers, but I didn't send out the invitations until after the party. I sat there a long time, but no one came."
Maybe time travelers don't think we are interesting.
I think the problem with time travel is that is you went back in to to kill say Hitler.
Killing him would negate your reason for going back in time and cancel your trip. Accidental changes in history might not cancel trips but also might not be noticed for the same reason. History might at least seem immutable and hence the Travelers only have reason to observe not interact.