To pick up on a very old thread... I've been reading David Abram's The Spell of the Sensuous lately. He has a chapter in there about how in oral cultures stories are very closely bound up with particular places, so that it makes no sense to tell a story without saying where it happened. The powers of the place are actively involved in the story. Only in cultures with alphabetical writing, he says (roughly), do we find being treated as a neutral setting (space) for action. I wonder if part of the issue isn't a prejudice against science fiction, but that we naturally need to understand stories in terms of relatable places - that we relate better to the story when the setting is relatable and naturally absorb it more deeply? Presumably most of us don't have much familiarity with spaceships and airlocks.
It sounds to me that the key is "oral cultures." When your body of knowledge exists as folklore shared amongst those you only know face-to-face, speculation is the same thing as lying. I'm unaware of any non-writing culture that thought hypothetically and thinking hypothetically is the crux of science fiction.