I know I wouldn't be able to focus on the movie so well if wandering were implemented. Perhaps movies will be a combination of wandering scenes and fixed scenes. I feel I would be distracted as a viewer with that liberty, but who knows?
What I would like is to watch a movie completely through the eyes of one secondary character. Or maybe, the movie can be watched from differing viewpoints. That would be an incredible feeling, to tangibly see movies from the characters' perspectives.
Or, say you watch a VR movie with some friends, and each of you watch from different characters' perspectives. The discussion would be so enriched and full. The depth could finally be explored to the average viewer.
Aesthetics seem to be the biggest boundary, but the possibilities excite me as well.