I can empathize with your hesitancy, especially if you've had negative experiences before. Honestly, I probably would have just gone into the grocery store too. If they were still there when I came out, however, I (hopefully!) would have taken a course of action, whether that's calling 911 or speaking with the kids. Societal pressures be damned, 28 degrees is damn hot here in the north, and children are especially susceptible. It is terrible that men can be labelled as "creepy" in society without context of the conversation. I've never experienced this, so hopefully someone else can weigh in.
They were gone when I came out about 20 minutes later.
I wouldn't call this a fuck-up, then. Nobody was actually hurt, and now you can reflect on the situation in case it happens again. Your concern is admirable. Too often, people are quick to say it is none of their business and move on without a second thought.