Displacement is neither inevitable nor necessary

I don't know about that. If you take displacement as an inevitable side effect of migration (empirically) and migration as a necessary human function...

and the benefits of displacement are received by the displacers, not the displaced.

Not always, not by any means. Short term/long term, also.

The Luddites were right about their own situation: they did experience hardship as a direct result of new technologies. Yes, society eventually moved on and labor requirements were reduced in that specific sector, but a lot of people suffered in the meantime. In the case of the Luddites whose labor was displaced by machines, it may have been better to reduce working hours first instead of abruptly employing fewer people.

Gradual changes are mostly better than instant ones. But we don't always get a choice.

I have a lot of big thoughts about the changing nature of structural unemployment (and structural displacement, as is the case here) which can be summarized thus: in the 21st century, it's not going away; in fact it will intensify, and that intensification is caused by factors that in the long run will be very, very good for us. In the short term, people get hurt, because that's what structural changes do. The locals in San Francisco are getting hurt.

One solution is careful government intervention to mitigate (but not stop!) the effects of the changing structure. In this case, that's zoning laws, housing controls, wages and so on in San Francisco. I'm not against that by any means, as long as it seeks rather to smooth the trend, not reverse it.