Glad to see Bender cite Wittgenstein, because he already settled this debate 100 years ago while fighting against the descriptivists whose basic argument was that you can understand anything if you describe it in sufficient detail, and that our lack of understanding of any topic derives from our not having enough facts about the subject (as an aside I see a very strong analogy to the descriptivists and those who think AI is "alive"). Wittgenstein made a lot of good counterarguments, but the one that turned out to be most famous and most illustrative was where he said that the way we learn what the color red is is by someone else pointing to a red object and saying "this is red." He was trying to argue that meaning derives from agreement about meaning and nothing more.

We use language, at least in part, to give others access to our inner thoughts, and without agreement on words' meanings, then others completely lack access to our inner selves. Where I think that the Mannings of the world completely lose me is in statements of this sort:

he allowed, humans do express emotions with their faces and communicate through things like head tilts, but the added information is “marginal.”

Um, what? What if I said in response, "Fuck you." It would probably taken as being pretty hostile, right? But I can think of a at least three other use cases that lack any hostility. Say, gentle ribbing; an expression of disbelief; and an expression of desire. One could think of almost any phrase, but especially any idiomatic phrase, for which there are many, many meanings, and the only way to disambiguate the meaning is by the "marginal" information.

I think this point has been made explicit in the age of text-based communication. How easy is it to interpret an email or a statement on social media as crass or aggressive, when the author was trying to be lighthearted? There's a famous experiment in neuroscience where you apply a force to a transducer, and that exact force is then applied to another person sitting across from you by another transducer. Then the person to whom the force was applied is asked to replicate exactly what they felt, which is in turn felt by person 1, who does the same, and so on. It spirals. Every time. Because we're hard wired to perceive things to be worse than they are when they're done to us and we respond in kind. The marginal information is the only thing that keeps us from not killing each other.

I literally can't even wrap my head around someone believing that non-verbal context around verbal information doesn't just inform but in many cases defines the meaning of the verbal information. It's like not agreeing on the color red. What kind of world will we live in where we can't agree on the meaning of words, because the speaker (i.e. the robot) doesn't have an inner self to try to give access to?

Edit: Just to add, I've recently begun using ChatGPT. I like it. It does some wonderful things. It helped me to design an experiment by cutting my overall search time from probably 5-10 hours on PubMed and Google to like 15-20 minutes of chat. I'm not against GPT specifically or AI generally. I'm against calling it conscious or anything other than really powerful computer code. It's the moral equivalent of MS Word to me, which also boosts my productivity immensely relative to pen and paper.