Genuine apologies for being combative. I got a little worked up at the insinuation that I am somehow trying to dispose of ethics. I do think that you could stand to be less combative yourself; even if you think I have absolutely nothing to offer you, you must think there's some value to having this discussion if you've gone this long, and I have to choose to continue as well.

I did in fact read the Wolfram article when it was posted here a few weeks ago. I appreciated the more straightforward overview of the architecture in the link I shared without the expository information. Referring to a model like ChatGPT as a lookup-table is obfuscating, and that claim is not made by an experts, including in the Wolfram article. I would maybe permit that it's something like a randomized lookup table, in that if its randomization happened the same way every time then any sequence of inputs and outputs would result in the same eventual output. But that's ignoring both the randomization and the feedback loop of rereading of its own outputs, not to mention the presence of a neural net in the model.

Lookup tables aren't mentioned in the article I linked at all. There's plenty of description of the tokenization and encoding of input data, which is superficially similar. The presence of the neural net, the attention scheme, and the encoding of the relative position of words in an input phrase allows for interaction between all the words in a sequence, and that interaction is tempered by all the sequences in its training data.

I don't mean to skip over the fact that ChatGPT itself is not learning anymore. I agree that it's important, but I am discussing a hypothetical scenario that will become reality before long. It's perfectly capable of continuously learning from its conversations (although highly inefficiently) should OpenAI choose to do that, although there's obvious logistical reasons that they don't want a bunch of random people inputting its training data for them.

To say that LLMs are entirely Markov chains is a misapprehension. An LLM like ChatGPT is not memoryless even in its current form, because of its internal feedback loop. If you would instead argue that the state we're referring to is not the most recent statement but instead the full conversation, then I would argue a human speaker IS comparable to a Markov chain in any particular conversation. The human speaker obviously differs in that they can both update their "model" over the course of the conversation and carry those updates forward into future conversations, but the hurdles for a computer model accomplishing that are logistical, not inherent. Am I missing something there? Even Wolfram says:

What ChatGPT does in generating text is very impressive—and the results are usually very much like what we humans would produce. So does this mean ChatGPT is working like a brain? Its underlying artificial-neural-net structure was ultimately modeled on an idealization of the brain. And it seems quite likely that when we humans generate language many aspects of what’s going on are quite similar.

When it comes to training (AKA learning) the different “hardware” of the brain and of current computers (as well as, perhaps, some undeveloped algorithmic ideas) forces ChatGPT to use a strategy that’s probably rather different (and in some ways much less efficient) than the brain. And there’s something else as well: unlike even in typical algorithmic computation, ChatGPT doesn’t internally “have loops” or “recompute on data”. And that inevitably limits its computational capability—even with respect to current computers, but definitely with respect to the brain.

It’s not clear how to “fix that” and still maintain the ability to train the system with reasonable efficiency. But to do so will presumably allow a future ChatGPT to do even more “brain-like things”.

ChatGPT is not a brain, but I don't think you can reasonably claim it has nothing in common with one. And I still don't see any reason why a brain-like model could not be created.

I'll leave aside the ethical questions (no beliefs here!) since I don't think that we'll come to an agreement. I trust that you'll be kind to my dog, and that's all that really counts.