Chat-based Large Language Models replicate the mechanisms of a psychic's con

The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.

> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think

LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.

Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.

Man I remember back when a psychic conned me by solving the Navier-Stokes problem.

Also >July 4th, 2023

Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.

These questions will never go anywhere because there is no good definition of intelligence or consciousness.

> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this

Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.

Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.

It does seem like LLMs share the language of psychics.

The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.

This is a strange newsletter post. It’s harping against AI but seems AI-written itself.

Its ultimate conclusion:

“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.

I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”

Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35

> The intelligence illusion is in the mind of the user and not in the LLM itself.

> Many AI critics, including myself, are firmly in the second camp.

How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?

Is the solution also illusory?

Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.

But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.

> The intelligence illusion is in the mind of the user and not in the LLM itself.

I think that might be the solution to consciousness illusion.

The consciousness might be purely in the mind of someone that believes some other entity to be conscious.

People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.

All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.