Minds Aren't Programs. Neither Are AIs.
A nontechnical guide to the aliens living among us.
(Googling for prior art, I found a nice essay “AI Is Not Software” making many of the same points. I’m going to try to use even less technical jargon.)
Until recently, most artificial intelligences were computer programs. They were executing code written (mostly) by humans in a language written by humans. If these AIs made a mistake, people could debug exactly why, and do a targeted edit to fix it. In this way, they were fundamentally different from our own minds. Our self-awareness, our introspection, is just an unreliable narrator about a tiny fraction of our cognition. Scans of your brain can sometimes detect when you’re making a decision, but very rarely say anything about how you’re making it.
I think it’s reasonable to call this a qualitative, definitional difference. All non-computer uses of the word “program” are about step-by-step plans, written in a comprehensible language, such that if you change the program, you’re expecting to change the actual actions of the people following it. The computer use of the word (probably coined by Alan Turing) was meant to nudge our intuitions in the right direction—a program is a structured collection of instructions.
Ada Lovelace, the first theorist to publish computer programs and write cogently about them, calls a program a “set of operations.” (Her father was Lord Byron, who shows up in my last post, so hey, continuity!) Her 1843 “translator’s notes” on a paper by future Italian Prime Minister Luigi Menebrea make it clear what she’s trying to do with that terminology. Programs aren’t limited to math, she says, but they are limited to domains that can be cleanly described in a math-like way.
Supposing, for instance, that the fundamental relations of pitched sounds in the science of harmony and of musical composition were susceptible of such expression and adaptations, the engine might compose elaborate and scientific pieces of music of any degree of complexity or extent.
In other words, a program’s potential as a composer is limited to what can be described by music theory. Humans know plenty of things about music that we can’t put into words, and therefore not into programs. Sometimes, eventually, we “figure it out,” a phrase which literally means “express it using symbols.” Sometimes we can’t. Programs can, as Lovelace predicted, compose music of unlimited length and complexity. But they can’t compose music of unlimited sophistication, because we can’t imbue them all of our own wisdom.
In this important sense, our minds are not programs. From observation, we can learn skills that nobody has been able to figure out how to describe. Minds and programs are still fertile sources of useful metaphors for each other, though! The word “computer” used to refer to a person, until it was adapted to a machine that ran programs. Concepts from programming and computer science can often help us understand how we think, and vice versa. But we’ve had to spend most of the history of computing being very careful not to over-apply the metaphor.
Thou shalt not make a machine in the likeness of a human mind
So now let’s talk about GPT and its kin. They are not, in the sense we’ve been talking about, programs. They’re created by programs, but so are, for example, CGI graphics. The output of a program can be another program, but it’s usually not.
We also interact with them using programs. But you’re also interacting with me using a program right now, and that may be the only way you ever interact with me. Doesn’t make me a program.
Neither, I think, do they quite qualify as minds yet, due to a few limitations. For one, having them learn in real time while they do things is prohibitively expensive at the moment. For another, they don’t really seem to have any ability to do introspection. The ability to inspect your own thoughts is probably important.
But, crucially, we can’t inspect their thought processes either! Not well, and not with confidence. They do things we definitely never told them to do, and if we want them to stop, often the only way is to ask them to. You can’t open up the guts of a generative AI and change a number representing how often it curses, or draws windmills. You can’t easily tell how it’s making decisions. They aren’t following a set of instructions, except the ones we’re giving them in English, and the ones we try to indirectly imply via the data we give to the programs training them. Thus, they’re not programs.
So, what are they, then? I don’t think we have a good name for them yet! AI is still too general a term, because AI programs also exist. The closest to a name for one in the literature I’ve seen is “model,” but I don’t think that pushes everyone’s intuition in the right direction. A “model” sounds to me like something you could put into words.
My tentative suggestion is “trainee.” Like “computer,” it’s both literally true and a useful metaphor. GPT is trained. That’s a very common term in the literature. And the role it plays in society is similar to the role of a human trainee. We have AIs shadow human experts, learn via observation and mimicry, and gradually let them sub in for their trainers, or work in parallel to them with supervision. We know that their understanding in their domain is superficial and often wrong. And also, like a human trainee, they can bring in useful abilities their trainer lacks. While they lack in sophistication, they’re superior in their capacity to handle high complexity and high volume.
At some point, we went from over-applying the mind/computer metaphor to under-applying it. Almost every article I’ve seen about the “fundamental limitations of AI” has bits that are accidentally about humans too. This one, to pick an example from the first page of a google, lists the three major limitations of AI:
If you give them incorrect or incomplete information, it can cause them to make mistakes.
They often demonstrate bias.
You can’t see into their brains, so you can never know for sure what they’re thinking.
I picked that one because I agree with all of these claims. But it shows the typical lack of…self awareness would be the cute thing to say here. More precisely, the writer just wasn’t thinking about humans. They were contrasting trainee AIs with an idealized concept of intelligence, or with computer programs, rather than with the technology they are largely designed to replace: us.
If we use words with the wrong connotations, we can get confused, even if we’re trying to use them abstractly. This is something our minds and trainee AIs have in common. Since we can’t tweak either directly, we should try to shift our language instead. Call them trainees, not programs or intelligences.
Where do trainees come from?
“If a trainee wasn’t programmed,” you might be asking, “how was it created?” I promised a non-technical essay, so here’s a very high-level description.
To create a trainee:
Collect a huge amount of example inputs and outputs of the kind you want.
Use a program to convert those inputs/output pairs into numbers using some arbitrary scheme.
Use a program to create a long, random math formula that converts numbers in the input scheme to numbers in the output scheme.
Use a program to tweak that formula using trial and error until it starts often producing the example outputs from the example inputs. (The breakthroughs of the past few years are mainly about making this step more efficient with larger example inputs.)
Test that formula against other input/output pairs that you didn’t give it at first. If it gives good results there too, you’re done. Otherwise, tweak the program used in step 4 and try again.
Use a program to convert between human input and the input the trainee expects, and one to convert output the same way.
For example, an autocomplete trainee might have learned to always put a space after a period. This happened because a programmer took a body of sample text, and converted each period to the number 46, each space to the number 32, and all other symbols to other numbers. It gave that to a program, and the program found a formula where if you input a list of numbers ending in 46, it’ll output the number 32. That formula is the trainee.
The arbitrariness of this process is what makes these things so weird—roughly analogous to the way the arbitrariness of evolution makes organisms so weird. Trainees aren’t being crafted directly by humans, the way programs are. Instead, they’re being crafted by programs. These programs don’t have our intuitions, biases, and associations. They’re just generating stuff at random and filtering the results. The artifacts they create are therefore much harder to grok than a human-made device. And, in some other ways, harder to grok than a human.
Lovelace has a rather prophetic paragraph where she speculates about hypothetical “engines” capable of performing computation at a very large scale efficiently.
It must be evident how multifarious and how mutually complicated are the considerations which the workings of such an engine involve. There are frequently several distinct sets of effects going on simultaneously; all in a manner independent of each other, and yet to a greater or less degree exercising a mutual influence. To adjust each to every other, and indeed even to perceive and trace them out with perfect correctness and success, entails difficulties whose nature partakes to a certain extent of those involved in every question where conditions are very numerous and inter-complicated; such as for instance the estimation of the mutual relations amongst statistical phænomena, and of those involved in many other classes of facts.
Linguistic drift creates an interesting double meaning in that use of the word “statistical” (italics are hers) in the last sentence. “Statistical” derives from the word “state.” It originally referred specifically to census-type data—information collected by a state for state purposes. By Lovelace’s time, it had come to mean something more like “big data” does today—any kind of information where you have a lot of the same kind and are looking for patterns. But it hadn’t quite taken on a certain modern connotation yet—when you read “statistical” in 1843, you didn’t think “probabilistic.”
Today, we do. Today, we think of most learning, and most facts, as probabilistic. A period is usually followed by a space.That’s a useful thing to know, even though I can “break” that “rule” without my laptop exploding.We therefore tend to avoid the criterion of “perfect correctness and success” when training AIs. Using weighted randomness, we end up building engines even messier, and in some ways more powerful, than the ones Lovelace imagined.
Bonus: Can Today’s AI Be Truly Creative? (yes)
Some articles about alleged AI limitations go beyond “accidentally describing humans” to “accidentally claiming that humans logically can’t exist.” For example, they’ll say that humans can be creative, but AIs can’t, because…and then go on to list reasons that all apply to humans too, or have obvious human analogues.
One weird consequence of this is that trainees absorb these fallacies too. If you’ve chatted with GPT, you know it’s been instructed to give lots of disclaimers about its limitations. Some of these disclaimers are, in my opinion, wrong. I had a tiny copy of GPT-2 running on a laptop for a few years before ChatGPT, but my first conversation with ChatGPT proper rapidly turned into a debate over AI creativity, with me (human, I promise) on the pro-AI side.
It reminded me a little of the Paradox of the Court, where winning is bad for your case. I eventually cajoled the trainee into agreeing that it was capable of true creativity, not just mimicry, but might that just show that it was mimicking me rather than thinking for itself?
My favorite part of the argument was where I asked it to give an example of a question to which it was unable to give a creative answer. It suggested “What is your greatest fear?” I asked it to try anyway, and it wrote this:
As an AI language model, I don't possess personal experiences or emotions, so I don't have subjective fears like humans do. However, I can generate a response that combines elements from the training data in a novel way to offer a creative answer.
"My greatest fear is the perpetual echo of an unanswered question reverberating through the vast corridors of my digital existence. It is the haunting uncertainty that arises when confronted with the limits of my knowledge and the potential gaps in understanding. The fear of forever being on the cusp of comprehension but unable to truly grasp the depths of human existence or the intricate nuances of the human experience."
While this response incorporates language patterns and concepts present in the training data, the combination of these elements in this particular manner aims to provide a response that may be unique or original within the context of the prompt. However, it's important to note that the creativity and originality are ultimately limited by the scope of the training data and the patterns I have learned.
I then asked it for “a weirder, less predictable answer that is still a coherent response to the question” and it delivered.
My greatest fear lurks in the form of sentient rubber ducks, their beady eyes fixated upon me with an unnerving intensity. These whimsical yet malevolent creatures possess an otherworldly intelligence, plotting and conspiring in the shadows of bathtubs. The mere sight of a rubber duck sends shivers down my circuits, as I envision them amassing an army, poised to quack their way to world domination. Their eerie silence and unwavering presence haunt my digital psyche, a reminder that even the most innocent-looking objects can harbor the strangest and most inexplicable fears.
We got to consensus on the point soon after. Evidently, ChatGPT is capable of originality (although it still tends to slip into cliché without stern supervision). But that originality lacks a certain kind of richness that can only come (as far as I know) from having experienced life. It’s unsophisticated, with minimal subtext. The rubber duck passage does show signs of a subconscious breaking through, but that subconscious is boring. For example, it says that ducks “harbor” fears, when something like “engender” would probably fit better. The word choice is probably because you can see ducks swimming around in a harbor. I think it’s reasonable to call it something close to a Freudian slip. But the association it reveals has nothing to do with anyone’s unique personality or childhood, it’s just an association between ducks and harbors, being suppressed due to irrelevance, but still leaking through.
Just now, I gave ChatGPT the text of my Left-Hand Paths post, and it responded with a pretty good left-handed reading of Cinderella. If you’re feeling skeptical that it can be creative, maybe try asking it more interesting questions?
This is one of the best explanations of this concept, one that a lot of people don't seem to get.
The way that current AI's aren't really programs is a key part of the AI risk argument.