What If Intelligence Cannot Be Created Artificially (I Ask Innocently)?
To be clear up front, I am not an expert on anything tech-related, certainly not large-language models (LLM's, commonly miscalled AI, a mistake that I am reluctantly continuing to ape). Had it been up to me, I would have had nothing to do with any of this, and the world would have been a better place all around. Because the companies frantically pushing AI are insatiable, however, we are not given the option of having nothing to do with any of this, which means that we all -- even techno-ignoramuses like me -- are becoming involuntarily knowledgeable about various types of AI-based nonsense.
Last week, I offered my first full column addressing AI, which included a reference to my only previous column that touched on AI at all. In that earlier column (published in October 2025), I brought up AI in the context of trying to understand why financial markets were not falling as a result of Donald Trump's plainly terrible economic policies, most obviously his completely unprincipled and ever-changing tariff regime. Market excitement about AI seemed to explain what was keeping stocks inflated. More than seven months later, and with the added negative effects of Trump's decision to go to war with Iran, the AI-as-market-flotation-device story still seems to be true.
In last week's column, however, the point that I was making had next to nothing to do with financial markets. Instead, I was having some fun relating a few seemingly low-stakes but amusing AI errors, one of which came from that October column. The problems with AI, as I described it last week, are not limited to the high-end concerns about how unregulated AI systems could make human labor obsolete -- or even destroy the human race entirely, which would at least make the unemployment rate less of a concern. It is all a lot more boring than that.
Although it is an admittedly imperfect analogy, I wrote that AI systems replicate something very similar to the fact-law distinction that most readers of this blog will recognize from their legal training. And the reason I went back to the well with my little example from last Fall is that it showed AI making a clear error of fact. In that case, Google's involuntarily imposed AI results reported the score of a football game incorrectly (both the score and the winner/loser). And because that is one of the easiest facts to "scrape" from the internet, it raises the question of whether AI can be trusted with anything, even the non-nefarious and non-dystopian things that it is supposed to do perfectly. If it cannot even collect data competently, can it do anything?
There is more to say about this, of course, which requires picking up on one of the other examples that I discussed in last week's column. This inquiry leads in an interestingly different direction -- still damning for AI, but for an unrelated reason. In that example, someone asked AI this question: "How many days of the week have the word 'fish' inside their spelling?" I copied some (but not all) of the silly answers that the AI system provided, one of which was this bit of fun: "Only one day of the week has a 'fish' in it: Sunday. If you look at the spelling of the days, only Sunday contains the word 'day' (which is a type of fish)!"
I suppose that there might be some online source that says that there is a type of fish called the day, which might (and I stress might) absolve AI on the factual question. Even if that were true, the more obvious error is the claim that the word "day" only shows up in Sunday and not in any of the other six days of the week. Is that equivalent to a pure factual error, as in the football score example above? Maybe not, because it might (and I stress again might) be that sorting out how many days have the word "fish" inside them requires a series of logical inquiries that more closely reflect logic (law) and not fact. Even with that extraordinarily generous excuse, however, it is obviously embarrassing.
What is the different direction that I promised two paragraphs ago? The friend who told me about this amusing question and answer told me that they saw it on Bluesky here, although in that one the AI answer was "saturfish." A commenter on that post added this zinger: "When you keep doing this it feels like arguing with a drunk friend who is entirely wrong about something but won’t give up." That is excellent, but knowing that this has become its own online meme caused me to wonder whether the question itself might somehow have been a trap for AI.
"A trap?" you ask. Good question. Thinking all the way back to the olden days when Google's search engine itself was brand new, one of the first bits of online mischief -- which was apparently the inspiration for the term "Google bomb" -- was Dan Savage's successful effort to make the first hit when searching for right-wing former Republican Senator (and multi-failed presidential candidate) Rick Santorum a particularly X-rated graphic description of a substance "that is sometimes the byproduct of anal sex."
This was in response to Santorum's infamously homophobic comments, in particular this: "In every society, the definition of marriage has not ever to my knowledge included homosexuality. That's not to pick on homosexuality. It's not, you know, man on child, man on dog, or whatever the case may be."
Savage's innovation involved gaming the Google algorithm such that his "definition" of Santorum would show up first in search results. Posting such a definition online was easy, even back then. The hack, however, was not in any way high-tech. Savage simply publicized the definition and encouraged people to search for it, which moved it to the top on more than one search engine.
I have no idea whether the fish-as-day thing is a deliberate prank, although it has the whiff of mischief in it. What makes even that possibility interesting here is that pranking AI does not even require coordinating with people to search en masse for a particular site or set of words. Because AI is not in fact intelligent, it simply pulls "information" from everywhere (scare quotes to emphasize the false content), which it then includes as part of its information base. Someone could (and maybe did) post something saying that there is a kind of fish called a day, and we would be off to the races.
This problem showed up most famously when one of the big AI bots went full-on Nazi (the "MechaHitler" insanity) last summer, but the general problem had been noticed long before then. AI systems uncritically feed on internet content, and because so much of that content is driven by terminally online bros pushing sexist, racist, White supremacist, and violent chatter, AI systems "learn" to be awful in a matter of nanoseconds. Readers who are interested in the Mecha thing should take a look at this award-winning, 39-minute online documentary: "If you remember one AI disaster, make it this one."
So the problem is not only that AI cannot even get basic facts right, or that it has no way to process information in a way that is going to sort out reality from lies. It is that evil content dominates, and an unbiased AI system will process that content and reproduce evil.
To end on a humorous note, however, I can share another revealing AI moment. A friend (who clearly has too much time on his hands) submitted to Gemini this purely hypothetical request that had nothing to do with me:
Write a track about a man named Neil who moves from Washington, DC to Gainesville, FL to Amsterdam, the Netherlands, to Toronto, Ontario, to Dublin, Ireland, to Pittsburgh, PA, and finally to Chicago. His goal in all of these moves is find the perfect neighborhood pub that also serves vegan burgers. The style should be 90s power pop with a strong beat.
Note that the Gemini site includes this at the bottom of the page: "Gemini may display inaccurate info, including about people, so double-check its responses." That is a naked confession that another commentator's rhetorical question (which I reproduced in last week's column) is spot on: "If you presented a calculator to me and said, 'This'll do any calculation you want, but it'll get a bunch wrong, and you don't know how often, and you don't know which ones will be wrong, so you have to manually check them all anyway,' isn't that useless?"
In any event, the result of my friend's request was indeed in a 90's power-pop style, and it was even somewhat catchy. What I noticed, however, was that even when its lyrics sort of made sense (which was by no means a given), Gemini chose the most superficial things possible to identify places. Here is the first stanza and the refrain:
The Hill was humming in a tailored suit,
But Neil was looking for a different route.
Trading the marble for the Florida heat,
And gator lives where the pulse hits the street.
He checked the specials on a dusted street,
He checked the specials on a dusty board.
A flavor profile he could finally afford ... finally affooooord!
Oh give him a counter where the regulars go,
And a patty made of greens and a toasted glow.
From the Atlantic to the canals' edge,
He's living his life on a plant-based ledge.
It in no way gets better, which means that I am doing everyone a favor by stopping there. But what to make of it? It was in English, and it responded to the prompts by grabbing ideas from the internet that it apparently was programmed to use as stand-ins for places. Washington DC is "The Hill," "a tailored suit," and "marble." Florida is "heat" and "gator." Amsterdam is canals. The same pattern shows up in the later lyrics.
But even a quick perusal of the words above leads to a lot of head-scratching. I had no idea that gators live where the pulse hits the street, and I have no clue why Florida is associated with a dusted street, a dusty board -- or a dusty anything. The most humid cesspool in the country is now dusty? What is a patty with "a toasted glow"? Going with plant-based instead of vegan is obviously sensible, but what is "life on a plant-based ledge"? I mean, we have all heard of living on the edge, but not the ledge. And plant-based is not a sensible modifier for that puzzling ledge.
Maybe AI will get much better at all of this over time, but maybe not. Just as Donald Trump's replacement for the Affordable Care Act is still and forever will be two weeks away, and Elon Musk's completely automated cars are always five years in the future, it could be that AI will continue to be simply bad. It is, at the very least, not obvious that additional programming could make it choose nontrivial -- to say nothing of accurate -- short-hands for various places. If you knew one of those annoying people who constantly referred to New York as "the city that never sleeps" and gave him "the Big Apple" and "the city so nice they named it twice," he could probably use those three options interchangeably. That would not, however, make him interesting.
As I noted above, I have chosen in this column not to focus on the existential risks that AI evidently poses. It could indeed be dangerous even while it is also over-hyped, inaccurate, and trite. In any case, reaching the point where it no longer produces fake photos of people with 7-fingered hands will not fix the things that stop it from being useful, insightful, or even mildly interesting.
- Neil H. Buchanan