Jess Zimmerman: I can’t believe how few conversations about the uses of LLMs seem to incorporate this fact. It is not a knowledge or understanding machine! It is a thing-saying machine!

I can’t believe how few conversations about the uses of LLMs seem to incorporate this fact. It is not a knowledge or understanding machine! It is a thing-saying machine!

Do not use LLMs for information. Of any kind. It does not *know* anything.

“Oh we can use it in publishing for things like press releases and title information” no you CAN’T! It will simply say shit! The press release has to be about what’s ACTUALLY IN THE BOOK

It’s a damned obvious Turing test that so many people somehow can’t pass

I have a theory that this is how most people do conversations with a lot of folks they aren't close to - just pick up on a couple of words and then auto compose a sentence in response. Which may explain a lot. (Possibly about me.)

It's also basically the register the entirety of the executive class operates in. Which of course is why they're so impressed by it.

If *people* can’t pass it, is it really a Turing test?

Using LLMs for actual information is like using a Roomba to clean your house. All it knows how to do is bump into things and suck up anything it rolls over.

Someone at this AI in Publishing panel I was at this week rapturously compared LLMs to Roombas (as in, they’ll do your scutwork without your involvement!) and I silently sent my coworkers the post about what happens when a Roomba runs over some dog shit

Same thing that happens when a self driving car runs over a person.

happy to report that the latest Roomba ads are on the theme "we fixed the slathering-every-surface-of-your-home-in-dogshit problem, really, we promise"

I saw one that was like "Roomba now with more pet applications" and I clicked bc I thought maybe they had added some amusing lights or a little chair for cats but no, they had added a sensor that can allegedly tell when something is made out of shit

okay but that chair for cats thing is a million dollar idea

Despite copious evidence, why do ostensibly intelligent people believe in this AI bullshit? It’s clearly a psychological flaw in a certain type of person. Did they fail all scientific, mathematical and technical subjects at school? Are they so devoid of creativity and talent they don’t see the lack?

It’s so funny how LLMs are quite literally “At long last, we have created the Chinese Room from classic thought experiment The Chinese Room Isn’t Knowledge or Understanding”

The rise in LLMs this year have convinced me to add the Chinese Room paper to my intro syllabus next semester because 🤦‍♀️🤦‍♀️🤦‍♀️🤦‍♀️ I’m so sick of people not understand this

Blindsight by Peter Watts is a good book

It’s literally just predictive text, like what you see on your phone keyboard, just scaled up. That people are using it as a search engine or a database is bonkers.

I’ve been in meetings and have had knock down arguments because someone used gpt as a source of truth

Occasionally someone in a FB group will “ask ChatGPT” something on my behalf and it makes me incandescent with rage

I feel like Google has been low key preparing us for this because "google it" used to actually mean getting meaningful information, but now might give you the right link if you scroll through 200 ads and other dogshit first. We've been prepped to accept crap now.

Which is maddening because my day job involves doing research on contemporary artists and their work, and google is just a string of hurdles slowing me down instead of the helpful tool it used to be.

I was talking about that a while back

And honestly, “do your research/just Google it” was ALWAYS lazy, smug advice because often people didn’t know where to start and what keywords to use. But it’s worse now.

Yes you did need a google foo, now you need blackbelt google kung foo

I know the guy that pitched paid advertising at the top of search results. Best I can tell, it was pure enshitification to decide to trash the core business. Not a long game.

Yuuup

i left a small discord server that had been going for years as basically an international group chat because one motherfucker refused to stop doing this

Yeah my brother did this regarding something to do with my replacement passport and I was just Thanks Bro but I think I will just assume the worst case from the website rather than trust chat GPTs version of reality.

Yeesh

And now 'people' includes Meta. (low growl)

when i see people dismissively say it's "a search engine that talks to you" i'm like no no no no no it's not even that lmao it's like. *even less useful than that.* at least the guy operating the mechanical turk knew how to play chess

I saw someone suggest using ChatGPT to convert a JSON file into a CSV file, and I'm sitting here thinking: this takes 30 seconds with a tool that works every time and can be interrogated in case of fault. By using ChatGPT to do this, you have taken data and potentially turned it into bullshit.

There was somebody asking the r/askMath subreddit why chatGPT couldn't add 30ish numbers correctly, and half the comments were explaining why, and the other half were just shouting "YOUR PHONE HAS A CALCULATOR! USE THE CALCULATOR!"

Jeez, and people spent 30 odd years trying to convince Microsoft to stop Excel from mangling CSV files only for some idiot to do this.

And used a whole lot more electricity than you needed to…

Someone recently described AI as “the world’s greatest bullshitter” & I think that b/c the people promoting it have only ever bullshit their way through life they think that IS what human intelligence is.

Bullshitting our way through things is what separates mankind from the animals... except for bulls.

I want to scream every time I see someone ask a question online followed by "I don't know if anyone can answer this, because I even asked ChatGPT and it was no help." IT. IS. NOT. A. SEARCH. ENGINE.

The simplest and least correct understanding of LLMs is that they function like large search engines that can also compose sea shanties

not as good of sea shanties as a person could write though, unless you absolutely need a sea shanty in the next 30 seconds

Surely in a pinch any good sea shanty people will just start singing and figure it out as they go? (Not that this is preferred. That in a SEA SHANTY NOW OR DEATH situation I'd still choose a human.)

yes

you can basically just throw together the words 'shore' 'captain' 'home' 'mate' 'grog' and 'awful' in a sentence and job's a good 'un

GenAI sea shanties are as likely to tell you to HEAVE when you're supposed to HO because they can't count and don't know why they should.

I always know when to ho and do not need any assistance in that arena.

It's an automated bullshitting machine

What frustrates me is that, even with the belief that an LLM can 'think' or 'understand', I don't get why someone would think "this thingie that has learned from bunches of humans, who often confidently say wrong things, will probably give me accurate information." Absolutely fucking bananas to me.

I wrote about blue green algae at Lough Neagh and other water bodies here in Northern Ireland (www.wired.com/story/blue-g...). In response a Redditor posted ChatGPT's advice: use chemicals and aeration to deal with it. Lough Neagh is not a pond. It is the largest lake in the British isles, 400km2.

Rivers Are Drowning in Toxic Sludgewww.wired.com Harmful algal blooms are taking over as the world warms and grows richer in carbon dioxide—and there’s no easy fix.

Absolutely correct. I wonder if the failure is that people insufficiently understand themselves or others as thinking-meatsacks vs being in saying-things or producing-things categories.

It is a machine for telling people what they want to hear and a lot of them assume that means it must provide a higher form of truth than other sources.

Exactly. LLMs produce discourse-shaped text probabilistically. There is NO code within them that performs any kind of semantic processing.

Post