One of the interesting parts of the ELIZA effect doesn't even involve computers at all: people like people who speak like them. Simply regurgitating a person's words back at them, makes them feel like they are being understood, that there is a bond being formed.
ELIZA made this very clear, by literally just rearranging a person's words in the form of a question.
But it's really the basis behind a lot of human in-group bonding. Every scammer knows that you gotta mirror your mark.
You can take someone's essay, mix it up, and give it back to them as a rebuttal, and they'll think it's a more intelligent argument against theirs than one actually written by a real person.
If you input your tweets as the corpus of a chatbot, you'd find the output to be an unusually intelligent, perceptive person. You'd find yourself bonding very, very easily with your own words.
People love seeing reflections of themselves, this is why elevators have mirrors.
You don't even need a computer to make a person think they're speaking to a friendly, intelligent person. Just show them a reflection of themselves and wait.
An *actual* very hard problem in AI, is that you literally do not need any AI involved for people to be utterly convinced they're dealing with AI.
Meet Claude Shannon's machine for playing Hex, it's just a resistor mesh, no logic gates or programming involved.
It is very, very hard to get people to *not* see AI where there is none, and this should be entirely expected of a species that can stub its toe on something then act like the inanimate object did it on purpose.
Ask any ttrpg player and they'll happily tell you about the personalities of individual dice they own.
Humans desperately want to see things as agents. Even when they built them and know exactly how they work.
The *real* big problem is how little AI you need to get people to assign it agency and thereby diffuse blame from themselves onto it.
What's the *shittiest, dumbest* AI you need for people to be convinced to do what it says?
We already know, via actually-existing AI's predating *electricity*: bureaucracies and corporations allow people to carry out absolutely amoral decisions with absolutely no sense of blame or responsibility for the outcome.
The question is not what a really fancy, very scifi AI could do.
It's what you could do with the shittiest, bargain basement knockoff AI that is just good enough to convince people it's an AI.
