What can this possibly mean?
Brains evolved over 500 million years ago; language probably less than 100,000. So he can't mean it literally.
Bach studies cognitive architectures and has a PhD in cognitive science, so he knows the difference between brains and transformers.
So Bach thinks that maybe instead of formulating a thought and then choosing how to say it, that he’s calculating the probability of the next word based on the previous word?
On a really basic level, wouldn’t that be a logical block to the ability to create new words? What’s the statistical chance of choosing a word that doesn’t exist?
They don’t actually produce anything new, though. They string pieces of existing bs together in different ways, but the atomic elements (words) are not new.
its also capable of interpolating between the existing things it was trained on. This will produce novelty which you can use to train future model. This is mostly a thing ai ppl should not do. Its not built around producing new words but it could be built to do so. (This wouldn’t make them good)
