What can this possibly mean?
Brains evolved over 500 million years ago; language probably less than 100,000. So he can't mean it literally.
Bach studies cognitive architectures and has a PhD in cognitive science, so he knows the difference between brains and transformers.
So Bach thinks that maybe instead of formulating a thought and then choosing how to say it, that he’s calculating the probability of the next word based on the previous word?
On a really basic level, wouldn’t that be a logical block to the ability to create new words? What’s the statistical chance of choosing a word that doesn’t exist?
you could probably make a model with word generation for a language like german (you do the prediction at the word-component level rather than the whole word) but english?
If it’s statistically generated, it’s not going to do it on its own. You’d need some rules for when and how to generate neologisms, but I’m not sure those are “learnable” as opposed to imposed by programming.
they probably are, in the same way grammatical rules are vaguely approximated by llms. and certainly llms have created sentences never before uttered by a human, so they should similarly create some fascinating new german words, lol
altho you’d have to ask someone much better at computers how much the difficulty increases if you no longer have the space character as easy end/beginning of learnable unit…
But the grammatical rules are approximated by parsing existing content and applying the approximation to existing elements, where neologisms by their nature are elements that do not exist until they are created.
oh, i see where we’re having a miscommunication. most german words, including neologisms, are made of the same set of core building blocks. it’s a language made almost entirely of portmanteaus that follow specific patterns. train on the core blocks and merging patterns.
the whole point is that in the specific context of this one specific language, it might be theoretically possible to create new words via the same mechanisms that create new sentences in english llms
I’m familiar with German, though it’s decades ago. The same issue is there, though. For instance, under what circumstances does someone (or an LLM) create a new word? Is the new word a portmaneau or is it a loanword? Is it an antimeria, where something like “Xerox” becomes a verb? How to decide?
They don’t actually produce anything new, though. They string pieces of existing bs together in different ways, but the atomic elements (words) are not new.
its also capable of interpolating between the existing things it was trained on. This will produce novelty which you can use to train future model. This is mostly a thing ai ppl should not do. Its not built around producing new words but it could be built to do so. (This wouldn’t make them good)
