It took several tries, but I finally found a way to ask Copilot why it fucked up. It gave a pretty good answer – but an answer that implies that you just can't rely on an LLM to answer even basic questions.
There is no truth or falisity baked anywhere into the thing whatsoever. What it has learned is "grammar and naturalness of language". Which is, don't get me wrong, an enormous technical achievement. But the thing doesn't know anything at all, and especially with mathematical things, it really sucks
I'm very much in the "it's a thing with a purpose, that can accomplish many things", but yeah, people's whose main interest is marketing and sales oversold it, and then the tech cheerleaders got involved.
And yeah, then people (rightly) started dunking on those two groups.
it's also worth contrasting the other technical task that ChatGPT is surprisingly good at - coding assistance. There, creativity is less valuable than a consistent following of dominant conventions -repeatable code is more valuable than hyperoptimized code. Programmers favor naturalness
I'm convinced this is a big reason why the tech crowd is so infatuated with AI. It can remind them to close the parenthesis or whatever and so they think it's good at everything else
I'm sorry, but that's nonsense. I know developers who use AI for things a lot more advanced than "closing a parenthesis."
Which is, incidentally something you don't need AI to do, as any VI user (invented in 1980) will tell you.
but the point is that good code is consistently styled, and there's a limited range of arrangments of code that will even run/compile. It's a good use case for AI in exactly the way that pure math is not.
Interestingly, code formatters are actually often based on an algorithm that's described in terms of an algebra for document formatting! It ends up being very math-y in ways you wouldn't expect:
homepages.inf.ed.ac.uk/wadler/paper...
I'm not a programmer and I will defer to someone who is when they say AI is a useful tool in that pursuit. I simply lack the language to provide a good example of writing code. If/then loop? I don't know.
The Cohere example I gave earlier is a pretty good example of how an LLM can be very useful for creating code. It solves a simple problem that a half-competent programmer doesn't actually need help to solve – but it show how an LLM can match a problem description to the code needed solve it.
