Technology

It's not a statistical method anymore. One of the breakthroughs of large model neural networks, has been that during training an emergent process, assigns neurons to both relatively high level and specific traits, which at the same time "cluster up" with other neurons assigned to related traits. Adding just a bit of randomness ("temperature") allows the AI to jump from activating one trait to a close one, but not to one too far away. Confidence becomes a measure of how close is the output, to a consistent set of traits trained into the network. Interestingly, a temperature of 0 gives a confidence of 100%... but produces gibberish.

If its data contains a commonly held belief, that is incorrect

This is where things start to get weird. An AI system based on an LLM, can iterate over its own answers looking for the optimal one (Q*), and even detect inconsistencies in them. What it does after that, depends on whoever programmed it:

• Maybe it casts any doubt aside, and outputs the first answer anyway (original ChatGPT did that, didn't even bother self-checking too much)
• Or it could ask an authoritative source (ChatGPT plugins work like that)
• Or it could search the web for additional info (Copilot and Gemini do that)
• Or it could alert the user to both the low confidence and the inconsistencies (...but people want omniscient AIs, not "err... I'm not sure, Dave" AIs)
• ...or, sometime in the future (or present?) they could re-train themselves, maybe via generating a LoRa, that would bring in corrected biases, or even additional concepts.

Over time, I think different AI systems will evolve to target accuracy, consistency, creativity, etc. Current systems are kind of rudimentary compared to what's yet to come, and too many are used in very rudimentary ways by anyone who can slap an "AI" label and sell them.