this post was submitted on 02 Oct 2026
42 points (90.4% liked)
Technology
88838 readers
4828 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I hate to say this but can't LLMs solve the cognitive debt problem better then people? Wouldn't having a stochastic machine that can have its state frozen in time, retrived at convience, and fed the same inputs be a potentially more transparent machine, thinking or otherwise?
I agree with you on stating thr problem though, and the term seems concise enough to me. The gap seems to be not in ability to do this but that creating graspable and reasonable audits for LLM usage is like many risk mitigation systems, an after thought in the industry.
This is where you go off the rails. LLMs are probalistic, not deterministic. It will probably give the same output, but thats still not actually true. This is where hallucinations come from.
Tbf the exact input and node activation is all you'd need for forensics, the only reason to refeed would old input plus new input mix, which should be new output.
Its not neural nets, its staticics. There aren't nodes to trace.
Most of the LLMs architectures that I know of are neural net based.
For training maybe. But execution as far as i know are just a bunch of probabilities chucked into matrices. Back in my physcos days I could u derstand the math, but not today.
Also what needs citing, that nodes translate to debugable, reproducible outputs.
Because again, it's all probablilties under the hood.
I mean probabilities in matrices are nodes in a neural net, right?
The only thing that makes it non-deterministic is the tempature value which is known after the fact from my understanding, so that should be able to deterministic.
No.... No they're not. Either that or nerual nets are even dumber than I thought. My understanding was emulating nodes of information like brains do. Thats not probabilities. But willing to be wrong if you can bring sources. The connection between points in probability doesn't make sense. Thats a nonsense statement.
Thats also no how simulated annealing works. The temperature value is roughly the probability it will pick a different, less optimal step, in order to try and find better alternative paths. The temperature value is roughly the probability of trying something 'random'. Not deterministic at all. The temperature value is lowered with time and progression. Its known the entire time, but its still a weighted coin flip that determines which path to take.
https://en.wikipedia.org/wiki/Simulated_annealing
Neural nets are dumber than you thought for sure. What I mean is the coin flip results are known after the fact.
I'm not convinced of that you do. Nor is knowing that flip helpful. Because it's non deterministic. An internal coin flip is not an input either.
What I mean is for post hoc analysis since you know the coin flips and the inputs you can step through the layers and nodes calculation deterministically.
I mean sure, but to what end?
You can see where it went wrong. Not why, how
You can try to reason about it which is an arguable improvement from the former engineer you can't get ahold of anymore. So you can could see that it triggered a node associated with c++ when it generated this string of tokens or compare it other generations to map it to a type of call.
Where it doesn't compare to a human engineer at the moment is quality or novelty.
But you can't?
It didn't reason its way down a branch, it made coinflips all along the way. So its not actually thinking, just weighted coin flips. There's no 'why' it chose a branch. Just the result of a coin flip.
You can't ask a coin why it chose heads. You can unroll the physics of it all day long til the cow comes home. But it still doesn't help you on the next one.
It's not really coin flips it and random adjustments of weighted values that decides which nodes are activated which, plus random adjustments, influences the next node activation until out
Statistical outputs are not as interesting as you think?
Like sure, but its still just a probabilistic outcome. Not one reasoned at any step.
I mean it is a semantic collection of weights tied to tokens (fractions of written language). There is also tool calls which could be analysied for additional input analys.
Better, probably not. And I'll give you one example. In a codebase there was an issue with Auth, after few runs of the LLM, the best suggestion resulted in 30~40% of change in the Auth workflow.
Took me a couple hours to figure out that the problem was a misconfiguration key (
camelCasetosnake_casein the vault store). Fixing this on the store (not on the code) fixed the Auth workflow.This two hours were the cognitive debt. And keep in mind, I'm familiar with this part of the code. A Mechanical Parrot happy-trigger boyz would accept the change in the codebase as an attempt to fix it.
So, mechanical parrot can help? Sure. But better than people, probably not. At least not yet, not with the current set of tool, not with the promise of fully solving it.
This is badly wrong. The same node can point to multiple places depending on the
Kfactor on this. And this isn't even theTapplied to it. So, the same node can infer multiple different others. This is the nature of the probabilistic of LLM.First agreed. Honestly I haven't an LLM do anything close to an engineer. You can get lucky for a code snippet or two and maybe a few ADRs but even those are fraught with the chance of slop clean up work. I am not arguing that.
The node is only probalistic because of the random number added. After the fact the number is known.
So you are suggesting that we should have the metadata for every decision and direction. This is a
N^Namount of data. Keeping this data is wastefull (even more than the usage of LLM right now). Not saying that is is useless, but for sure this won't help to mitigate the cognition since this data don't carry meaning for us humans.No that should be discoverable with the models weights, input and random numbers added to the weight at the time.
Say for example you find that a collection of outputs behave oddly or in an undesired way, you could use this to find what simularties they share with each other but delta with other and naively prune the nodes or simply decrease their weights. Those you could also try to corralate that to certain input tokens to engineer better prompts or try to trace it back to initial training data.
It could also be a failed tool call adding garbage data in, or malicious. A trace could catch that as well.
No, the random part happens in the query.
There is also random in the training, but the query also generate more random numbers.
Otherwise this would be a deterministic procedure, and it's not.
We are saying the samethings but you are adding no to it.
Right, during inference random numbers are generated, that plus the numbers from input are added to the weight values and the matrix multiplication happens. If you used the same random numbers and inputs it is deterministic. For regular use you don't do that because you want a stochastic output, if you wanting to do forensics and trace what led to an output you would benifit from that determinism.
Gotcha,
But this means we need to provide not only the same query (and be sure that the tokenizer is the same) but also provide the seed for the number generation. In this case we will have a deterministic outcome. (Unless we provide the list of numbers used, which for me feels wasteful)
But at this point none of the providers have this feature.
And I don't think the open source have this also. But open source can be updated/changed.
Not that I see either looking at opensource inference engines. vLLM supports setting the seed so if you have that you do, and maybe it's just as simple as harness setting and recording that. At least for replay this request type of operation but that does not show where in model each random numbers would have been applied.