Machine Learning

I will give a difference answer, systems that do online learning are certainly not deterministic in the common sense of the world as their internal changes based on non deterministic behaviour.

Systems that rely on noise generation via non deterministic processes are also non deterministic.

This non determinism is rooted in the change of parts of the state or the input, but for identical state and inputs, the systems are deterministic as long as no bitflips or quantum effects occur in the silicon.

He is right tho

Your teacher's argument is based on the fact that pseudo-random generators are deterministic, which is entirely irrelevant to ML theory.

If you want to make the point that the "can only be" part is extremely far-reaching, just bring quantum physicists in the discussion.

A single AI network is deterministic. If you apply the same input, you get the same output. If you train on the same dataset, in the same order, with the same initial weights and hyperparameters, you will get an identical training result.

The tricky thing is that AI is high dimensional and non-linear. So what appears to be a very small change to the input can cause a large change in the output. I think the clearest example of this is adversarial AI.

Your teacher's argument is based on the fact that pseudo-random generators are deterministic, which is entirely irrelevant to ML theory.

If you want to make the point that the "can only be" part is extremely far-reaching, just bring quantum physicists in the discussion.

He is right tho

A single AI network is deterministic. If you apply the same input, you get the same output. If you train on the same dataset, in the same order, with the same initial weights and hyperparameters, you will get an identical training result.

The tricky thing is that AI is high dimensional and non-linear. So what appears to be a very small change to the input can cause a large change in the output. I think the clearest example of this is adversarial AI.

Human output is the same - deterministic with a stochastic factor, although you may prefer to call the latter free will.

The joke is that so are you, and so is him.

floating point arithmetic is "non-deterministic"

I will give a difference answer, systems that do online learning are certainly not deterministic in the common sense of the world as their internal changes based on non deterministic behaviour.

Systems that rely on noise generation via non deterministic processes are also non deterministic.

This non determinism is rooted in the change of parts of the state or the input, but for identical state and inputs, the systems are deterministic as long as no bitflips or quantum effects occur in the silicon.

Basically yes. As far as we know, human brains don't employ quantum randomness in any meaningful manner, so they're also deterministic.

What difference does it make? It doesn't say much about what AI systems or humans can or can't do.

Basically yes. As far as we know, human brains don't employ quantum randomness in any meaningful manner, so they're also deterministic.

What difference does it make? It doesn't say much about what AI systems or humans can or can't do.

Ultimately it depends on whether the system is closed or open. If the system is closed (a model with inputs and outputs) then it's deterministic. If the system is open (if it reaches out to the internet, it asks you for your own opinion, it hires mechanical turks from Amazon to fine tune it, etc) then it might not be deterministic (if any of the inputs are not deterministic).

I think MCMC is a family of methods that puts lie to the claim of determinism. Unless his point is “if you set the random seed the same, then this code block will produce the same result with perfect fidelity”. In which case, sure, okay.

Ultimately it depends on whether the system is closed or open. If the system is closed (a model with inputs and outputs) then it's deterministic. If the system is open (if it reaches out to the internet, it asks you for your own opinion, it hires mechanical turks from Amazon to fine tune it, etc) then it might not be deterministic (if any of the inputs are not deterministic).

floating point arithmetic is "non-deterministic"

sounds like free will vs determinism.

The real question is, are minds deterministic.

I think MCMC is a family of methods that puts lie to the claim of determinism. Unless his point is “if you set the random seed the same, then this code block will produce the same result with perfect fidelity”. In which case, sure, okay.

True. But what's their point? So are we.

sounds like free will vs determinism.

The real question is, are minds deterministic.

Human output is the same - deterministic with a stochastic factor, although you may prefer to call the latter free will.

True. But what's their point? So are we.

The joke is that so are you, and so is him.

btw

instead of starting from a premise and trying to justify it.

Evaluate the facts before forming your thesis.

Because this sounds like a religious argument.

'Help me defend my belief.'

You shouldn't be asking for facts to support your conclusion, because you shouldn't have a conclusion without facts.

It depends on what level of abstraction you are claiming deterministic behaviour. As stated elsewhere, at the upper level of qualia, it’s hard to say whether something that looks and feels like a decision made with free will is or isn’t.

Likewise, if you move to the lower levels of bit patterns, electron flow or quantum events, it looks to an outside observer to be non-deterministic.

So, at the absurd level of abstraction that posits symbols being manipulated by executing software are real phenomena, you could argue that neural nets are deterministic.

But at what point and to which observer does complexity become indistinguishable from randomness?

It’s a shaky argument that is based on the perfect functioning of an ideal of a computer to claim determinism, when we know in practice that abstraction levels bleed into each other, form strange loops and the Blue Screen Of Death is only ever a couple of bits away, especially when the sun flares and you’re not using ECC RAM.

If the seed is set, it is deterministic.

AI is absolutely deterministic. You can get different results for your model by switching around the the seed used for your random initialization, but if you give it the same seed and the same training data in the same order you'll get the same model every time. As for evaluation, once the model weights are fixed you'll get the exact same results every time you run the model with your prediction data set.

By his definition, everything is deterministic.

He's not wrong, but they're are lots of things that can throw a wrench into the predictability, for example, if you're using a hugging face model, and the weights file changes out from under your nose.

Or if the hardware you're executing on has a bug (like the IEEE floating point issue on 486s back in the day).

Or if the model has the precision reduced or increased by the hardware it's running on in a significant way.

Or the stochastic random bits are unobservable, etc.

In these cases it still is deterministic, it's just not easy to determine, especially when small hardware changes (as opposed to algorithmic ones) can change the output.

Once trained, a models outputs are completely determined by its inputs at a mathematical level. This is easy to prove. You write out the equation for each output variable (it's large but possible) and observe that each parameter is constant and not a random variable. Thus the output is also constant given some inputs and weights.

Training is arbitrary but also deterministic, since the combination of the initial states, training batch order, and optimization algorithm and parameters, determines the output.

If you feed random parameters into any of these then the output is arbitrary, but not random

For example at inference time, a transformer is not using any random number, not even any pseudo random number.

I think you are mistaking the sampling procedure with the model itself. The sampling procedure is often pseudo random. The model usually is something that produces a probability distribution. That's deterministic.

How would they be non-deterministic?

Answer to him with 4.1 :) smth like this: This is an interesting discussion, and you are making an important argument about determinism in the performance of artificial intelligence (AI) and neural networks! But, there are:

1. Determinism in AI and Neural Networks

In general, your lecturer is correct that most AI algorithms, including neural networks, are deterministic in the sense that they are a fixed set of mathematical operations. If you put the same input data set into an AI model, you will get the same result every time you run it, provided the model and input data do not change.

Example:

Consider a neural network for image recognition. If you give the same image as input, the recognition result will be the same each time you run it, because the mathematical operations performed by the network will be the same.

2. Stochastic Factor and Indeterminacy

However, there are aspects in AI where stochasticity plays a role:

• Initialization of Weights: In machine learning, especially deep learning, the initial weights of a neural network are often initialized randomly. This means that different initializations can lead to different learning paths and possibly different results.

• Stochastic Gradient Descent: Many learning algorithms use stochastic gradient descent, where the training data is sampled randomly in each iteration.

Example:

Suppose we are training a neural network for image classification. If the initial weights and the order in which the data is fed in each training session are different, the final model will probably produce different results on the same input image.

3. The Role of Randomness in Some AI Algorithms

In some AI algorithms, such as genetic algorithms or Monte Carlo-based methods, randomness is an important part of the process.

Example:

Genetic algorithms use random mutations and crossovers to generate new solutions, which makes their results non-deterministic in a sense.

Conclusion

In summary, while the basic operation of a neural network or other AI algorithm may be deterministic, the processes that lead to the creation and tuning of these algorithms often involve stochastic elements. This means that there may be non-deterministic aspects to the overall process of creating and using AI models.

How about VAEs? You sample from a distribution during training and inference

Determinism in computational models, including binary systems, relies on the ability to reproduce results given the same initial conditions and operations. In a deterministic system, if you run the same sequence of instructions with the same inputs (including the seed for random number generation), you should expect the same output every time, assuming the system is isolated from external non-deterministic factors.

Multithreading introduces non-determinism when threads operate in a shared environment and their execution order affects the outcome. It’s the responsibility of the programmer to manage this through synchronization mechanisms to ensure deterministic behavior if required.

There is also analog and quantum computers. Analog computers work on the principle of approximation and continuous variable manipulation, which can introduce non-deterministic elements due to physical variations. Quantum computers operate on quantum bits (qubits) and can produce non-deterministic results because they exploit quantum superposition and entanglement.

In the context of machine learning models and AI, these principles apply as well. Binary-based AI will be deterministic if the conditions are controlled, while quantum and analog AI might introduce non-deterministic elements by their nature or design for efficiency.

If its running on a computer, its deterministic.

What an edge lord.

Yes. But not always computationally reducible, you tell him kindly.

“systems” doing a lot of heavy lifting there. Any system is deterministic if you recreate exact conditions and expunge all randomness outside of the “system”.

An AI system (or for simplicity) an agent, can output an stochastic policy. (I.e., in a tennis match, use your backhand 60% of the time). But for the same input sequence, the same network will always reach the same outcome, which is deterministic.

I think the smartest response in your discussion with your teacher is to clarify what is and what isn't an AI system. If sampling from a stochastic distribution is within the AI box, it might well be stochastic to an observer.

Can you explain how humans are non-deterministic from that point of view? Isn't this more of a philosophical/physics question than it is a computer science one? In my view, it's all deterministic. The whole lot of it. Both brains and computers are deterministic. I'm not really fully understanding the distinction. Neurons perform calculations like this as well, taking the graded sum of potentials etc...

He’s right, unless you factor in hardware effects such as bit flips which will always occur at non-zero probability. Anyways the same is true for people. Neglecting quantum effects, we are of course also made of fully deterministic processes. So to me it’s not so interesting a distinction

Maybe for silicon neural networks, but what about quantum neural networks??

Why do you feel the need to prove your professor(who's right) wrong? are you that annoying classmate?

computers are deterministic so any computer program is deterministic...
quantum computers do change that but AI as we know it today is a program that runs on classical computers.

/!\ Unpopular opinion /!\
It seems that everyone here seems to "like" determinism, like everything in a computer should be deterministic, and the only things we could be talking about are floating point additions and random numbeer generators...
Indeed a model IS a determination. Most modern AI, like most classical math functions, are models.
When you have an algorithm that compute the result of a model, of course the computation should be deterministic !
Unless your model is stochastic : what you are modelling can be a form of randomness. In that case, the result of the model can be the density function itself, or a generator, that draws item from this distribution.
But if you are doing a probability exercise in a math class, the simulation you are coding is made "deterministically" : you determine what the density is.
Statistical learning is about letting the data determine most of the model (its parameters). And assuming the data you have sampled is "random", then "randomness" has determined the precise model you have trained. (And in theory, for that to be possible, a lot of assumptions must hold. Assuming the object you want to model contain some randomness, statistical learning can be the best way to make a good probabilistic model.)
In that sense, AI is "non-deterministic" : the developper don't determine what it do exactly, and it is often a model from a random reality.
To train a statistical model IS to draw a set of parameters from a given distribution. The answers are in the questions of statistical learning : you're model is assumed to be "random".

"Four" is a random number, no a deterministic one... in this sentence at least.

I'm looking at you in bayesian...

Deep learning models are probabilistic models implemented on deterministic machines. If you implemented it on a quantum computer you could say it’s completely probabilistic

I would thank him humbly for this piece of knowledge