Once trained, a models outputs are completely determined by its inputs at a mathematical level. This is easy to prove. You write out the equation for each output variable (it's large but possible) and observe that each parameter is constant and not a random variable. Thus the output is also constant given some inputs and weights.
Training is arbitrary but also deterministic, since the combination of the initial states, training batch order, and optimization algorithm and parameters, determines the output.
If you feed random parameters into any of these then the output is arbitrary, but not random
For example at inference time, a transformer is not using any random number, not even any pseudo random number.
I think you are mistaking the sampling procedure with the model itself. The sampling procedure is often pseudo random. The model usually is something that produces a probability distribution. That's deterministic.
I have been able to expand the context window of multimodal models like gpt4 simply by rendering the text to images at a small font size and then feeding it in as images. I've not done large scale studies to determine the total increase in perplexity or anything but my empirical results have been great. Plus you get the ability to analyze non standard text.
If it were me, I would LoRA adapt a model to take in image input. There's a lot of space in the token embedding space that is completely barren that could be used for reasoning.