wojcech

joined 10 months ago
sorted by: new top controversial old
1
[R] "It's not just memorizing the training data" they said: Scalable Extraction of Training Data from (Production) Language Models (arxiv.org)
1
[R] "It's not just memorization" they said: Extracting Training Data from ChatGPT (not-just-memorization.github.io)
1
EPFL releases an open Medical Llama 2 finetune, including weights and training data, within 5%/10% of GPT-4/Med-PaLM-2 (arxiv.org)
Is anyone experimenting with non-instruction tuned models? in c/localllama@poweruser.forum
[–] wojcech@alien.top 1 points 9 months ago (1 children)

Just to be clear, you aren't doing fine tuning here as in gradient updates, you are using the base model + ICL?

Is anyone experimenting with non-instruction tuned models? in c/localllama@poweruser.forum
[–] wojcech@alien.top 1 points 9 months ago

Fine tune as in gradient updates or as in ICL?

1
Is anyone experimenting with non-instruction tuned models? (alien.top)
 

My main usecase for LLMs is literally as an auto-complete, mainly via coding, so I was wondering whether anyone has played with/had any luck using the base model for use cases that are close to simply auto completing? I could imagine the instruction tuning adding a sycophancy bias in those areas

[D] Simplest mathematical example of a function that can only be solved by gradient descent in c/machinelearning@academy.garden
[–] wojcech@alien.top 1 points 10 months ago

Go back to your history: Cauchy is the earliest person I'm aware of to have used gradient descent, and he motivated it as

one ordinarily starts by reducing them to a single one by successive eliminations, to eventually solve for good the resulting equation, if possible. But it is important to observe that 1◦ in many cases, the elimination cannot be performed in any way; 2◦ the resulting equation is usually very complicated, even though the given equations are rather simple

That is, the usefulness of gradient descent is motivated when you have rough idea of when you are close to the minimum, but you don't want to go through the hassle of algebra. (realistically, if you can solve it with gradient descent, you could probably solve it algebraicly, we just don't have the same stupidly easy to implement computational routines for it)

https://www.math.uni-bielefeld.de/documenta/vol-ismp/40_lemarechal-claude.pdf

[R] Feeling lost about way to enter AI/ML research in c/machinelearning@academy.garden