Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

A[r]xiv Dives - Fine-tuning with LoRA paper deep dive (blog.oxen.ai)

submitted 10 months ago by FallMindless3563@alien.top to c/machinelearning@academy.garden

3 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] HighFreqAsuka@alien.top 1 points 10 months ago (2 children)

LoRA fine tuning is an incredibly simple idea. For each matrix you want to fine-tune, introduce a low rank matrix ΔW = BA where the inner dimension is r << d, and compute (W + ΔW)x. Freeze all pretrained parameters and only update B and A. B is initialized to 0 so that the initial model is equal to the pretrained model. After training, you can also write V = W + ΔW to preserve latency.

Saved you a click.

[–] residentmouse@alien.top 1 points 10 months ago (1 children)

Well now I feel almost obligated to click - is the part of the title "deep dive" completely misleading or is the post really just a LoRA explanation?

[–] FallMindless3563@alien.top 1 points 10 months ago

I’d like to think we dove deep, but let me know!