LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Identity-PO: DeepMind takes the ELO out of DPO (alien.top)

submitted 1 year ago by georgejrjrjr@alien.top to c/localllama@poweruser.forum

6 comments fedilink hide all child comments

https://x.com/kylemarieb/status/1728281581306233036

New DeepMind paper just dropped.

Background: Direct Preference Optimization (DPO) is the simpler, more robust, higher performing successor of RLHF used in Zephyr, Intel’s new model, and others.

Identity-PO simplifies DPO, removing its reliance on ELO scores (and the mathematical assumptions that come with them). The authors claim this solves overfitting, which is huge if true.

The trend towards simpler solutions and sounder mathematical grounding in alignment is fun to watch. These inscrutable matrices are looking awfully controllable, and the failure modes of the old methods were things like wedding party collapse.

you are viewing a single comment's thread
view the rest of the comments

[–] big_ol_tender@alien.top 1 points 1 year ago (1 children)

It’s already available in huggingface dpo trainer too

[–] georgejrjrjr@alien.top 1 points 1 year ago

Ty, that’s helpful to know.