this post was submitted on 21 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
Orca 2: Teaching Small Language Models How to Reason (www.microsoft.com)
submitted 10 months ago by Memories-Of-Theseus@alien.top to c/localllama@poweruser.forum
14 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] LinuxSpinach@alien.top 1 points 10 months ago

Progressive Learning: We start with LLaMA-2-7B or LLaMA-2-13B checkpoint and
finetune it on the train split of FLAN-v2 dataset for one epoch. Note that FLAN-v2 dataset
contains both zero-shot and few-shot problems. We then train on 5 million ChatGPT data
from Orca 1 for 3 epochs. Then we train on the combination of 1 million GPT-4 data from
Orca 1 and Orca 2’s 817K data for 4 epochs.