Featureless_Bug

I mean, Falcon 40 B with lora can easily be trained on 2x A100 with lora (even llama 70b can be trained on just 2x A100). But maybe accelerate is doing something stupid - in my experience, both deepspeed and accelerate are very slow and require way too much memory compared to manual gpu distribution strategy.

Macbook Air is not money well spent. Workstation with 4090, and A100 in the cloud on the money you saved