this post was submitted on 30 Nov 2023
1 points (100.0% liked)

LocalLLaMA

11 readers
4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago
MODERATORS
communick@poweruser.forum
1
A new way to speed up the work of transformers. (alien.top)
submitted 2 years ago by BIITKN@alien.top to c/localllama@poweruser.forum
5 comments fedilink hide all child comments
 

Has anyone already read this new article on ArXiv? https://arxiv.org/abs/2311.10770

Looks very promising, potential inference acceleration of PyTorch x30, and when implemented on native CUDA x117, and also an estimate of the maximum acceleration x341 times.

As far as I understand, this is achieved by replacing traditional forward propagation layers with so-called fast forward propagation layers.

Is there anyone here with real experience of contributing to the development of PyTorch, llama.cpp or releasing open models, what do you say to this?

you are viewing a single comment's thread
view the rest of the comments
[โ€“] Wonderful_Ad_5134@alien.top 1 points 2 years ago (1 children)

" we provide high-level CPU code achieving 78x speedup over the optimized baseline feedforward implementation"

Big if true, we wouldn't need to buy 3090 cards anymore to get sufficiant memory, just buying more RAM would suffice

[โ€“] pmp22@alien.top 1 points 2 years ago

Huge, if true.