LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Any interest in C#/.NET for serving LLMs? (alien.top)

submitted 11 months ago by Key-Comparison3261@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

K024/llm-sharp: Language models in C# (github.com)

I've recently drafted this. But adding more models & features & tests & documentation will just cost too much time. Seeking for comments & colaborators.

top 3 comments

sorted by: hot top controversial new old

[–] No-Cod6872@alien.top 1 points 11 months ago (1 children)

Definitely interested, I loved C#/.NET but haven't used it much in recent years. Have you used python for inference? I'm wondering how it compares to C# for LLM related programming

[–] Key-Comparison3261@alien.top 1 points 11 months ago

You have exllama, vllm, lmdeploy in python. And in most cases fastapi is used for serving an http endpoint.

I wrote llm-sharp just for dropping python (GIL, pip deps) and getting flexible adaptation to dynamic model structures apart from standard llama.

[–] Languages_Learner@alien.top 1 points 11 months ago

This may be useful: saddam213/LLamaSharp: C#/.NET binding of llama.cpp, including LLaMa/GPT model inference and quantization, ASP.NET core integration and UI. (github.com)