LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Looking for Python script to deploy custom LLM in Azure (alien.top)

submitted 11 months ago by MyObjectivism@alien.top to c/localllama@poweruser.forum

2 comments fedilink hide all child comments

Background is.. trying to build interface for users to choose LLM (like Falcon, Deepsake etc from Huggingface) from my portal which will make script to download and deploy that particular LLM in Azure.

Once it is deployed, users will use those LLMs to build apps. Deploying custom LLM in user/client cloud environment is mandate as there is data security policies in play.

If anyone worked on such script or have an idea then please share your inputs.

you are viewing a single comment's thread
view the rest of the comments

[–] kivathewolf@alien.top 1 points 11 months ago

While I have not tried this in azure, my understanding is that you can deploy a Linux vm with A100 in azure (T4or V100 may not work for all use cases, but will be a cheaper option). Once you have a Linux vm with GPU, you can choose how you would like to host the model(s). You can write some code and expose the LLM via an API ( I like Fast chat, but there are other options as well). Heck you can even use ooba if you like. Just make sure to check the license for what you use.