Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

[D] How do you deal with LLM observability? What tools do you guys use? (alien.top)

submitted 10 months ago by Ok_Cartographer5609@alien.top to c/machinelearning@academy.garden

10 comments fedilink hide all child comments

I want to know the tools and methods you use for the observability and monitoring of your ML (LLM) performance and responses in production.

you are viewing a single comment's thread
view the rest of the comments

[–] Serquet1@alien.top 1 points 10 months ago

Hey, we recently rolled out Nebuly, a tool focused on LLM observability in production. Thought it might be of interest to some here, and potentially useful for your needs. Here are some highlights:
- Deep User Analytics: More insightful than thumbs up/down, it delves into LLM user interactions.
- Easy Integration: Simply include our API key and a user_id parameter in your model call.
- User Journeys: Gain insights into user interactions with LLMs using autocapture.
- FAQ Insights: Identifies the most frequently asked questions by LLM users.
- Cost Monitoring: Strives to find the sweet spot between user satisfaction and ROI.

For a deeper dive, here's our latest blog post on the topic: What is User Analytics for LLMs.