https://cerebralvalley.ai logo
#06-technical-discussion
Title
# 06-technical-discussion
t

Ted Cheng

08/09/2023, 4:00 PM
I am curious how does everyone do chatbot/agent conversation monitoring and QA? I am looking at LangSmith but also curious if there are other tools.
šŸ‘€ 1
i

Ivan Porollo

08/09/2023, 6:22 PM
I think @Alex Reibman may have some insight here
a

Alex Reibman

08/09/2023, 6:23 PM
Yo! Iā€™m behind agentops.ai. Happy to chat
šŸ†’ 1
šŸ‘€ 1
m

Mikyo King

08/10/2023, 1:17 AM
šŸ‘‹ Maintainer of arize-phoenix, we have LLM App analysis tooling that's open-source to help you troubleshoot synthesis. For production arize has monitoring and evals support as well. Check out our recent webinar!

https://www.youtube.com/watch?v=hbQYDpJayFw&t=196sā–¾

Happy to chat too!
h

Henry Scott-Green

08/10/2023, 7:42 AM
Hi @Ted Cheng I'm building a product analytics tool for LLM-powered products. We give builders visibility into how people use their products, and how well the product meets the user needs. Would love to have a chat! We're at context.ai
r

Rahul Parundekar

08/10/2023, 2:53 PM
I think HumanLoop just launched tracing. Baserun.ai, a friend, is also getting in this space. Weights and Biases has a solution to do waterfall tracking as well. Seems like someone beat me to mentioning Arize. One of my biggest challenges I've seen with tools in PromptOps space is that many of them assume you talk directly with OpenAI - and not have any Retrieval Augmented Generation. Or, for example there might be a distributed flow between multiple models or systems that you want to track. To solve for some of these problems across the entire lifecycle of a prompt, Ive been working on PromptStash (see the "Deploying the Variant in Production" section for the traces/stashing). We are very agnostic to the chaining libraries and model vendors. If you're open to it, would appreciate feedback.
t

Ted Cheng

08/11/2023, 8:38 PM
@Alex Reibman Thanks for sharing Is there a way to get a demo or early access?
@Mikyo King Will take a look at Phoenix. For observability within Arize, I think I'd need to be an Arize platform customer first?
@Henry Scott-Green Is there a way to do a trial of the product?
@Rahul Parundekar super interesting. will take a look!
m

Mikyo King

08/11/2023, 8:48 PM
@Ted Cheng - Arize is free to get started under a certain volume!
šŸ™Œ 1
h

Henry Scott-Green

08/12/2023, 10:34 AM
@Ted Cheng yep! It's free and very easy to get set up with our SDKs. Check out docs.context.ai
I'll DM you now