I am curious how does everyone do chatbot/agent conversation monitoring and QA? I am looking at LangSmith but also curious if there are other tools.

I think @Alex Reibman may have some insight here

Yo! I’m behind agentops.ai. Happy to chat

👋 Maintainer of arize-phoenix, we have LLM App analysis tooling that's open-source to help you troubleshoot synthesis. For production arize has monitoring and evals support as well. Check out our recent webinar! Happy to chat too!

Hi @Ted Cheng I'm building a product analytics tool for LLM-powered products. We give builders visibility into how people use their products, and how well the product meets the user needs. Would love to have a chat! We're at context.ai

I think HumanLoop just launched tracing. Baserun.ai, a friend, is also getting in this space. Weights and Biases has a solution to do waterfall tracking as well. Seems like someone beat me to mentioning Arize. One of my biggest challenges I've seen with tools in PromptOps space is that many of them assume you talk directly with OpenAI - and not have any Retrieval Augmented Generation. Or, for example there might be a distributed flow between multiple models or systems that you want to track. To solve for some of these problems across the entire lifecycle of a prompt, Ive been working on PromptStash (see the "Deploying the Variant in Production" section for the traces/stashing). We are very agnostic to the chaining libraries and model vendors. If you're open to it, would appreciate feedback.

@Alex Reibman Thanks for sharing Is there a way to get a demo or early access?

@Ted Cheng - Arize is free to get started under a certain volume!

@Ted Cheng yep! It's free and very easy to get set up with our SDKs. Check out docs.context.ai