👋 Maintainer of arize-phoenix, we have LLM App analysis tooling that's open-source to help you troubleshoot synthesis. For production arize has monitoring and evals support as well.
Check out our recent webinar!
Hi @Ted Cheng I'm building a product analytics tool for LLM-powered products. We give builders visibility into how people use their products, and how well the product meets the user needs. Would love to have a chat! We're at context.ai
08/10/2023, 2:53 PM
I think HumanLoop just launched tracing. Baserun.ai, a friend, is also getting in this space. Weights and Biases has a solution to do waterfall tracking as well. Seems like someone beat me to mentioning Arize.
One of my biggest challenges I've seen with tools in PromptOps space is that many of them assume you talk directly with OpenAI - and not have any Retrieval Augmented Generation. Or, for example there might be a distributed flow between multiple models or systems that you want to track. To solve for some of these problems across the entire lifecycle of a prompt, Ive been working on PromptStash (see the "Deploying the Variant in Production" section for the traces/stashing). We are very agnostic to the chaining libraries and model vendors. If you're open to it, would appreciate feedback.
08/11/2023, 8:38 PM
@Alex Reibman Thanks for sharing Is there a way to get a demo or early access?
@Mikyo King Will take a look at Phoenix. For observability within Arize, I think I'd need to be an Arize platform customer first?
@Henry Scott-Green Is there a way to do a trial of the product?
@Rahul Parundekar super interesting. will take a look!
08/11/2023, 8:48 PM
@Ted Cheng - Arize is free to get started under a certain volume!
08/12/2023, 10:34 AM
@Ted Cheng yep! It's free and very easy to get set up with our SDKs. Check out docs.context.ai