What do y’all think about the MLOps side of running an LLM? I’d like to be able to give admins the ability to a) manage feedback from conversations to tweak prompts and run A/B experiments b) manage documents in vector stores ( for example running ETL ) Is there frameworks I should be looking at, or is this a build from scratch endeavor?

what scale are you working at? feels like you can probably actually get a long way with the most primitive of tools. my current best prompt is stored in an email to myself lolol

It’s not so much scale as it is UX. I want a non-technical user to be able to say “here is a new article / website / pdf / book / video” that I want added to the corpus of materials that this chat bot references

Hey @Jim Park, you can check Scale AI’s Spellbook or Humanloop for managing feedbacks/prompt experimentation, though I have myself built an internal tool ( using Retool) to manage feedbacks and for Human in Loop

Thanks! I’ll experiment with both! I’m curious if anyone has thoughts on retrieval corpus management?