https://cerebralvalley.ai logo
#06-technical-discussion
Title
# 06-technical-discussion
j

Jim Park

06/06/2023, 3:43 AM
What do y’all think about the MLOps side of running an LLM? I’d like to be able to give admins the ability to a) manage feedback from conversations to tweak prompts and run A/B experiments b) manage documents in vector stores ( for example running ETL ) Is there frameworks I should be looking at, or is this a build from scratch endeavor?
l

Leon Wu

06/06/2023, 5:04 AM
what scale are you working at? feels like you can probably actually get a long way with the most primitive of tools. my current best prompt is stored in an email to myself lolol
j

Jim Park

06/06/2023, 6:14 AM
It’s not so much scale as it is UX. I want a non-technical user to be able to say “here is a new article / website / pdf / book / video” that I want added to the corpus of materials that this chat bot references
d

Deepanshu

06/06/2023, 4:04 PM
Hey @Jim Park, you can check Scale AI’s Spellbook or Humanloop for managing feedbacks/prompt experimentation, though I have myself built an internal tool ( using Retool) to manage feedbacks and for Human in Loop
j

Jim Park

06/06/2023, 5:54 PM
Thanks! I’ll experiment with both! I’m curious if anyone has thoughts on retrieval corpus management?