I'm busy building an evaluation framework that I c...
# 06-technical-discussionj
Jono Millin
03/12/2024, 10:13 PMI'm busy building an evaluation framework that I can use to test changes in prompts and comparing different models (OpenAI / Google / Anthropic / OSS), and realized this probably already exists somewhere. Any good recommendations for model evaluation out there?
Jono Millin
03/12/2024, 10:21 PMMore info on the problem - I'm looking to compare different vision models (GPTV / Gemini 1.5, Claude 3.0, LLaVA etc..) for issue detection. Both:
1. Was an issue of a specific category detected or not (compared to ground truth on test set, showing true/false negative/positives)
2. Was the description of the issue similar to ground truth data (AI evaluated)
I'm looking to run a fairly small test set (~100 samples) across different models/prompts to see what works best, and iterate and track progress and which combination is winning.
t
Tom Smoker
03/12/2024, 10:48 PMj
Jono Millin
03/13/2024, 3:07 PMThanks Tom! Helpful paper. Have you had any experience with commercial / OSS systems for this? (e.g. W&B, LLM Spark etc..)
Jono Millin
03/13/2024, 8:49 PMTook a quick poke and came across:
Paid:
• Weights and Biases
• https://www.braintrustdata.com/
• https://www.getscorecard.ai/
• https://promptlayer.com/
• https://www.traceloop.com/
• https://athina.ai/
• https://www.helicone.ai/
OSS
• https://github.com/uptrain-ai/uptrain
• https://github.com/Arize-ai/phoenix
• https://github.com/promptfoo/promptfoo
• (see more: https://github.com/topics/llm-evaluation )
Curious to hear if anyone has direct/personal experience?
e
Ehren Maedge
04/11/2024, 9:09 PMBraintrust seems solid. Great customers, investors and the team is active on discord helping.
ty fixed 1
10 Views