Has anyone built/bought a good solution for measuring image output quality of SD, MJ, or Dall-E? Had a few questions about strengths and weaknesses if you’ve used something. Also interested in the answer for text and voice apps.

This is actually something I've been looking into. I'm creating custom diffusion models trying to get it to perform for a specific use case. In terms of your question: there are general benchmarks, but it really depends on your use case and what you're defining as quality. (e.g art, or using it to generate synthetic data)

These have been the main players in the space that I have seen so far • Freeplay.ai • Braintrustdata.com • Kolena.io • Rungalileo.io https://wandb.ai/site But will drop you a DM as would love to chat more on this

What's your def of quality? The classic metric in research is FID score, which measures realism or more accurately similarity to a reference set. But that may not. be what you want - quality may be artisticness, or realism, or just whether someone stops for an extra second if that see it in a feed.

Quality in this sense is more tied to business outcomes such as user acceptance i.e. was it successful in helping the user complete the desired action or outcome.

if I were tackling this problem, my first hypothesis would be that I'd have to model this myself, first with my own hand labeling (you could label 1000 images a day yourself) and eventually from user behavior. It's going to be tough to find some 3P platform that understands your user and business problem.