Has anyone built/bought a good solution for measuring image output quality of SD, MJ, or Dall-E? Had a few questions about strengths and weaknesses if you’ve used something.
Also interested in the answer for text and voice apps.
11/05/2023, 3:48 AM
This is actually something I've been looking into. I'm creating custom diffusion models trying to get it to perform for a specific use case.
In terms of your question: there are general benchmarks, but it really depends on your use case and what you're defining as quality. (e.g art, or using it to generate synthetic data)
I haven't found an existing solution on the market yet? Mostly have been reading papers and trying to create my own. If you've looked into any, would love to hear about them too. Happy to discuss and bounce some ideas together on this
What's your def of quality? The classic metric in research is FID score, which measures realism or more accurately similarity to a reference set. But that may not. be what you want - quality may be artisticness, or realism, or just whether someone stops for an extra second if that see it in a feed.
11/10/2023, 6:24 AM
Quality in this sense is more tied to business outcomes such as user acceptance i.e. was it successful in helping the user complete the desired action or outcome.
11/10/2023, 4:26 PM
if I were tackling this problem, my first hypothesis would be that I'd have to model this myself, first with my own hand labeling (you could label 1000 images a day yourself) and eventually from user behavior. It's going to be tough to find some 3P platform that understands your user and business problem.