Has anyone used any good labeling solutions to create ground truth dataset for RAG evaluation? The task is to label the relevant context from a corpus of documents (mostly pdfs) given a question. The existing NLP labeling seem to be more focused on text entity extraction than labeling a whole chunk of content.

What type of labeling do you need? Have you tried approaches like Llamaindex's automated document hierarchy construction

Essentially we need to label the ground truth contexts (chunks) that should be retrieved to answer a user query and use it to evaluate / benchmark retrieval systems.

Got it - yeah I would say to try playing around with document hierarchies as that would be the solution. https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/usage_documents.html You can use LLMs to help generate summary context too to throw into the metadata. We've been exploring using knowledge graphs for document hierarchies too, and happy to chat on that front if helpful.

I understand the llamaindex document structure. I'm more looking for a human labeling UI, like labelbox / scaleAI type of solution to get human-labeled ground truth.

We're working on something along these lines at Oxen.ai if you want to chat!

@Sumanyu you might have some ideas here