> For the complete documentation index, see [llms.txt](https://docs.eduplex.eu/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eduplex.eu/text-analysis/ai-tutor-rag-system/answer-generation-evaluation.md). # Answer Generation Evaluation **Goal**\ To evaluate the generator component of the RAG system, assessing how well the responses answer the question and are supported by the provided context.\ \ **Data**\ Set of questions, retrieved chunks, and generated responses from Kritisches Denken (serviceID 2699) and Agiles Mindset (serviceID 2700). 20 content-related questions for each course.\ \ **Method/Approach**\ Replicating the RAG pipeline, doing a vector search in OpenSearch to retrieve relevant chunks (context). The retrieved chunks and query were passed to the LLM, which generated an answer. The generated answers were evaluated by the LLM for relevance and groundedness.\ \ **Results**\ For both KD and AM, the average relevance score was 0.89, indicating that most answers were highly relevant to the queries. Groundedness scores varied: KD (0.75); Agiles Mindset (0.83).\ \ **Evaluation Metrics**\ Answer Relevance: Evaluates whether the final response addresses the entirety of the user’s question.\ Groundedness: Evaluates if the generated response is based on the context provided by assessing the information overlap between each sentence of the response and relevant parts of the context.\ Scores produced by the LLM are mapped to a range from 0 to 1 (0 = not relevant/not grounded, 1 = relevant/grounded) and averaged to get an overall score.\ \ **Conclusions**\ Most answers were highly relevant to the queries. Outliers with low scores revealed two main issues: retrieval failures where the correct chunk wasn’t retrieved, and content limitations where the available context was insufficient to fully answer the question. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.eduplex.eu/text-analysis/ai-tutor-rag-system/answer-generation-evaluation.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.