# Retrieval Performance Indexing pdf vs xapi, and Keywords vs Questions

**Goal**\
Comparing retrieval performance across two indexing methods (pdf and xapi) and two query methods (Keywords vs Questions)\
\
**Data**\
Future Skills Module with 20 test questions.\
Keywords generated from questions using GPT-4 (gpt-4-1106-preview). Indexed documents in OpenSearch from pdf and xapi structure.\
\
**Method/Approach**\
Retrieval evaluation using two different indexing formats in OpenSearch (pdf and xapi) and two query methods (keywords vs full question). Vectoization with sbert. Relevance evaluated using gpt-4 as relevance grader (prompt based on trulens-eval library) to score similarity between query and retrieved chunks.\
\
**Results**\
Average relevance score for xapi index: 0.45. Average relevance score for pdf index: 0.3\
Full questions as queries resulted in a higher average relevance score (0.47) compared to keywords (0.34).\
\
**Evaluation Metrics**\
Mean Context Relevance Score (LLM-based): score from 0 to 1, averaged over 2 runs for each query.\
\
**Conclusions**\
Indexing based on xapi structure resulted in better retrieval relevance scores compared to pdf indexing.Using full questions as queries provides more accurate retrieval compared to keywords. So question-based retrieval is preferable for higher context relevance.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.eduplex.eu/text-analysis/ai-tutor-rag-system/retrieval-performance-indexing-pdf-vs-xapi-and-keywords-vs-questions.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.